Skip to content

Views fk guesser - #152

Merged
tim-band merged 25 commits into
mainfrom
views-fk-guesser
Nov 4, 2026
Merged

tim-band merged 25 commits into
mainfrom
views-fk-guesser

Conversation

@tim-band

@tim-band tim-band commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Separate the Foreign Key / Primary Key guesser from the Parquet file reading.
Allow the guesser to be used independently, make it work when --views-to-tables is used (views don't have FK/PK constraints so the normal method of understanding PKs or FK relationships won't work).
Fixes #151

Comment thread datafaker/make.py Outdated
Comment on lines +731 to +732
if guess_keys:
add_guessed_keys(meta_dict["tables"])

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add_guessed_keys() runs over the entire merged tables dict, not just the views/parquet tables that actually need guessing, so it can silently overwrite correct primary/foreign_keys/nullable metadata on ordinary already-reflected tables, including tables reflected straight from the database with real PK/FK constraints already set by metadata_to_dict/column_to_dict.

For example, when running the make-tables command, I got the following warnings:

Found multiple likely primary keys for table appointment_practitioner: appointment_id, id, practitioner_id
Found no likely primary keys for table concept_map
Found no likely primary keys for table emis_clinical_code
Found no likely primary keys for table emis_drug_code
Found multiple likely primary keys for table location_contact: id, location_id
Found no likely primary keys for table national_data_opt_out
Found multiple likely primary keys for table patient_address: id, patient_id
Found multiple likely primary keys for table patient_contact: id, patient_id
Found no likely primary keys for table patient_opt_out
Found multiple likely primary keys for table patient_person: id, patient_id, person_id
Found no likely primary keys for table patient_uprn
Found no likely primary keys for table postcode_hash
Found multiple likely primary keys for table practitioner_in_role: id, practitioner_id
Found multiple likely primary keys for table schedule_practitioner: id, practitioner_id, schedule_id

Perhaps we can do the following:

  • have get_metadata/make_tables_file track which table names actually need guessing (views — e.g. via inspect(engine).get_view_names() — plus the parquet-derived names already returned by get_initial_parquet_orm), and only guess those.
  • skip any column in add_guessed_keys that already has primary: true or an existing foreign_keys entry, regardless of which table it’s in.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very interesting. Yes, I didn't do this because of the messiness, and we haven't yet encountered a situation where we have both tables that need guessing and tables that do not. But you are correct, it would be much better if we coped with this correctly. Good idea about the get_view_names(). Perhaps I should remove the guess_keys argument also? I will add a test and see how far I get.

Comment thread datafaker/parquet2orm.py Resolved
Comment thread datafaker/utils.py Resolved
Comment on lines +215 to +216
parsy.string("TIMESTAMP_TZ").result(lambda: sqltypes.DATETIME(timezone=True)),
parsy.string("TIMESTAMPTZ").result(lambda: sqltypes.DATETIME(timezone=True)),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parsing "TIMESTAMP_TZ" yields a bare <function> object instead of a DATETIME instance.

Shall we use the following?

Suggested change
parsy.string("TIMESTAMP_TZ").result(lambda: sqltypes.DATETIME(timezone=True)),
parsy.string("TIMESTAMPTZ").result(lambda: sqltypes.DATETIME(timezone=True)),
parsy.string("TIMESTAMP_TZ").result(sqltypes.DATETIME(timezone=True)),
parsy.string("TIMESTAMPTZ").result(sqltypes.DATETIME(timezone=True)),

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's confusing. These results need to be callable, hence why mostly they return types (as opposed to objects).

tim-band and others added 5 commits November 3, 2026 16:18
text_list returns natural order

Co-authored-by: yhong123 <yean-hoon.ong@ucl.ac.uk>
Co-authored-by: yhong123 <yean-hoon.ong@ucl.ac.uk>
@tim-band
tim-band merged commit 4420611 into main Nov 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

make-tables --views-to-tables requires PK and FK guessing

3 participants