Repository navigation
Views fk guesser - #152
Views fk guesser#152
Conversation
These don't work in Snowflake
Simplified the code as a result
because Snowflake doesn't like this.
Also loading hive partitioned parquet is now done by fastparquet.
Updated --views-to-tables help text. Added a test to show that PK/FK guessing works with views. Removed a spurious log call.
| if guess_keys: | ||
| add_guessed_keys(meta_dict["tables"]) |
There was a problem hiding this comment.
add_guessed_keys() runs over the entire merged tables dict, not just the views/parquet tables that actually need guessing, so it can silently overwrite correct primary/foreign_keys/nullable metadata on ordinary already-reflected tables, including tables reflected straight from the database with real PK/FK constraints already set by metadata_to_dict/column_to_dict.
For example, when running the make-tables command, I got the following warnings:
Found multiple likely primary keys for table appointment_practitioner: appointment_id, id, practitioner_id
Found no likely primary keys for table concept_map
Found no likely primary keys for table emis_clinical_code
Found no likely primary keys for table emis_drug_code
Found multiple likely primary keys for table location_contact: id, location_id
Found no likely primary keys for table national_data_opt_out
Found multiple likely primary keys for table patient_address: id, patient_id
Found multiple likely primary keys for table patient_contact: id, patient_id
Found no likely primary keys for table patient_opt_out
Found multiple likely primary keys for table patient_person: id, patient_id, person_id
Found no likely primary keys for table patient_uprn
Found no likely primary keys for table postcode_hash
Found multiple likely primary keys for table practitioner_in_role: id, practitioner_id
Found multiple likely primary keys for table schedule_practitioner: id, practitioner_id, schedule_id
Perhaps we can do the following:
- have
get_metadata/make_tables_filetrack which table names actually need guessing (views — e.g. viainspect(engine).get_view_names()— plus the parquet-derived names already returned byget_initial_parquet_orm), and only guess those. - skip any column in
add_guessed_keysthat already hasprimary: trueor an existingforeign_keysentry, regardless of which table it’s in.
There was a problem hiding this comment.
Very interesting. Yes, I didn't do this because of the messiness, and we haven't yet encountered a situation where we have both tables that need guessing and tables that do not. But you are correct, it would be much better if we coped with this correctly. Good idea about the get_view_names(). Perhaps I should remove the guess_keys argument also? I will add a test and see how far I get.
| parsy.string("TIMESTAMP_TZ").result(lambda: sqltypes.DATETIME(timezone=True)), | ||
| parsy.string("TIMESTAMPTZ").result(lambda: sqltypes.DATETIME(timezone=True)), |
There was a problem hiding this comment.
parsing "TIMESTAMP_TZ" yields a bare <function> object instead of a DATETIME instance.
Shall we use the following?
| parsy.string("TIMESTAMP_TZ").result(lambda: sqltypes.DATETIME(timezone=True)), | |
| parsy.string("TIMESTAMPTZ").result(lambda: sqltypes.DATETIME(timezone=True)), | |
| parsy.string("TIMESTAMP_TZ").result(sqltypes.DATETIME(timezone=True)), | |
| parsy.string("TIMESTAMPTZ").result(sqltypes.DATETIME(timezone=True)), |
There was a problem hiding this comment.
No, it's confusing. These results need to be callable, hence why mostly they return types (as opposed to objects).
text_list returns natural order Co-authored-by: yhong123 <yean-hoon.ong@ucl.ac.uk>
Co-authored-by: yhong123 <yean-hoon.ong@ucl.ac.uk>
Separate the Foreign Key / Primary Key guesser from the Parquet file reading.
Allow the guesser to be used independently, make it work when
--views-to-tablesis used (views don't have FK/PK constraints so the normal method of understanding PKs or FK relationships won't work).Fixes #151