Repository navigation
Conversation
| ty = "^0.0.43" | ||
| colorama = "^0.4.6" | ||
| snowflake-sqlalchemy = "^1.11.1" | ||
|
|
There was a problem hiding this comment.
Shall we add snowflake-connector-python = { version = ">=4.8,<5", optional = true, extras = ["secure-local-storage"] } here?
When I ran poetry run datafaker make-tables after setting SRC_DSN, I was prompted to install snowflake-connector-python[secure-local-storage] via pip install snowflake-connector-python[secure-local-storage].
There was a problem hiding this comment.
Are we going to make snowflakes as dependencies rather than optional? Anyone that uses the tools needs snowflakes whether they are using it or not at the moment.
There was a problem hiding this comment.
It's probably not too heavy a dependency? If so we'll need to fix it but it makes a huge difference to installability. As it is you can just pipx install it, but I can't find any way of pipx installing any optional dependencies.
If we have a nice way to do it I'm happy to split these things out.
There was a problem hiding this comment.
We can remove the optional, and keep snowflake-sqlalchemy/snowflake-connector-python as required dependencies. Making them truly optional would need a guarded-import fix (try/except ImportError) in places that import the snowflake modules.
|
When running |
| element: snowflake_types.TIMESTAMP_LTZ, compiler: Any, **kw: Any | ||
| ) -> str: | ||
| """Render TIMESTAMP_LTZ as TIMESTAMP_LTZ.""" | ||
| return compiler.process(element, **kw) |
There was a problem hiding this comment.
compile_timestampltz_ddb is registered via @compiles(snowflake_types.TIMESTAMP_LTZ, "snowflake"), and inside it calls compiler.process(element, **kw) on the same element. Since this function is the dispatch target for (TIMESTAMP_LTZ, "snowflake"), that call re-enters itself — infinite recursion. Any DDL/query involving a TIMESTAMP_LTZ column against a Snowflake destination will raise RecursionError, for example when running the make-tables command.
Shall we call the type compiler’s native visit_TIMESTAMP_LTZ method directly, bypassing @compiles dispatch?
| return compiler.process(element, **kw) | |
| return compiler.visit_TIMESTAMP_LTZ(element, **kw) |
| _element: snowflake_types.TIMESTAMP_LTZ, compiler: Any, **kw: Any | ||
| ) -> str: | ||
| """Render TIMESTAMP_LTZ as DateTime.""" | ||
| return compiler.process(DateTime(timezone=False), **kw) |
There was a problem hiding this comment.
The generic fallback renders TIMESTAMP_LTZ as DateTime(timezone=False). Since TIMESTAMP_LTZ is timezone-aware by definition, should this be DateTime(timezone=True) instead?
There was a problem hiding this comment.
A really tricky one. This is Local TimeZone; how on earth do we model that? This is what I thought was the least bad option, but I'm totally open to different ideas.
There was a problem hiding this comment.
Can we set all timezone to UTC for example? So everything is converted to UTC
There was a problem hiding this comment.
@tim-band , what do you think of Nik's suggestion? Or should we just stick with your existing implementation?
| case _: | ||
| tables = [] | ||
| logger.error( | ||
| "--parquet-dir cannot be used with %s, only duckdb.", dialect_name | ||
| ) |
There was a problem hiding this comment.
When --parquet-dir is passed with a non-duckdb dialect, this logs an error but continues normally (tables = []), unlike the DSN-malformed path above which does raise Exit(1). Should this fail hard instead, so it doesn’t look like a silent success in CI/batch runs?
|
|
||
|
|
||
| def _not_copying_incompatible_fk(column: Column, fk_col: RelatedColumn) -> bool: | ||
| """Return True if ``column`` and ``fk_col`` are incompatible foreign key types.""" |
There was a problem hiding this comment.
The docstring says this returns True when the columns are incompatible, but the call site (line 587) keeps entries where it returns True, and the logic matches compatible pairs. Docstring looks inverted from the actual/intended behavior.
| src_table = column.table | ||
| dialect = engine.dialect | ||
| col = literal_column(f'"{column_name}"') | ||
| src_table = column.table # preserves schema for schema-qualified databases |
There was a problem hiding this comment.
src_table = column.table is assigned twice in a row (331 and 333)
|
Found one blocking bug: infinite recursion compiling |
Co-authored-by: nikk-nikaznan <48319650+nikk-nikaznan@users.noreply.github.com>
Implement UUID proposer, fix compile_timestampltz_ddb and pyproject.toml, cast UUID columns to string in feature extractors.
Can connect to a Snowflake database like so:
We can get the tables with the new flag
datafaker make-tables --views-to-tables, but the PK and FK constraints don't come across.Fixes #148