Repository navigation
fix(gridsight-ext): SQL-safe view names, partitions without PIVOT - #44
Merged
Merged
Conversation
…thout PIVOT - _view_prefix(project, scenario) replaces every non-word character with "_", so scenario names with spaces (e.g. "Summer peak 2019") give valid DuckDB view names. Before, every default chart failed with a parser error. - get_dataset_partition reads the day's hour columns as stored and transposes in Arrow (_partitions.transpose_day) instead of UNPIVOT + PIVOT ON entity_id. Same output; on CATS line flows (~10.8k entities) backend memory after one partition drops from ~7.3 GB to ~0.4 GB. - tests/test_partitions.py: new layout equals the old PIVOT query.
Collaborator
Author
|
The 4 pytest failures are in core Sienna regression tests and are not caused by this PR (it changes only |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two fixes to the GridSight extension (
gridsight-ext), found while loading the CATS week (examples/cats-siennain GridSight) into the desktop app.1. Scenario view names are valid SQL identifiers
The extension builds its DuckDB view names from the project and scenario names. It replaced only
-, so a scenario namedSummer peak 2019gaveext_gat.CATS_Summer peak 2019__load, and every default chart failed with a DuckDB parser error._view_prefix(project, scenario)replaces every non-word character with_(re.sub(r"\W", "_", …)). It is used in all 4 places that built the prefix.2. Daily partitions without PIVOT (memory)
get_dataset_partitionbuilt each day withUNPIVOT+PIVOT ON entity_id. For line flows, that pivots on ~10,800 entity ids. DuckDB allocated several GB for it and kept the memory (defaultmemory_limitis 80% of RAM), so the GridSight backend grew from ~0.4 GB to ~7.3 GB on the first partition request._partitions.transpose_day). The result is the same table: adatetimecolumn, one float64 column per entity, entities sorted, first row wins for duplicate ids.Tests
tests/test_partitions.py(new): the new layout equals the old PIVOT query (column order, timestamps, NULLs, duplicate ids), plus an empty table.uvx ruff check gridsight-ext: pass.pytest gridsight-ext/tests: 13 passed.get_errorsis empty.