Skip to content

fix(gridsight-ext): SQL-safe view names, partitions without PIVOT - #44

Merged
micahpw merged 1 commit into
mainfrom
hermes/gridsight-view-names
Nov 20, 2026
Merged

micahpw merged 1 commit into
mainfrom
hermes/gridsight-view-names

Conversation

@micahpw

@micahpw micahpw commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two fixes to the GridSight extension (gridsight-ext), found while loading the CATS week (examples/cats-sienna in GridSight) into the desktop app.

1. Scenario view names are valid SQL identifiers

The extension builds its DuckDB view names from the project and scenario names. It replaced only -, so a scenario named Summer peak 2019 gave ext_gat.CATS_Summer peak 2019__load, and every default chart failed with a DuckDB parser error.

  • New _view_prefix(project, scenario) replaces every non-word character with _ (re.sub(r"\W", "_", …)). It is used in all 4 places that built the prefix.

2. Daily partitions without PIVOT (memory)

get_dataset_partition built each day with UNPIVOT + PIVOT ON entity_id. For line flows, that pivots on ~10,800 entity ids. DuckDB allocated several GB for it and kept the memory (default memory_limit is 80% of RAM), so the GridSight backend grew from ~0.4 GB to ~7.3 GB on the first partition request.

  • Now it reads the day's hour columns as stored (entities × hours) and transposes in Arrow (_partitions.transpose_day). The result is the same table: a datetime column, one float64 column per entity, entities sorted, first row wins for duplicate ids.
CATS, one day Before After
Backend memory after a line-flow partition 7,261 MB 425 MB
Line-flow partition time 0.8 s 0.16 s
Response 3,325,136 bytes 3,325,136 bytes (same)

Tests

  • tests/test_partitions.py (new): the new layout equals the old PIVOT query (column order, timestamps, NULLs, duplicate ids), plus an empty table.
  • uvx ruff check gridsight-ext: pass. pytest gridsight-ext/tests: 13 passed.
  • Checked in the GridSight desktop app with the CATS scenario: default charts load (no SQL errors), the map and the time-series animation are unchanged, get_errors is empty.

…thout PIVOT

- _view_prefix(project, scenario) replaces every non-word character with
  "_", so scenario names with spaces (e.g. "Summer peak 2019") give valid
  DuckDB view names. Before, every default chart failed with a parser error.
- get_dataset_partition reads the day's hour columns as stored and
  transposes in Arrow (_partitions.transpose_day) instead of UNPIVOT +
  PIVOT ON entity_id. Same output; on CATS line flows (~10.8k entities)
  backend memory after one partition drops from ~7.3 GB to ~0.4 GB.
- tests/test_partitions.py: new layout equals the old PIVOT query.
@micahpw micahpw changed the title gridsight-ext: SQL-safe view names, partitions without PIVOT fix(gridsight-ext): SQL-safe view names, partitions without PIVOT Nov 20, 2026
@micahpw

micahpw commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

The 4 pytest failures are in core Sienna regression tests and are not caused by this PR (it changes only gridsight-ext/). They come from the rebuilt Sienna fixture cache; see #45.

@micahpw
micahpw merged commit d423e92 into main Nov 20, 2026
6 of 9 checks passed
@micahpw
micahpw deleted the hermes/gridsight-view-names branch November 20, 2026 04:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant