Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: test fixture-deck fixture-deck-sienna fixture-deck-all regen-snapshots \
.PHONY: test gridsight-bundle fixture-deck fixture-deck-sienna fixture-deck-all regen-snapshots \
sienna-image-v4 sienna-image-v5 sienna-fixture-v4 sienna-fixture-v4-multi sienna-fixture-v5 sienna-fixture-pjm5

PYTHON ?= python
Expand Down Expand Up @@ -58,3 +58,15 @@ sienna-fixture-pjm5: sienna-image-v4
docker run --rm -e SIM_SYSTEM=pjm5 -e SIM_STEPS=$(SIM_STEPS) \
-v "$(PWD)/$(SIENNA_OUT_PJM5):/output" gat-sienna:v4


# GridSight extension bundle: nlr-gat + gridsight-gat wheels in one zip that
# the GridSight desktop app side-loads (Settings -> "Choose wheel..."). See
# gridsight-ext/BUILDING.md.
GRIDSIGHT_EXT_VERSION := $(shell grep -m1 '^version' gridsight-ext/pyproject.toml | sed -E 's/.*"([^"]+)".*/\1/')
gridsight-bundle:
rm -rf dist/gridsight-bundle
mkdir -p dist/gridsight-bundle
$(UV) build --wheel --out-dir dist/gridsight-bundle .
$(UV) build --wheel --out-dir dist/gridsight-bundle gridsight-ext
rm -f dist/gridsight-gat-bundle-$(GRIDSIGHT_EXT_VERSION).zip
cd dist/gridsight-bundle && zip -j ../gridsight-gat-bundle-$(GRIDSIGHT_EXT_VERSION).zip *.whl
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,20 @@ Highest-traffic doc pages (also readable directly in the repo):
- [Migrating from v0.x to v1.0](docs/source/migration_v1.md)
- [Changelog](docs/source/CHANGELOG.md)

## Use with GridSight

[GridSight](https://github.com/NatLabRockies/GridSight) can show GAT
scenarios as map layers, charts and animated timeseries. The extension
lives in [`gridsight-ext/`](gridsight-ext/) (package `gridsight-gat`).

- **GridSight desktop:** build the side-load bundle with
`make gridsight-bundle`, then pick the zip in GridSight → Settings →
"Choose wheel…". See [`gridsight-ext/BUILDING.md`](gridsight-ext/BUILDING.md).
- **GridSight from source or Docker:** install both packages into the
GridSight server environment:
`pip install "nlr-gat[sienna,server]" "gridsight-gat @ git+https://github.com/NatLabRockies/GridAnalysisToolkit.git#subdirectory=gridsight-ext"`.
GridSight's `scripts/setup-extensions.sh` does this for a sibling clone.

## Example data

`example_data/` is gitignored — the binaries are too large to track
Expand Down
1 change: 1 addition & 0 deletions docs/source/_llms_root/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ dependencies (h5py, duckdb, geopandas, polars) that format needs, e.g.

- [Custom plots and the reporting framework](extending_plots.html): the `@plot_function` decorator and how quickplots/reports are assembled
- [v1 architecture & migration pattern](architecture/v1_migration_pattern.html): the `BaseSystem`/`BaseSimulation` interfaces new format backends implement
- [GridSight extension](https://github.com/NatLabRockies/GridAnalysisToolkit/tree/main/gridsight-ext): `gridsight-gat`, which shows GAT scenarios in the GridSight desktop app and server

## Reference

Expand Down
147 changes: 147 additions & 0 deletions gridsight-ext/BUILDING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Building the GridSight Wheel

This directory holds `gridsight-gat`, the GridSight
extension that surfaces GAT scenarios (Sienna H5 + system JSON) as
GridSight layers, charts, and timeseries.

For GridSight desktop users, the extension can be:

- **Bundled in the installer** (default — no action needed; the installer
builds these wheels as part of the release).
- **Side-loaded** into an existing GridSight desktop app via Settings →
"Choose wheel…". This doc covers building the side-loadable artifact.

---

## TL;DR

From the repo root:

```bash
make gridsight-bundle
```

Produces `dist/gridsight-gat-bundle-<version>.zip`. Hand that file to a
user; they pick it in Settings → "Choose wheel…" and click **Restart
backend**. Done.

---

## What's inside the bundle

```
gridsight-gat-bundle-1.0.0.zip
├── nlr_gat-<gat-version>-py3-none-any.whl
└── gridsight_gat-<ext-version>-py3-none-any.whl
```

Both wheels are in the bundle so it installs offline. `gridsight-gat`
depends on `nlr-gat` (also on PyPI), and the bundled wheel pins the GAT
version that the extension was built against. The desktop's installer
extracts the zip, then passes both wheels to `uv pip install` in a
single invocation — uv resolves the install order from wheel metadata.

The bundle version (in the zip filename) tracks
`gridsight-ext/pyproject.toml`'s `version` field, not GAT's own. So a
GAT-only release that doesn't change the extension keeps the same
bundle name.

---

## Prerequisites

- **uv** ≥ 0.4 ([install](https://docs.astral.sh/uv/getting-started/installation/)).
- A clean checkout of this repo.
- `zip` on PATH (default on macOS/Linux; Git Bash on Windows provides it).

No virtual environment needed — `uv build` runs in isolation.

---

## Building from a tagged release

For a reproducible bundle, build from a worktree pinned to the tag
rather than your live checkout:

```bash
git worktree add /tmp/gat-release nlr-gat-vX.Y.Z
cd /tmp/gat-release
make gridsight-bundle
cp dist/gridsight-gat-bundle-*.zip ~/Desktop/
git worktree remove /tmp/gat-release
```

This guarantees the wheels match the tag exactly — no stray edits or
unstaged files.

---

## Verify before shipping

```bash
make gridsight-bundle
unzip -l dist/gridsight-gat-bundle-*.zip
```

You should see exactly two `.whl` files. Optionally smoke-test the
contents in a throwaway venv:

```bash
unzip -d /tmp/bundle-check dist/gridsight-gat-bundle-*.zip
uv venv /tmp/gat-check
source /tmp/gat-check/bin/activate
uv pip install /tmp/bundle-check/*.whl
python -c "from importlib.metadata import entry_points; print([ep.name for ep in entry_points(group='gridsight.extensions')])"
```

Expect `['gat']` — that's the entry-point GridSight's sidecar discovers
at startup.

---

## Installing into GridSight desktop

1. Open GridSight desktop → Settings.
2. Click **Choose wheel…** under "Install extension".
3. Pick the bundle zip. Wait for "Install succeeded".
4. Click **Restart backend**.

The window reloads and `gat` appears in the extensions list. To
upgrade later, rebuild the bundle at the new tag and repeat — `uv pip
install` upgrades in place.

---

## Manual build (if you can't run `make`)

The Makefile target is equivalent to:

```bash
mkdir -p dist/gridsight-bundle
uv build --wheel --out-dir dist/gridsight-bundle . # nlr-gat
uv build --wheel --out-dir dist/gridsight-bundle gridsight-ext # gridsight-gat
version=$(grep -m1 '^version' gridsight-ext/pyproject.toml | sed -E 's/.*"([^"]+)".*/\1/')
(cd dist/gridsight-bundle && zip -j "../gridsight-gat-bundle-${version}.zip" *.whl)
```

---

## Troubleshooting

**`uv: command not found`**
Install uv: `curl -LsSf https://astral.sh/uv/install.sh | sh` (macOS/Linux)
or `irm https://astral.sh/uv/install.ps1 | iex` (Windows PowerShell).

**`uv build` fails on the gridsight-ext target**
The `gridsight-ext/pyproject.toml` declares `nlr-gat` as a `path = "../"`
dependency for editable installs. `uv build` ignores that at wheel-build
time (the path dep becomes a plain `nlr-gat` requirement in the wheel's
metadata), so this should just work. If it doesn't, check that
`uv --version` is recent enough.

**Desktop "Restart backend" shows no new extension**
Check `~/Library/Logs/GridSight/backend.log` (macOS) for entry-point
discovery errors. The most common cause is a version conflict with the
bundled `nlr-gat`/`gridsight-gat`. If the venv is in a stuck state, wipe it
(`rm -rf ~/Library/Caches/GridSight/venv`) and GridSight will reprovision
from scratch on next launch.
24 changes: 24 additions & 0 deletions gridsight-ext/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "gridsight-gat"
version = "0.1.0"
description = "GridSight extension for GAT (Grid Analysis Toolkit) power system simulation results"
requires-python = ">=3.11"
dependencies = [
"gridsight-sdk",
"duckdb>=1.0",
"pyarrow",
"nlr-gat[sienna,server]",
]

[tool.uv.sources]
nlr-gat = { path = "../", editable = true }

[project.entry-points."gridsight.extensions"]
gat = "gridsight_gat:GATExtension"

[tool.hatch.build.targets.wheel]
packages = ["src/gridsight_gat"]
5 changes: 5 additions & 0 deletions gridsight-ext/src/gridsight_gat/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
"""GridSight extension for GAT power system simulation analysis."""

from .extension import GATExtension

__all__ = ["GATExtension"]
89 changes: 89 additions & 0 deletions gridsight-ext/src/gridsight_gat/_gateway.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
"""Single locked code path for DuckDB access from this extension.

DuckDB Python connections are not thread-safe: concurrent ``execute``
calls can race inside the C++ result machinery and crash the process
with ``InternalException: Attempted to dereference unique_ptr that is
NULL`` (which arrives as a SIGSEGV that takes the container down).

Every site in the extension that touches the shared connection must go
through ``_DuckDBGateway`` so the lock is held for both ``execute`` and
the follow-up fetch. The lock is the same RLock used by the server's
``DuckDBEngine`` (``ExtensionContext.duckdb_lock``), so chart queries
and extension reads serialize against each other on the same handle.

Lives in its own module so the contract can be tested without the
gridsight-sdk import chain.
"""

from __future__ import annotations

import threading
from typing import Any


class _DuckDBGateway:
def __init__(self, conn: Any, lock: threading.RLock) -> None:
self._conn = conn
self._lock = lock

@property
def lock(self) -> threading.RLock:
return self._lock

@property
def raw(self) -> Any:
"""Escape hatch — only for code that must pass the underlying
connection to a third party. Callers MUST wrap their use in
``with gateway.transaction():`` if they execute statements."""
return self._conn

def fetchall(self, sql: str, params: list | tuple | None = None) -> list:
with self._lock:
return self._conn.execute(sql, params or []).fetchall()

def fetchone(self, sql: str, params: list | tuple | None = None):
with self._lock:
return self._conn.execute(sql, params or []).fetchone()

def fetch_arrow(self, sql: str, params: list | tuple | None = None):
with self._lock:
result = self._conn.execute(sql, params or [])
# DuckDB 2.x renamed fetch_arrow_table() to to_arrow_table().
if hasattr(result, "to_arrow_table"):
return result.to_arrow_table()
return result.fetch_arrow_table()

def execute(self, sql: str, params: list | tuple | None = None) -> None:
"""For DDL and statements with no result. Discards the cursor so
no follow-up fetch can race outside the lock."""
with self._lock:
self._conn.execute(sql, params or [])

def execute_silent(self, sql: str) -> None:
"""Like ``execute`` but swallows exceptions. For idempotent
DETACH calls where the table may already be detached."""
with self._lock:
try:
self._conn.execute(sql)
except Exception:
pass

def transaction(self):
"""Context manager that holds the lock across multiple statements.

Use for sequences that must not be interleaved with other queries
(the ATTACH / write / DETACH dance, or any read that depends on
a prior DDL). Yields the raw connection — callers use
``conn.execute(...)`` directly inside the ``with`` block."""
gateway = self

class _Txn:
def __enter__(self):
gateway._lock.acquire()
return gateway._conn

def __exit__(self, *exc):
gateway._lock.release()
return False

return _Txn()
Loading
Loading