Generate precise catalog type stubs

[damonbayer]

100% vibecoded alternative to #98.

I'm not super comfortable adding this to the repo, but it does work as advertised

:

Codex Explanation

Summary

This PR replaces the broad inline CatalogNamespace type-hint workaround with generated .pyi stubs for the installed catalog tree.

The original approach improved editor inference for expressions like:

datacat.public.stf.nhsn_hrd_prelim.load.get_dataframe(output="lazy")

but it did so by declaring endpoint attributes such as load, extract, and data on every catalog namespace. That made invalid paths look valid to static analyzers.

This PR adds a dataops_catalog_stubs command that inspects the installed catalog maps and dataset TOML configs, then writes project-local stubs under:

typings/cfa/dataops/

The generated stubs model:

• real datacat namespace paths
• real reportcat namespace paths
• terminal dataset endpoints as DatasetEndpoint
• actual runtime blob stages from TOML, including load, extract, data, and stage_*
• BlobEndpoint.get_dataframe() overloads, preserving inference such as output="lazy" -> polars.LazyFrame

Why

cfa-dataops discovers catalogs dynamically from installed packages such as CDCgov/cfa-catalog-pub, so inline annotations in catalog.py cannot accurately describe the real installed namespace tree.

Generated stubs are a better fit because they are created from the same installed catalog state used at runtime. For cfa-catalog-pub, this means paths like:

datacat.public.stf.nhsn_hrd_prelim.load

can be represented precisely, while non-terminal namespaces like:

datacat.public
datacat.public.stf

do not falsely appear to have blob endpoint methods.

Maintainability

The first generated-stub draft built the .pyi output with a large hand-assembled string, which was too fragile. This version hardens the implementation by:

• moving stable .pyi syntax into Mako templates
• keeping type_stubs.py focused on catalog-tree/model construction
• adding a golden-file fixture for representative generated output
• parsing generated stubs with ast.parse() in tests
• packaging the templates explicitly in both wheels and sdists

User Workflow

After installing the relevant catalog packages, run:

dataops_catalog_stubs

Pyright/Pylance can use the generated typings/ directory directly. For mypy, use:

MYPYPATH=typings

Regenerate the stubs whenever dataset TOMLs, report notebooks, or catalog paths change.

Validation

• uv run ruff check cfa/dataops/type_stubs.py tests/test_type_stubs.py cfa/dataops/stub_templates
• uv run pytest tests/test_type_stubs.py -q
  • Repo pytest config expanded this to the full test suite.
  • 126 tests passed.
• uv run dataops_catalog_stubs --output-root /private/tmp/cfa-dataops-typings-template-smoke
• uv build --out-dir /private/tmp/cfa-dataops-build-template-smoke
• Confirmed the built wheel contains cfa/dataops/py.typed and both Mako templates.

[ryanraaschCDC]

This seems really promising. However, when I install cfa-dataops into a virtual environment, it doesn't pick up the dataops_catalog_stubs command

[damonbayer]

@ryanraaschCDC did you do uv run dataops_catalog_stubs ?