diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 6b7c669..03d6c00 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -7,12 +7,8 @@ on: workflow_dispatch: jobs: - pytest-fast: + pytest: runs-on: windows-latest - strategy: - fail-fast: false - matrix: - python-version: ["3.10", "3.12"] steps: - name: Check out repo @@ -21,7 +17,7 @@ jobs: - name: Set up Python uses: actions/setup-python@v5 with: - python-version: ${{ matrix.python-version }} + python-version: "3.12" - name: Install uv uses: astral-sh/setup-uv@v4 @@ -30,29 +26,7 @@ jobs: run: uv sync --locked --group dev - name: Run correctness lint - if: matrix.python-version == '3.12' run: uv run ruff check . - - name: Run fast tests - run: uv run pytest --basetemp .pytest_tmp -m "not full_export" - - pytest-full-export: - runs-on: windows-latest - - steps: - - name: Check out repo - uses: actions/checkout@v4 - - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: "3.12" - - - name: Install uv - uses: astral-sh/setup-uv@v4 - - - name: Sync locked dependencies - run: uv sync --locked --group dev - - - name: Run exhaustive export tests - run: uv run pytest --basetemp .pytest_tmp -m full_export + - name: Run tests + run: uv run pytest --basetemp .pytest_tmp diff --git a/.gitignore b/.gitignore index 9debff0..5bb61a2 100644 --- a/.gitignore +++ b/.gitignore @@ -244,3 +244,10 @@ plans/ /prepared_table_filter_example.ipynb /scratch.ipynb /test +/simor_project_outputs/ + +AGENTS.md + +.agents/ +old_r_based_visualizer/ +outside_summary_tables/ \ No newline at end of file diff --git a/README.md b/README.md index 04b8812..32ab900 100644 --- a/README.md +++ b/README.md @@ -1,619 +1,156 @@ # ActivitySim Visualizer -`activitysim_visualizer` is a Panel-based dashboard for exploring and comparing [ActivitySim](https://activitysim.github.io/) outputs. It can: +ActivitySim Visualizer turns [ActivitySim](https://activitysim.github.io/) +output into an interactive dashboard. Use it to examine one model run, compare +multiple runs, or compare model results with survey data. -- compare multiple model runs side by side -- build and reuse prepared and summary caches -- serve a live local dashboard -- export a standalone HTML version for offline sharing +ActivitySim Visualizer can: + +- prepare and summarize household, person, tour, and trip output; +- optionally add skim-derived trip and tour measures from OMX, HDF5, or CSV inputs; +- compare travel patterns, model choices, and validation measures for multiple runs; +- reuse valid cached results for faster startup; and +- launch a local dashboard or create a standalone HTML file. ## Quick Start -Install dependencies with `uv`: +### 1. Install the project -```bash -uv sync --locked -``` - -Notebook tooling is optional; install it only when working with the repository's -notebooks: +From the repository root, use `uv` to create the environment and install the +locked dependencies: ```bash -uv sync --locked --group notebooks +uv sync --locked ``` -If `uv sync` fails because of a hardlink issue, retry with: +If Windows reports a hardlink problem, use: ```bash uv sync --locked --link-mode=copy ``` -Create a project-specific config: +### 2. Create a configuration -```bash -Copy-Item config.yaml local_config.yaml -``` - -Edit `local_config.yaml`, then run the app with that config: - -```bash -uv run activitysim-viz --config local_config.yaml -``` - -By default, `activitysim-viz` follows `pipeline.steps` from the loaded config when no explicit step flags are supplied. The shipped example config defaults to `summarize` + `dashboard`, so a normal run will reuse summary caches when possible, rebuild them when needed, and then start the live dashboard on [http://localhost:5006](http://localhost:5006). - -## Dashboard Pages - -Dashboard pages now use one shared authoring model: - -- page classes use `@dashboard_page(...)` and subclass `DashboardPage` -- dropdowns use `select(...)`; custom widgets use `selector(...)` -- dynamic selectors declare an option provider and default policy -- refreshable regions are registered with `section(...)` -- large pages compose related selectors and sections with `feature(...)` -- repeated chart transforms use `query(...)` without page-authored cache keys -- live refresh and export metadata both derive from those registrations - -The main shared page-helper modules live under `dashboard/helpers/`: - -- `category_helpers.py` -- `geography_helpers.py` -- `person_type_helpers.py` -- `time_distance_helpers.py` -- `comparison_helpers.py` - -If you are adding or refactoring a page, start with the -[dashboard page recipes](wiki/33-dashboard-page-recipes.md) and -[figures/widgets guide](wiki/32-figures-and-widgets.md). The -[dashboard extension cookbook](wiki/45-dashboard-extension-cookbook.md) covers -the complete contributor path. - -## Config Setup - -The repo ships with `config.yaml` as a template. In practice, most people should: - -1. Copy `config.yaml` to `local_config.yaml` or another machine-specific file. -2. Update the `runs` section to point at real ActivitySim output folders. -3. Update `prepare.distance_skim`, `zones`, and `files` if your model layout differs from the defaults. -4. Run with `--config your_file.yaml`. - -The canonical config layout is organized around a few top-level sections: +Copy `config.yaml` to `local_config.yaml`, then set each `runs.dir` value to an +ActivitySim output directory: ```yaml -root: artifacts/ -log_level: INFO - -pipeline: - steps: [summarize, dashboard] - dashboard_mode: live - overwrite: false - -prepare: ... -summarize: ... -segment: ... -dashboard: ... -display: ... -skimjoin: ... -extensions: ... -``` - -Removed keys such as `processor.*`, `summaries.*`, `visualizer.*`, top-level -`dashboard_labels`, and top-level `run_colors` now fail validation and name the -canonical replacement. Unknown keys also fail instead of being silently ignored. - -The minimum useful config is usually: - -```yaml -root: artifacts/summary_cache - -pipeline: - steps: - - summarize - - dashboard - dashboard_mode: live - -runs: - - dir: path\to\run1 - label: Base - - dir: path\to\run2 - label: Build - -skimjoin: - distance_skim: - file: path\to\skims.omx - matrix: SOV_DIST__MD - -zones: - use_maz: false - maz_col: zone_id - taz_col: TAZ - -files: - households: final_households - persons: final_persons - tours: final_tours - trips: final_trips - joint_tour_participants: final_joint_tour_participants - land_use: final_land_use -``` - -If runs use different raw filenames, keep `files:` as the default mapping and -override only the differences inside `runs[*].file_map`: - -```yaml -files: - households: final_households - persons: final_persons - tours: final_tours - trips: final_trips - runs: - - dir: path\to\run1 + - dir: C:\models\base\output label: Base - file_map: - households: final_hh - trips: trip_linked - - - dir: path\to\run2 + - dir: C:\models\build\output label: Build - file_map: - households: household - persons: person - tours: tour - trips: trip -``` - -If a run should skip raw prepare and use externally managed canonical prepared -tables instead, point it at those files with `runs[*].prepared_table_map`: - -```yaml -runs: - - dir: path\to\raw_run - label: Raw Run - - - label: Custom Prepared Run - prepared_table_map: - households: path\to\custom\households.parquet - persons: path\to\custom\persons.csv - tours: path\to\custom\tours.parquet - trips: path\to\custom\trips.csv - joint_tour_participants: path\to\custom\joint_tour_participants.parquet - land_use: path\to\custom\land_use.csv -``` - -`prepared_table_map` is intended for canonical prepared tables that were already -skimjoined and then optionally filtered or otherwise post-processed outside this -repo. When a run uses `prepared_table_map`, the workflow loads those prepared -tables directly and does not rerun raw prepare or integrated skimjoin for that run. - -If a run already has dashboard-ready summary tables, point directly at those -files with `runs[*].summary_table_map`: - -```yaml -runs: - - label: Summary Only Demo - summary_table_map: - population_totals: path\to\summaries\population_totals.csv - traffic_count_comparisons: path\to\summaries\traffic_count_comparisons.parquet ``` -`summary_table_map` uses registered summary IDs as keys, accepts explicit -`.csv` or `.parquet` paths, and resolves relative paths from the config file -directory. Mapped summaries are expected to already use the dashboard's canonical -columns. During summarize they override the listed generated summaries; missing -summaries can still be generated from raw/prepared inputs when those inputs exist. -Some registered summary IDs are external/demo-only and are not generated by -default for raw/prepared runs, which avoids writing `__empty__` cache CSVs just -to make those IDs available to `summary_table_map`. +The default file names are `final_households`, `final_persons`, `final_day`, +`final_tours`, `final_trips`, `final_vehicles`, +`final_joint_tour_participants`, and `final_land_use`. Day, vehicle, joint +participant, and land-use tables are optional. The visualizer accepts CSV and +Parquet input files. -Integrated skim enrichment can now be selected per run without forcing one -shared skimjoin config for every skim structure. Keep the explicit skimjoin -YAML logic in separate files, then choose the file and optional project-input -overrides per run: +For a small example configuration and instructions for nonstandard files or +zones, see [Getting Started](wiki/10-getting-started.md) and +[Configuring Your Data](wiki/11-configuring-your-data.md). -```yaml -skimjoin: - defaults: - config_path: configs/skimjoin_default.yaml - -runs: - - dir: path\to\run_a - label: Run A - skimjoin: - config_path: configs/skimjoin_odot_series15.yaml - skim_files: - - path\to\run_a\skims\*.omx - - path\to\run_a\skims\maz_stop_walk.csv - network_los_file: path\to\run_a\network_los.yaml - - - dir: path\to\run_b - label: Run B - skimjoin: - config_path: configs/skimjoin_combined_walk.yaml - skim_files: - - path\to\run_b\skims\*.omx -``` +For the included Metro, LCOG, and SKATS configurations, use the +[SimOR Quick Start](SIMOR_QUICK_START_GUIDE.md). -Skimjoin override rules: +### 3. Start the visualizer -- `runs[*].skimjoin.config_path` overrides global `skimjoin.config_path`. -- `runs[*].skimjoin.skim_files` overrides the selected skimjoin config's `project.skim_files`. -- `runs[*].skimjoin.network_los_file` overrides the selected skimjoin config's `project.network_los_file`. -- `skimjoin.failure_policy` defaults to `record`; use `error` when skimjoin failures must stop a validation or batch run. -- If a run omits `runs[*].skimjoin`, it uses the global skimjoin settings exactly as before. - -Recommended rule of thumb: - -- If runs differ only by skim file locations, share one skimjoin config and override `runs[*].skimjoin.skim_files`. -- If runs differ only by period definitions, share one skimjoin config and override `runs[*].skimjoin.network_los_file`. -- If runs differ by lookup logic, fallback behavior, combined vs split components, or directional semantics, use different skimjoin config files. - -VOT bin preparation stays in `prepare.vot_bins` and remains run-aware by run label. - -Skimjoin dimensions are now standardized under `dimensions`, while -`activitysim` only carries the structural trip/tour fields. The recommended -integrated-runtime pattern is: - -```yaml -activitysim: - trip_mode_column: trip_mode - trip_id_column: trip_id - tour_mode_column: tour_mode - tour_id_column: tour_id - outbound_column: outbound - -dimensions: - PERIOD: - source_columns: - trip_source_column: depart_hour - outbound_tour_source_column: start_hour - inbound_tour_source_column: first_inbound_trip_depart - values_from_network_los: true - values: - 8: AM - 17: PM - VOT: - source_columns: - trip_source_column: vot_bin - outbound_tour_source_column: vot_bin - inbound_tour_source_column: vot_bin - values: - L: L - M: M - H: H +```bash +uv run activitysim-viz --config local_config.yaml ``` -Period behavior is directional by design: - -- trips use `dimensions.PERIOD.source_columns.trip_source_column` -- outbound tours use `dimensions.PERIOD.source_columns.outbound_tour_source_column` -- inbound tours use `dimensions.PERIOD.source_columns.inbound_tour_source_column` +On the first run, the visualizer prepares the input, builds the required summary +tables, and starts a local server at +[http://localhost:5006](http://localhost:5006). Later runs reuse valid caches. +Press `Ctrl+C` to stop the server. -In the standard prepare workflow, `first_inbound_trip_depart` is derived from -the first inbound trip on each tour before integrated skimjoin runs. +If data is missing or the first run fails, see +[Troubleshooting](wiki/90-troubleshooting.md). -Prepared endpoint columns are also standardized before skimjoin runs: +## How It Works -- prepared trips and tours always include `OTAZ` and `DTAZ` -- when `zones.use_maz: true`, prepare also materializes `o_maz` and `d_maz` -- inbound tour lookups reuse those same column names, while skimjoin swaps - their logical direction in the inbound tour context - -The normal prepare step can also write prepared caches as CSV when needed: - -```yaml -prepare: - output: - file_format: csv - validation: - relationship_checks: warn +```text +ActivitySim outputs + -> prepare canonical tables + -> optionally join skim-derived trip and tour values + -> summarize travel measures + -> display a live dashboard or export standalone HTML ``` -Important path rules: - -- `root` is resolved relative to the config file if you give a relative path. -- The prepared cache is created automatically next to `root` as `prepared_cache/`. -- `runs[*].dir` should point at an ActivitySim output directory. -- `prepare.distance_skim.file` may be absolute, or relative to each run directory. -- File entries under `files` can be bare stems like `final_trips` or explicit filenames like `final_trips.csv`. -- `runs[*].file_map` uses the same filename rules as `files`, but applies only to that run. -- `runs[*].prepared_table_map` must use explicit `.parquet` or `.csv` paths and resolves relative paths from the config file directory. -- `runs[*].summary_table_map` must use registered summary IDs with explicit `.parquet` or `.csv` paths and resolves relative paths from the config file directory. -- `prepare.output.file_format` controls how standard prepared caches are written; supported values are `parquet` and `csv`, with `parquet` as the default. -- `prepare.validation.relationship_checks` controls prepared-table foreign-key validation. Use `warn` to log inconsistencies and continue, `error` to fail the run, or `off` to skip the checks. -- `dashboard.export.output_path`, when relative, is resolved from `root`. - -## Config Reference +The YAML configuration selects the input, workflow steps, output location, and +dashboard mode. The start command stays the same for every workflow. -These are the sections most people need to touch: - -| Section | Purpose | +| Goal | Where to learn more | |---|---| -| `root` | Where summary caches are stored | -| `pipeline` | Default workflow steps, dashboard mode, and overwrite behavior | -| `runs` | Run directories, display labels, and optional per-run skim, raw file-map, custom prepared-table map, custom summary-table map, and weight overrides | -| `prepare.distance_skim` | Default distance skim file and matrix name used by summaries | -| `zones` | MAZ/TAZ settings for skim joins and zone normalization | -| `files` | Default ActivitySim output file stems or filenames used unless a run overrides them | -| `columns` | Column aliases when outputs use non-default names | -| `prepare.output.file_format` | On-disk format for prepared caches written by the normal prepare workflow | -| `prepare.validation.relationship_checks` | Whether cross-table prepared-key validation is disabled, warns, or errors | -| `prepare.student_types` | Optional school/university enrollment definitions for shadow pricing pages | -| `dashboard.title` | Title used in the live dashboard and HTML export | -| `dashboard.include_notes` | Show per-plot and per-table calculation notes in the live dashboard and HTML export (default: `true`) | -| `dashboard.live.pages` | Ordered list of live pages/groups to show | -| `dashboard.export` | Export-only output path, page selection, and selector-state controls | -| `display.run_colors` | Plot colors by run | -| `display.labels` | Presentation-only labels and ordering for dashboard/export | -| `weighting.modes` | Named weighting alternatives backed by household, person, and/or trip columns | -| `extensions` | Advanced importable weighting calculations and their summary-affecting settings | -| `summarize.weighting_modes` | Ordered built-in, declarative, or custom weighting-mode IDs to build | -| `summarize.failure_policy` | `record` keeps failed summaries visible as diagnostics; `error` stops immediately on a builder exception | -| `summarize.geography` | Optional configured district/county/zone mappings | -| `summarize.pnr_tour_modes` | Which tour modes count as park-and-ride in summary builders | -| `summarize.group_*_tour_purposes` | Summary-time purpose regrouping switches | -| `summarize.category_normalization` | Summary-affecting category normalization/regrouping | -| `modes` | Optional mode ordering and grouped mode display | -| `display.labels.person_type` | Optional display labels for `ptype` values | - -Weighting rules: - -- If a run sets `hh_weight_col`, `person_weight_col`, or `trip_weight_col`, those are used. -- Otherwise, if a `sample_rate` column is available, weights are derived from it. -- Otherwise, weights default to `1`. -- `weighting.modes` can select additional prepared household, person, and trip columns as named alternatives without replacing the primary `weighted` mode. - -Geography summary notes: - -- Summaries may emit `all_geographies` total rows independently of the geography config. -- Native prepared geographies such as `home_taz`, `home_county`, and `home_mpo` may appear whenever those columns are available in prepared data, even when `summarize.geography.enabled: false`. -- `summarize.geography` controls additional mapped geography aggregations, such as `home_geo__school_district`, `work_geo__county`, or `land_use_geo__district`. - -Removed config notes: - -- Prefer the canonical top-level schema: `root`, `pipeline`, `dashboard`, `display`, `summarize`, `segment`, and `skimjoin`. -- Older keys such as `processor.root`, `summaries.weighting_modes`, `visualizer.dashboard_pages`, top-level `run_colors`, top-level `summary_categories`, and top-level `student_types` are rejected with their canonical replacement. - -Geography note: - -- `summarize.geography.enabled: false` disables mapped geography aggregation columns. Set it to `true` for aggregation-based geography summaries. - -Category config note: - -- Use `summarize.category_normalization` when a mapping changes summary values, grouping membership, or canonical category values. -- Use `display.labels` when a change is cosmetic and should only affect dashboard/export labels or ordering. - -## Live Pages And Export Pages - -`dashboard.live.pages` controls the live dashboard only. `dashboard.export` controls what goes into the standalone HTML export. - -Current top-level page ids are: - -- `overview` -- `long_term_choices` -- `daily_travel` -- `joint_travel` -- `tour_summaries` -- `trip_summaries` -- `validation` -- `raw_trip_demo` - -Grouped page ids support either the whole group or specific child pages. For example: +| Use raw ActivitySim output directories | [Configuring Your Data](wiki/11-configuring-your-data.md#raw-activitysim-output) | +| Use already-prepared tables | [Already-Prepared Tables](wiki/11-configuring-your-data.md#already-prepared-tables) | +| Add skim-derived trip and tour fields | [Skimjoin](wiki/22-skimjoin.md) | +| Use dashboard-ready summary tables | [Dashboard-Ready Summary Tables](wiki/11-configuring-your-data.md#dashboard-ready-summary-tables) | +| Run only the processor | [Processor-Only Workflow](wiki/12-running-workflows.md#configure-a-processor-only-workflow) | +| Create a standalone HTML dashboard | [HTML Export](wiki/34-html-export.md) | +| Understand caches and workflow steps | [Running Workflows](wiki/12-running-workflows.md) | +| Build summaries for configured subsets | [Segmentation](wiki/24-segmentation.md) | +| Add district, county, or other zone groupings | [Geography](wiki/27-geography.md) | +| Find an exact configuration field | [Configuration Reference](wiki/13-configuration-reference.md) | +| Verify raw or prepared table requirements | [Input Data Contract](wiki/14-input-data-contract.md) | +| Interpret cache manifests and rebuild decisions | [Cache And Manifest Reference](wiki/15-cache-manifest-reference.md) | +| Understand a summary table or field | [Summary Catalog](wiki/26-summary-catalog.md) | -```yaml -dashboard: - # Set to false to omit all per-plot and per-table calculation notes. - include_notes: true - live: - pages: - - overview - - long_term_choices: - - individual_choices - - mandatory_location_choice - - shadow_pricing - - daily_travel: default - - tour_summaries: all - - trip_summaries: - - trip_mode - - trip_stop_time -``` - -Notes: - -- `default` means "the group's default enabled children". -- `all` means every child page in the group. -- A plain group id like `tour_summaries` behaves like the group's default selection. -- `raw_trip_demo` is disabled by default and requests prepared trip tables, so keep it out unless you explicitly want that behavior. - -For HTML export, start with the live page set and override selector states or -parts as needed: - -```yaml -dashboard: - export: - dashboard: - weighting: [unweighted] - values: [percent] - exclude_groups: [validation] - pages: - long_term_choices: - shadow_pricing: - geography_level: [all] - student_type: [all] - parts: - workplace_table: - enabled: false - school_table: - enabled: false -``` - -Rules worth remembering: - -- If `dashboard.live.pages` is omitted, the app uses its built-in default page set. -- Export always starts from the live page set. Entries under - `dashboard.export.pages` modify matching pages; they are not an allow-list. -- Export selector requests accept `default`, `all`, or a list of explicit values. -- Set a page override's `enabled` to `false`, or use - `dashboard.export.exclude_pages` / `exclude_groups`, to remove pages from - export without changing the live dashboard. - -## Run Modes - -The CLI exposes three workflow steps: - -1. `prepare` -2. `summarize` -3. `dashboard` - -Common commands: +## Documentation -| Command | What it does | -|---|---| -| `python run.py --config local_config.yaml` | Reuse or build summaries, then start the live dashboard | -| `python run.py --config local_config.yaml --prepare-only` | Build prepared caches and exit | -| `python run.py --config local_config.yaml --summarize` | Reuse or build summary caches and exit | -| `python run.py --config local_config.yaml --summarize --dashboard` | Explicit form of the default live workflow | -| `python run.py --config local_config.yaml --dashboard` | Start the dashboard from existing summary caches for the configured runs | -| `python run.py --config local_config.yaml --prepare --summarize --dashboard` | Force the full prepare -> summarize -> dashboard chain in one run | -| `python run.py --config local_config.yaml --from-csvs` | Start the dashboard from existing summary caches only | -| `python run.py --config local_config.yaml --from-csvs --export-html output.html` | Build a standalone HTML export from existing summary caches | -| `python run.py --config local_config.yaml --summarize --write-csvs` | Rebuild summaries and write fresh cache files | -| `python run.py --config local_config.yaml --summarize --skip-summary-cache-write` | Build summaries for this run without writing cache updates | -| `python run.py --config local_config.yaml --summarize --refresh-summary-cache` | Delete and rebuild summary caches for the selected runs | -| `python run.py --config local_config.yaml --summarize --refresh-prepared-cache` | Rebuild summaries from freshly prepared tables instead of prepared-cache hits | -| `python run.py --config local_config.yaml --prepare --summarize --refresh-caches` | Delete and rebuild both prepared and summary caches for the selected runs | - -Behavior details: - -- `--from-csvs` is cache-only: it reads visualizer summary-cache directories with manifests, not loose summary CSVs. -- `--from-csvs path\to\cache1 path\to\cache2` lets you point directly at specific summary cache directories. -- Use `runs[*].summary_table_map` when you have loose dashboard-ready summary files instead of visualizer cache directories. -- `--dashboard` by itself is valid when summary caches already exist for the configured runs. -- During summarize, the app will reuse prepared cache when possible and rebuild from raw outputs only when needed. -- `--refresh-prepared-cache` deletes the selected runs' prepared-cache directories first, then disables prepared-cache reuse for that invocation. -- `--refresh-summary-cache` deletes the selected runs' summary-cache directories first, then disables summary-cache reuse for that invocation. -- `--refresh-caches` is shorthand for both refresh flags together. - -## Cache Layout - -Prepared caches are written automatically next to the summary cache root: +The [wiki home](wiki/00-home.md) is the main documentation index. -```text -/ - prepared_cache/ - / - manifest.json - households.parquet|csv - persons.parquet|csv - tours.parquet|csv - trips.parquet|csv - joint_tour_participants.parquet|csv - land_use.parquet|csv -``` +For a standard setup, read these chapters in order: -Summary caches are written under `root`: +1. [Getting Started](wiki/10-getting-started.md) +2. [Configuring Your Data](wiki/11-configuring-your-data.md) +3. [Running Workflows](wiki/12-running-workflows.md) -```text -/ - / - manifest.json - weighted/ - unweighted/ -``` +Other user references: -Both cache layers validate manifests before reuse. Cache invalidation is driven by: +- [Output Visualizer](wiki/30-output-visualizer.md) explains the dashboard. +- [Dashboard User Guide](wiki/16-dashboard-user-guide.md) lists the available + analyses and explains how to interpret them. +- [Posit Connect Cloud](wiki/17-posit-connect-cloud.md) explains how to publish + a standalone dashboard with the free public plan. +- [Input Data Contract](wiki/14-input-data-contract.md) defines source and canonical table boundaries. +- [Cache And Manifest Reference](wiki/15-cache-manifest-reference.md) explains stored identities and diagnostics. +- [HTML Export](wiki/34-html-export.md) explains how to create an offline file. +- [Summary Catalog](wiki/26-summary-catalog.md) documents every summary table. +- [Segmentation](wiki/24-segmentation.md) explains subset summaries and their caches. +- [Geography](wiki/27-geography.md) explains zone mappings and spatial outputs. +- [Glossary](wiki/99-glossary.md) defines project terminology. +- [Troubleshooting](wiki/90-troubleshooting.md) covers common failures. -- the run inputs -- the prepare and summary config digests -- the prepared-manifest identity used to build summary caches -- per-summary summary digests inside the summary-cache manifest +## For Contributors -That means presentation-only config changes usually do not force summary rebuilds, and adding a newly requested summary can backfill just that table instead of rebuilding the entire summary bundle. +Start with [Architecture](wiki/01-architecture.md) and +[Developer Workflows](wiki/40-developer-workflows.md), then use the guide for +your task: -## CLI Overrides +- [extending prepared data](wiki/41-data-extension-cookbook.md); +- [adding a summary function](wiki/44-summary-function-cookbook.md); +- [adding dashboard pages, figures, or widgets](wiki/45-dashboard-extension-cookbook.md); +- [changing configuration, columns, or labels](wiki/42-config-column-label-cookbook.md); +- [skim enrichment](wiki/22-skimjoin.md); and +- [testing](wiki/46-testing.md). -You can override runs on the command line instead of putting them in the config: +Run focused tests during development. To run the full test suite, use: ```bash -python run.py --config local_config.yaml ^ - --run C:\path\to\run1 "Base" ^ - --run C:\path\to\run2 "Build" +uv run pytest --basetemp .pytest_tmp ``` -Optional per-run skim overrides can be supplied in the same order: +If you change summary declarations or dashboard page definitions, regenerate +the wiki catalogs from the code: ```bash -python run.py --config local_config.yaml ^ - --run C:\path\to\run1 "Base" ^ - --run C:\path\to\run2 "Build" ^ - --run-skim C:\path\to\base_skims.omx C:\path\to\build_skims.omx +uv run python scripts/generate_wiki_catalogs.py ``` -Use `null`, `None`, or an empty string in `--run-skim` to fall back to the configured `prepare.distance_skim.file`. - -## Codebase Map - -```text -activitysim_visualizer/ -|-- run.py -|-- runtime/ -| |-- workflows/ -|-- runtime/ -| `-- config/ -|-- processor/ -| |-- prepare/ -| |-- summarize/ -| `-- models.py -|-- dashboard/ -| |-- app.py -| |-- export/ -| |-- page_base.py -| |-- page_declarations.py -| |-- page_diagnostics.py -| |-- page_features.py -| |-- page_lifecycle.py -| |-- page_navigation.py -| |-- page_definitions.py -| |-- page_registry.py -| |-- state.py -| `-- pages/ -`-- tests/ -``` - -## Documentation +## License -The main user and contributor documentation lives in the -[`wiki/`](wiki/00-home.md) chapter set. Start with: - -- [Getting Started](wiki/10-getting-started.md) -- [Architecture](wiki/01-architecture.md) -- [Configuration Reference](wiki/13-configuration-reference.md) -- [Output Processor](wiki/20-output-processor.md) -- [Output Visualizer](wiki/30-output-visualizer.md) -- [Developer Workflows](wiki/40-developer-workflows.md) -- [Data Extension Cookbook](wiki/41-data-extension-cookbook.md) -- [Config, Columns, and Labels](wiki/42-config-column-label-cookbook.md) -- [Weighting and Hosting Extensions](wiki/43-weighting-hosting-extensions.md) -- [Summary Function Cookbook](wiki/44-summary-function-cookbook.md) -- [Dashboard Extension Cookbook](wiki/45-dashboard-extension-cookbook.md) -- [Testing](wiki/46-testing.md) -- [Troubleshooting](wiki/90-troubleshooting.md) - -The wiki is the sole documentation source. Add or revise a wiki chapter instead -of creating a parallel documentation tree. - -## Documentation Maintenance Checklist - -When behavior changes, update docs in the same change: - -- New config key or config behavior: update chapters 11 and 13. -- New summary declaration or contract: update chapter 23 and regenerate catalogs. -- New page, selector, or plotting behavior: update chapters 31 through 33 and regenerate catalogs. -- New export payload/runtime behavior: update chapter 34. -- Architecture or runtime-flow changes: update chapters 12, 20, and 30 as applicable. - -## Tests - -See [Developer Workflows](wiki/40-developer-workflows.md) for the normal test -loop and [Testing](wiki/46-testing.md) for the fast/full split and -offline-export boundary. +The GNU General Public License v3.0 applies to this project. See +[`LICENSE.txt`](LICENSE.txt). diff --git a/SIMOR_QUICK_START_GUIDE.md b/SIMOR_QUICK_START_GUIDE.md new file mode 100644 index 0000000..513169b --- /dev/null +++ b/SIMOR_QUICK_START_GUIDE.md @@ -0,0 +1,184 @@ +# SimOR Quick Start + +This guide explains how to use the included Metro, LCOG, and SKATS +configurations to create three area dashboards and one combined comparison +dashboard. It covers only the paths and file names normally needed on a new +computer. For other changes, use the [main documentation](wiki/00-home.md). + +## 1. Install the visualizer + +Open a terminal in the repository root and install the locked dependencies: + +```bash +uv sync --locked +``` + +If Windows reports a hardlink problem, use: + +```bash +uv sync --locked --link-mode=copy +``` + +## 2. Update the three area configs + +The area configs are: + +- [Metro](simor_configs/metro_configs/metro_config.yaml) +- [LCOG](simor_configs/lcog_configs/lcog_config.yaml) +- [SKATS](simor_configs/skats_configs/skats_config.yaml) + +Make the following changes in each config that you plan to run. The SimOR +runner runs all three areas, so all three configs must be valid when using that +runner. + +### Output locations + +`root` is where the area config writes prepared tables, summary tables, and +other working files. A relative path starts from the directory containing the +config. The included value writes to `simor_project_outputs/` in this +repository, so it can usually be left unchanged. + +`dashboard.export.output_path` is the exported HTML file. A relative export +path starts from `root`. The included configs place all exports in +`simor_project_outputs/exports/`. + +If you change an area's `root`, set `dashboard.export.output_path` relative to +the new root (or use an absolute path). Also update that area's paths under +`runs[*].prepared_table_map` in the +[comparison config](simor_configs/comparison_configs/estimation_mode_outputs_comparison.yaml). +The comparison reads the prepared tables created by the area dashboards. + +### Input directories and file names + +Set every `runs[*].dir` to the directory that contains that run's ActivitySim +output files. Each included area config expects four runs: + +1. Raw Statewide Survey +2. Regional: Filtered & Attributed +3. Estimation Mode Inputs +4. Estimation Mode Outputs + +The `files` section defines the default table names for the config. +`runs[*].file_map` overrides those names for one run. Update a `file_map` entry +only when the files on your computer use a different name. A name without an +extension accepts either CSV or Parquet; an explicit `.csv` or `.parquet` +extension requires that format. + +Set `fallback_files.land_use` to a land-use CSV or Parquet file that can be +shared by runs that do not contain their own land-use table. If every run has +the correct land-use file, this fallback can be removed. + +For more detail about input files, see +[Configuring Your Data](wiki/11-configuring-your-data.md). + +### Distance and time-period files + +Set these paths in the `prepare` section: + +- `prepare.distance_skim.file`: the motorized distance OMX used when skimjoin + is skipped. Confirm that `prepare.distance_skim.matrix` names the correct + distance matrix. +- `prepare.non_motorized_distance_skim.file`: the walk/non-motorized distance + CSV or OMX. For an OMX file, also set its `matrix`. +- `prepare.time_periods.network_los_file`: the ActivitySim `network_los.yaml` + used to convert period numbers to time-period labels. + +Also set `skimjoin.defaults.network_los_file` to that valid +`network_los.yaml` path. The config loader validates the configured skimjoin +settings before applying the runner's `--no-skimjoin` workflow selection, even +though that run does not read the skim files. + +These files provide distance, VMT, and time-period values for the exported +dashboard when you run with `--no-skimjoin`. + +### Geographic aggregations + +Choose one setup before running: + +- To include the configured district, county, city, or other aggregations, + leave `summarize.geography.enabled: true` and update every + `summarize.geography.aggregations.*.file` path to the area's land-use file. +- If you do not need those aggregations, set + `summarize.geography.enabled: false`. No aggregation file paths are then + required. + +## 3. Check the comparison config + +The +[comparison config](simor_configs/comparison_configs/estimation_mode_outputs_comparison.yaml) +uses the prepared `Estimation Mode Outputs` from the three area configs. +Its included `prepared_table_map` paths work with the included area `root` +values. Update the maps only if you changed those roots, output format, or the +`Estimation Mode Outputs` run label. + +Also set `prepare.time_periods.network_los_file` in the comparison config to a +valid `network_los.yaml` path. The comparison inputs are already prepared, but +the config loader still validates this configured file. + +## 4. Export the dashboards without skimjoin + +From the repository root, run: + +```bash +uv run python scripts/run_simor_scenarios.py --no-skimjoin +``` + +This command does not require changing or commenting out `pipeline.steps`. It +prepares, summarizes, and exports Metro, LCOG, and SKATS with up to two area +jobs running at once. After all three succeed, it exports the comparison +dashboard. + +With the included output settings, the finished files are: + +```text +simor_project_outputs/exports/metro_dashboard.html +simor_project_outputs/exports/lcog_dashboard.html +simor_project_outputs/exports/skats_dashboard.html +simor_project_outputs/exports/estimation_mode_outputs_comparison.html +``` + +The runner prints `PASS` or `FAIL` as each dashboard finishes. Detailed runtime +and console logs are written to a timestamped directory under +`simor_project_outputs/logs/scenario_runner/`. + +## Run with skimjoin + +Skimjoin adds the skim-derived fields and skim summary pages. Before using it, +set these values in each area config: + +- `skimjoin.defaults.skim_files`: the area's OMX and supporting CSV files; +- `skimjoin.defaults.network_los_file`: the area's `network_los.yaml`; and +- `runs[*].skimjoin.config_path`: only when that run needs rules different from + the area's default rules. The included regional survey runs already select + the alternate-ID rules files. + +Then run the same runner without `--no-skimjoin`: + +```bash +uv run python scripts/run_simor_scenarios.py +``` + +The area-specific skimjoin rule files are next to each area config. Editing +their lookup rules is outside this quick start; see +[Skimjoin](wiki/22-skimjoin.md) and the +[Skimjoin Config Reference](wiki/23-skimjoin-config-reference.md). + +## Add another SimOR area + +Adding an area is a short manual registration; the runner does not discover +area configs automatically. + +1. Copy the closest existing area config directory and update its main config, + logo, output paths, and, if needed, skimjoin rule files. +2. Add the new area name and main config path to `AREA_CONFIGS` in + [`scripts/run_simor_scenarios.py`](scripts/run_simor_scenarios.py). +3. To include the area in the combined dashboard, copy an existing run block in + the comparison config, point its `prepared_table_map` to the new area's + `Estimation Mode Outputs` prepared tables, and add a matching + `display.run_colors` entry. +4. Run the command above. The runner will build the new area before starting + the comparison. + +For changes to tables, columns, summaries, or dashboard pages, use the +[configuration reference](wiki/13-configuration-reference.md) rather than +expanding this quick-start setup. diff --git a/config.yaml b/config.yaml index ede454f..b94b0a0 100644 --- a/config.yaml +++ b/config.yaml @@ -1,42 +1,25 @@ -# ActivitySim Visualizer Configuration -# Canonical config example using the current schema. +# ActivitySim Visualizer starter configuration +# +# Copy this file to local_config.yaml, then: +# 1. Replace the two run directories below. +# 2. Confirm the zone settings match your model. +# 3. Run: uv run activitysim-viz --config local_config.yaml +# +# Standard ActivitySim final_* CSV or Parquet files work without other changes. +# For other input types or advanced options, start at wiki/00-home.md. -name: "Example ActivitySim Visualizer" +name: ActivitySim Run Comparison root: artifacts -log_level: INFO +# Summarize prepares raw inputs automatically when no valid prepared cache +# exists, then the dashboard opens at http://localhost:5006. pipeline: - steps: - - summarize - - dashboard - # Add `prepare`, `skimjoin`, or `segment` when those explicit stages are - # needed. Summarize automatically prepares data when no valid cache exists. - dashboard_mode: live # none | live | export | host - overwrite: false - -# Optional named alternatives backed by columns retained in prepared tables. -# weighting: -# modes: -# calibrated: -# label: Calibrated -# columns: -# households: calibrated_hh_weight -# persons: calibrated_person_weight -# trips: calibrated_trip_weight + steps: [summarize, dashboard] + dashboard_mode: live + refresh: [] -# Advanced trusted calculations only. Each importable module defines -# register_weighting_modes(registry); settings are available on Config and enter -# summary cache identity. -# extensions: -# modules: [my_project.weighting] -# settings: -# calibrated: -# multiplier: 1.0 - -# --------------------------------------------------------------------------- -# ActivitySim output file names -# Use stems (no extension) for automatic format detection. -# --------------------------------------------------------------------------- +# These are the default ActivitySim file stems. Edit only names that differ in +# your model outputs; the visualizer accepts either .csv or .parquet files. files: households: final_households persons: final_persons @@ -45,261 +28,22 @@ files: joint_tour_participants: final_joint_tour_participants land_use: final_land_use -# Optional shared fallback files for optional inputs that may be missing in -# some run folders. These must be explicit .csv or .parquet paths. -# fallback_files: -# land_use: C:\path\to\shared\land_use.csv - -# --------------------------------------------------------------------------- -# Runs to compare -# `runs[*].skimjoin` overrides the global `skimjoin.defaults` settings. -# `prepared_table_map` entries must be explicit .csv or .parquet paths. -# -# Use `file_map` when using non-standard input filenames -# -# use `prepared_table_map` if you have prepared tables you want to use other -# than the cached prepared tables. For example, you could run the `prepare` -# step of the pipeline, perform your own filtering operations on the -# prepared tables, then read your filtered prepared tables into the -# `summarize` and `dashboard` steps. -# --------------------------------------------------------------------------- +# Required: replace these example directories. Add or remove runs as needed. runs: - - dir: path\to\activitysim\output\run1 + - dir: C:\path\to\base\output label: Base - # skimjoin: - # config_path: example_skimjoin_config.yaml - # skim_files: - # - C:\path\to\model_skims\*.omx - # - C:\path\to\model_skims\maz_stop_walk.csv - # network_los_file: C:\path\to\model_skims\network_los.yaml - - - dir: path\to\activitysim\output\run2 + - dir: C:\path\to\build\output label: Build - # file_map: - # households: household - # persons: person - # tours: tour - # trips: trip_linked - # joint_tour_participants: joint_tour_participants - # land_use: land_use - # prepared_table_map: - # households: path\to\prepared\households.parquet - # persons: path\to\prepared\persons.parquet - # tours: path\to\prepared\tours.parquet - # trips: path\to\prepared\trips.parquet - # joint_tour_participants: path\to\prepared\joint_tour_participants.parquet - # land_use: path\to\prepared\land_use.parquet -# --------------------------------------------------------------------------- -# Zone system -# Set use_maz: false for TAZ-only models. -# --------------------------------------------------------------------------- +# Confirm these fields before the first run. For a TAZ-only model, keep +# use_maz false. For a MAZ/TAZ model, set it to true and name both columns. zones: use_maz: false maz_col: zone_id taz_col: TAZ -# --------------------------------------------------------------------------- -# Column names in the ActivitySim output files -# Alias-capable fields may be a single string or an ordered list of candidates. -# --------------------------------------------------------------------------- -columns: - ptype: ptype - hhsize: hhsize - auto_ownership: auto_ownership - num_workers: num_workers - num_adults: num_adults - # sample_rate: sample_rate - # household_id: [household_id, hh_id] - # person_id: [person_id, pid] - # tour_id: [tour_id, tid] - # trip_id: [trip_id, tripid] - # tour_purpose: [tour_purpose, primary_purpose, purpose] - # trip_purpose: [trip_purpose, purpose] - # tour_mode: [tour_mode, mode] - # trip_mode: [trip_mode, mode] - - -# --------------------------------------------------------------------------- -# Settings for the `prepare` step. -# --------------------------------------------------------------------------- -prepare: - output: - file_format: parquet # parquet | csv - validation: - relationship_checks: warn # off | warn | error - distance_skim: - file: path\to\skims.omx - matrix: SOV_DIST__MD - # auto_sufficiency_basis: licensed_drivers # licensed_drivers | workers | adults - # vot_bins: - # source_column: income_segment - # output_column: vot_bin - # fallback_value: M - # mappings: - # base: - # 1: L - # 2: M - # 3: H - -# --------------------------------------------------------------------------- -# Settings for the `skimjoin` step, to be used as defaults. -# --------------------------------------------------------------------------- -skimjoin: - failure_policy: record # record | error - create_hypothetical_skim_tables: false - defaults: - config_path: example_skimjoin_config.yaml - skim_files: - - C:\path\to\model_skims\*.omx - - C:\path\to\model_skims\maz_stop_walk.csv - - C:\path\to\model_skims\maz_maz_walk.csv - network_los_file: C:\path\to\model_skims\network_los.yaml - -# --------------------------------------------------------------------------- -# Settings for the `segment` step. -# --------------------------------------------------------------------------- -segment: - dashboard: - segmentation_type: signup_platform - visibility: segments_only # full_only | segments_only | full_and_segments - definitions: - signup_platform: - include_full: true - persist_segmented_prepared_tables: false - allow_overlapping: false - on_empty_segment: warn - source: - type: prepared_column - source_table: hh - column: signup_platform - segments: - - id: rmove - label: RMove - values: ["rmove"] - - id: browser - label: Browser - values: ["browser"] - - id: call - label: Call - values: ["call"] - -# --------------------------------------------------------------------------- -# Settings for the `summarize` step. -# --------------------------------------------------------------------------- summarize: weighting_modes: [weighted, unweighted] - failure_policy: record # record | error - pnr_tour_modes: - - PNR_TRANSIT - # Controls additional mapped geography aggregations only. Summaries may still - # emit all_geographies totals, and native prepared home geographies such as - # home_taz, home_county, and home_mpo can appear when those columns exist. - geography: - enabled: false - # Configured mappings create columns such as home_geo__district, - # work_geo__county, or land_use_geo__district. - # landuse_col: COUNTY - # mapping: - # 1: County 1 - # aggregations: - # district: - # source_zone_system: maz - # file: C:\path\to\land_use.csv - # zone_id_col: MAZ - # geography_col: DISTRICT -# --------------------------------------------------------------------------- -# Settings for the `dashboard` step. -# --------------------------------------------------------------------------- dashboard: - title: "ActivitySim Comparison Visualizer" - include_notes: true - enable_maz_geographies: false - live: - pages: - - overview - - long_term_choices - - daily_travel - - joint_travel - - tour_summaries - - trip_summaries - - validation - export: - output_path: exports/dashboard.html - # `pages` overrides matching live pages; it is not an inclusion list. - # dashboard: - # weighting: [unweighted] - # pages: - # long_term_choices: - # shadow_pricing: - # geography_level: [all] - # student_type: [all] - # parts: - # workplace_table: - # enabled: false - # school_table: - # enabled: false - # Reserved for a future hosted-dashboard implementation. These settings are - # validated but intentionally ignored by the current runtime. - # host: - # account: my-connect-cloud-account - # app_id: 12345 - # title: ActivitySim Comparison Visualizer - # verify: true - -display: - missing_data_display: card # card | blank - # bar_hover_mode: all # closest | all - # density_hover_mode: all # closest | all - labels: - person_type: - mapping: - all_person_types: All Person Types - 1: Full-time worker - 2: Part-time worker - 3: University student - 4: Non-worker adult - 5: Retired - 6: Driving-age student - 7: Non-driving-age student - 8: Preschool - - geography: - mapping: - all_geographies: All Geographies - county: County - - tour_purpose: - mapping: - all_tour_purposes: All Tour Purposes - work: Work - school: School - escort: Escort - shopping: Shopping - othmaint: Other Maintenance - eatout: Eat Out - social: Social - othdiscr: Other Discretionary - atwork: At-Work - joint: Joint - - mode: - mapping: - SOV: Drive Alone - HOV2: Shared Ride 2 - HOV3: Shared Ride 3+ - WALK: Walk - BIKE: Bike - WALK_TRANSIT: Walk-Transit - PNR_TRANSIT: PNR-Transit - KNR_TRANSIT: KNR-Transit - TNC_SINGLE: TNC-Single - TNC_SHARED: TNC-Pool - - run_colors: - - "#298c8c" - - "#a00000" - - "#b8b8b8" - - "#384860" - - "#ff7f0e" + title: ActivitySim Run Comparison diff --git a/dashboard/app.py b/dashboard/app.py index dc998d8..9ad47a8 100644 --- a/dashboard/app.py +++ b/dashboard/app.py @@ -148,9 +148,11 @@ def _on_value_change(event) -> None: template = pn.template.FastListTemplate( title=config.dashboard_title, + logo=config.dashboard_logo or "", sidebar=sidebar_items, main=[main_content], theme="default", + theme_toggle=False, accent_base_color="#4E79A7", header_background="#4E79A7", sidebar_width=340, diff --git a/dashboard/calculation_notes.py b/dashboard/calculation_notes.py index f3fa0a0..167a4c3 100644 --- a/dashboard/calculation_notes.py +++ b/dashboard/calculation_notes.py @@ -93,6 +93,7 @@ class CalculationNote: method_text: str | None = None sources: tuple[str, ...] = () source_filters: tuple[str, ...] = () + column_definitions: tuple[str, ...] = () def _nonempty_text(value: object, *, field: str) -> str: @@ -117,6 +118,7 @@ def _parse_note( "method_text", "sources", "source_filters", + "column_definitions", } unexpected = sorted(set(raw_note) - allowed_fields) if unexpected: @@ -176,6 +178,16 @@ def _parse_note( for item in raw_source_filters ) + raw_column_definitions = raw_note.get("column_definitions", []) + if not isinstance(raw_column_definitions, list): + raise ValueError( + f"Calculation note {note_id!r}.column_definitions must be a list." + ) + column_definitions = tuple( + _nonempty_text(item, field=f"{note_id!r}.column_definitions item") + for item in raw_column_definitions + ) + raw_details = raw_note.get("details", {}) if not isinstance(raw_details, dict): raise ValueError(f"Calculation note {note_id!r}.details must be a mapping.") @@ -205,6 +217,7 @@ def _parse_note( method_text=method_text, sources=sources, source_filters=source_filters, + column_definitions=column_definitions, ) @@ -288,6 +301,16 @@ def render_calculation_note_html(note: CalculationNote) -> str: f"
    {rendered_items}
" "" ) + if note.column_definitions: + rendered_items = "".join( + f"
  • {html.escape(item)}
  • " for item in note.column_definitions + ) + sections.append( + "
    " + "Table columns:" + f"
      {rendered_items}
    " + "
    " + ) if note.sources: rendered_sources = "".join( f"
  • {html.escape(source)}
  • " for source in note.sources diff --git a/dashboard/calculation_notes.yaml b/dashboard/calculation_notes.yaml index da1c174..5e7a7f6 100644 --- a/dashboard/calculation_notes.yaml +++ b/dashboard/calculation_notes.yaml @@ -4,7 +4,7 @@ # method_text is visualization-specific prose shown in the expanded note; # omitting it avoids repeating a generic method already conveyed by the summary. # source_filters records only eligibility/exclusion rules applied while building -# the summary. Legacy detail sections may remain as authoring context but are +# the summary. Extended detail sections may remain as authoring context but are # intentionally not rendered. methods: totals: >- @@ -56,17 +56,19 @@ methods: notes: overview.kpis: method: totals - method_text: Each KPI is summed independently from its prepared run-level population or VMT total for the current run and weighting mode. + method_text: Each KPI is summed independently from its prepared run-level population or VMT total for the current run and weighting mode. Joint tour, trip, and stop records are participant-expanded because these KPIs represent person travel. sources: [population_totals, auto_vmt_totals] summary: KPI cards show run-level population, household, tour, trip, stop, and auto VMT totals. source_filters: - The Stops KPI counts only trip records marked as intermediate stops. + - Participant counts of 995 or greater, missing counts, and nonpositive counts are invalid and fall back to one rather than acting as multipliers. details: Source data: - Person, household, tour, trip, and stop totals come from the population totals summary. - Auto VMT comes from the auto VMT totals summary. Important details: - Weighting follows the dashboard Weighting control. + - People and households are counted from their own tables. Joint tour, trip, and stop records use the sum of known participant person weights when participant attribution is available, with valid party-size expansion only as a fallback. overview.demographics: sources: [person_type_distribution, household_size_distribution] @@ -93,14 +95,17 @@ notes: - Trips with a missing trip mode are excluded and the remaining mode categories are sorted for display. joint_travel.frequency: - method_text: Joint tours are counted within each household to form its joint-tour-frequency pattern, then each household contributes its household weight once to the matching pattern. + method_text: Joint tours are counted within five fixed purpose groups to form each household-day's 21-alternative joint-tour-frequency pattern, then each household-day contributes its household weight once to the matching pattern. sources: [jtf_distribution] summary: Household counts are grouped by joint-tour-frequency pattern. source_filters: - Joint-tour patterns are built only from tours categorized as joint. + - JTF is a one-day household choice. Multi-day diaries use `(household_id, day_num)`; one-day runs use `household_id`. details: Aggregation: - The joint-tour-frequency summary supplies the household count for each pattern. + - Participant-row representations are collapsed to unique joint-tour IDs before daily purpose counts are formed. + - Purpose slots are fixed as shopping, maintenance, eating out, visiting/social, and other discretionary; they do not depend on which values happen to appear in a run. - Percent mode divides each pattern count by all household-pattern counts in the run. Filters: - Hide No Joint Tours removes that category after percentages have been calculated against the full household total. @@ -112,6 +117,7 @@ notes: Aggregation: - Joint-tour counts are grouped independently by household size and party size. - Composition counts are filtered to the selected party size and then grouped by composition category. + - Existing composition values are preserved; missing values are derived from linked participants as adults, children, or mixed. Canonical person types are used when available because some sources encode age as categories; age 18 is the fallback boundary only for year-valued ages. Display: - Missing size categories are completed with zeros so runs use the same category axis. - Percent mode normalizes each displayed distribution within a run. @@ -121,8 +127,8 @@ notes: summary: Participation charts compare household joint-tour patterns and person participation across household sizes. details: Aggregation: - - Household participation is filtered to the selected household size and grouped by joint-tour-frequency category. - - Person participation is grouped by household size using the prepared person participation totals. + - Household participation is filtered to the selected household size and grouped by daily joint-tour-frequency category. + - Multi-day person participation is linked to joint tours by day and grouped in person-day units; one-day runs retain person units. Display: - Percent mode uses the applicable household or person total as the denominator for each run. @@ -179,6 +185,7 @@ notes: Aggregation: - Each chart uses its purpose-specific mandatory location distance summary. - Counts in matching distance bins are summed after applying the geography filter. + - Exact zero distances use the `0` bin; positive distances below one mile use `>0-<1`; remaining bins use whole-mile lower bounds. Filters and display: - The distance-range controls limit the bins shown without changing the underlying summary totals. - Percent mode normalizes the displayed purpose distribution within each run. @@ -207,6 +214,7 @@ notes: - Mandatory average-distance rows are filtered by geography and arranged by tour purpose. Important details: - The first loaded run supplies the base average for each row. + - Work averages exclude workers who work from home and workers assigned to external workplaces. shadow_pricing.workplace_plot: method: aligned_comparison @@ -268,16 +276,22 @@ notes: daily_activity_pattern.summary: method: counts_and_rates - method_text: Activity-pattern and frequency charts total eligible persons by choice. Tour and trip rates instead divide weighted activity totals by the matching weighted person population. + method_text: Activity-pattern and frequency charts total eligible persons by choice. Tour and trip rates divide person-tour or person-trip totals by the matching person-day population using the selected weighted or unweighted mode. sources: [daily_activity_pattern_by_person_type, mandatory_tour_frequency_by_person_type, nonmandatory_tour_frequency_by_person_type, tour_rates_by_person_type_and_tour_purpose, trip_rates_by_person_type_and_trip_purpose] summary: These charts summarize daily activity pattern, mandatory and non-mandatory tour frequency, and daily tour and trip rates for the selected person type. - formula: tour or trip rate = weighted tours or trips for a purpose ÷ weighted person-days + formula: tour or trip rate = person-tours or person-trips for a purpose / person-days, using the selected weighting mode details: Aggregation: - Distribution charts group person counts by the applicable activity or frequency category. - - Rate charts aggregate purpose totals and divide by the matching person-type population. + - Rate charts aggregate purpose totals and divide by the matching eligible person-day population. A nonempty day table defines that exposure universe even when every modeled day has a unique person ID; without a day table, each eligible person contributes one modeled day. + - Every eligible activity and person-day row contributes; repeated source rows are not deduplicated. + - Joint travel is attributed with each known participant's person weight; when participant rows are unavailable, a valid party size expands the joint record. + - A non-null person-table DAP or mandatory-frequency choice is used for that person. Persons without a usable person choice fall back to all eligible day rows, using each day row's weight when available. + - For genuinely multi-day inputs, nonmandatory frequency is classified per person-day when all individual tours and joint participants have safe day keys; otherwise it retains the person-level fallback rather than guessing or dropping rows. + - A configured day-weight column is authoritative. A run whose source day weight is only a placeholder can explicitly disable that column and inherit the corresponding person weight; genuine day-specific weights remain unchanged. Display: - Selecting all person types recomputes totals using the person-type population weights. + - Weighted mode uses prepared expansion weights; unweighted mode sets eligible record weights to one while retaining all rows and participant attribution. - Percent mode applies to distributions; rate charts remain rates per person-day. escorted_tours.static: @@ -302,6 +316,7 @@ notes: Aggregation: - Chauffeur tour counts are filtered by direction and grouped by chauffeur person type. - Tour and trip distances are filtered by direction and summed into distance bins. + - Exact zero distances use the `0` bin, while positive distances below one mile use `>0-<1`. Filters and display: - Distance-range controls limit the bins shown. - Percent mode normalizes each displayed distribution within a run. @@ -311,17 +326,20 @@ notes: summary: Tours are grouped separately by broad tour category and detailed tour purpose. details: Aggregation: - - Tour counts come from the tour-category and tour-purpose distribution summaries. + - Tour counts come from the tour-category and tour-purpose distribution summaries in household-tour units. + - When a survey stores one verified joint-tour row per participant, every row is retained with one divided by the represented party size so the rows jointly contribute one household tour. Display: - Configured category labels and ordering are applied across runs. - Percent mode divides each category by all tours in its distribution. tour_mode.distribution: + method_text: Tours are grouped by purpose, mode, and auto sufficiency and summed as person-tours; joint records use all known participant person weights, with valid household party-size expansion as a fallback. sources: [tour_mode_by_tour_purpose_and_auto_sufficiency] summary: Tour counts are filtered by purpose and grouped by tour mode for each auto-sufficiency segment. details: Aggregation: - Zero-auto, auto-deficient, and auto-sufficient tour counts are taken from separate columns and shown as coordinated views. + - Participant values of 995 or greater, missing values, and nonpositive values are never literal multipliers; when participant attribution is unavailable they fall back to one. Filters and display: - Hide Drive Alone removes the configured drive-alone modes from the displayed categories. - Percent mode normalizes tour modes within each auto-sufficiency segment and run. @@ -338,28 +356,41 @@ notes: tour_time.distributions: method: binned_values + method_text: Each eligible tour contributes in household-tour units to its departure, arrival, and duration bins; a joint tour is not multiplied by party size. sources: [tour_time_of_day_by_tour_purpose] summary: Tour start, end, and duration counts are filtered by tour purpose and displayed in time bins. details: Aggregation: - Departure and arrival tours are grouped by clock-time bin; duration tours are grouped by duration bin. + - Every stored row is retained. Verified participant-row representations of one joint tour contribute fractionally so their combined weight equals one household tour; ordinary one-row joint tours retain their full weight. Display: - Clock-time bins use a 3:00 a.m. day boundary for ordering. - Percent mode normalizes each of the three distributions independently within a run. tour_distance.distribution: method: binned_values - method_text: Tours with a usable distance are restricted to the selected purpose, assigned to a tour-distance bin, and summed with tour weights within each bin. + method_text: Nonmandatory, joint, and at-work tour records with a usable distance are restricted to the selected purpose, assigned to a tour-distance bin, and summed in household-tour units within each bin. sources: [tour_distance_by_tour_purpose] summary: Tours are filtered by purpose and counted in tour-distance bins. source_filters: - Tours without a tour purpose or usable tour distance are excluded. + - Mandatory tours are excluded because this distribution covers nonmandatory, joint, and at-work tours. details: Aggregation: - Weighted or unweighted tour counts are summed within each distance bin. + - Every stored row is retained. Verified participant-row representations of one joint tour contribute fractionally so their combined weight equals one household tour; ordinary one-row joint tours retain their full weight. + - Exact zero distances use the `0` bin; positive distances below one mile use `>0-<1`; remaining bins use whole-mile lower bounds. Filters and display: - Distance-range controls limit the bins shown without recomputing the source summary. - Percent mode divides each displayed bin by the tour total for the selected purpose. + Important details: + - The comparison uses prepared origin-to-primary-destination `SKIMDIST` consistently across runs. + - Home-based tour origins are constructed from household `home_zone_id`; at-work subtour origins are constructed from the parent work-tour destination before TAZ conversion and skimming. + - The comparison fixes the metric to `SKIMDIST` so a run cannot silently switch distance concepts based on which columns are available. + - Reported `distance_miles` and realized full-tour path distance are not substituted because they measure different travel concepts. + - A run without an applicable scenario skim can explicitly map a reported linked-tour distance to `SKIMDIST`; this fallback is never selected merely because another distance column happens to exist. + - Participant-row survey records can disagree on a joint tour's reported distance; the row-preserving fractional calculation displays that observed mixture rather than selecting and retaining only one participant record. + - Prepared caches created before schema version 11 must be refreshed for corrected canonical origins and skim distances. tour_distance.averages: method: rates_and_averages @@ -374,6 +405,7 @@ notes: - Average-distance summaries are filtered to the selected geography level and geography. Important details: - The first loaded run supplies the base average for each tour-purpose row. + - The average uses the same prepared origin-to-primary-destination `SKIMDIST` as the tour-distance histogram. tour_stop_frequency.distributions: sources: [tour_stop_frequency_by_tour_purpose, atwork_subtour_frequency_distribution] @@ -381,6 +413,7 @@ notes: details: Aggregation: - Direction-specific tour counts are filtered by tour purpose and grouped by stop-frequency category. + - Counts use household-tour units; verified participant-row representations are retained with fractional weights that sum to one joint tour. - At-work subtour counts use the dedicated at-work frequency summary. Display: - Percent mode normalizes each distribution independently within a run. @@ -449,25 +482,26 @@ notes: - Percent mode normalizes the trip and stop distributions independently within a run. trip_stop_purpose.trip_purpose: - method_text: Eligible trips are grouped by their trip-purpose category and summed with trip weights. + method_text: Eligible trips are grouped by their trip-purpose category and summed as person-trips, using all known participant person weights for joint records and valid party-size expansion only as a fallback. sources: [trip_purpose_distribution] - summary: This plot sums trip weights by trip purpose. + summary: This plot sums person-trip weights by trip purpose. source_filters: - Trips without a trip purpose are excluded. trip_stop_purpose.stop_purpose: - method_text: Intermediate-stop records are restricted to the selected tour purpose, grouped by the stop destination's trip-purpose category, and summed with trip weights. + method_text: Intermediate-stop records are restricted to the selected tour purpose, grouped by the stop destination's trip-purpose category, and summed as person-stops using participant-attributed weights for joint records. sources: [stop_destination_purpose_by_tour_purpose] summary: This plot sums intermediate-stop weights by destination purpose within tour purpose. source_filters: - Only records marked as intermediate stops and having both a tour purpose and trip purpose are included. trip_mode.distributions: - method_text: Eligible trips are restricted to the selected tour purpose and, for segmented charts, tour mode; trip weights are then summed by trip mode within each requested slice. + method_text: Eligible trips are restricted to the selected tour purpose and, for segmented charts, tour mode; person-trip weights are then summed by trip mode within each requested slice. Joint records use all known participant person weights and fall back to valid party-size expansion only when attribution is unavailable. sources: [trip_mode_by_tour_purpose_and_tour_mode] summary: Trips are filtered by tour purpose and grouped by trip mode, with additional charts available for each tour-mode segment. source_filters: - Trips without a tour purpose, tour mode, or trip mode are excluded. + - Joint trips use all matching participant records when available; non-joint trips contribute once. Sentinel, missing, and nonpositive fallback counts resolve to one rather than acting as literal multipliers. details: Aggregation: - The overall chart sums trip counts across tour modes; segment charts retain the selected tour-mode grouping. @@ -488,15 +522,15 @@ notes: trip_stop_time.trips: method: binned_values - method_text: Eligible trips are restricted to the selected purpose, assigned to the configured bin containing their departure time, and summed with trip weights. + method_text: Eligible trips are restricted to the selected purpose, assigned to the configured bin containing their departure time, and summed as person-trips using participant-attributed weights for joint records. sources: [trip_departure_time_by_purpose] - summary: This plot sums trip weights by departure-time bin. + summary: This plot sums person-trip weights by departure-time bin. source_filters: - Trips must have a purpose and a departure bin within the configured model day. trip_stop_time.stops: method: binned_values - method_text: Intermediate stops are restricted to the selected purpose, assigned to the configured bin containing their departure time, and summed with trip weights. + method_text: Intermediate stops are restricted to the selected purpose, assigned to the configured bin containing their departure time, and summed as person-stops using participant-attributed weights for joint records. sources: [trip_departure_time_by_purpose] summary: This plot sums intermediate-stop weights by departure-time bin. source_filters: @@ -509,21 +543,26 @@ notes: details: Aggregation: - Weighted or unweighted counts are summed within the distance bins supplied by each summary. + - Exact zero distances use the `0` bin; positive distances below one mile use `>0-<1`; remaining bins use whole-mile lower bounds. Filters and display: - Distance-range controls limit the bins shown for both charts. - Percent mode normalizes the trip and stop distributions independently. trip_stop_distance.trip_distance: method: binned_values - method_text: Eligible trips are restricted to the selected tour purpose, assigned to a bin using origin-to-destination trip distance, and summed with trip weights. + method_text: Eligible trips are restricted to the selected tour purpose, assigned to a bin using origin-to-destination trip distance, and summed as person-trips. Joint records use all known participant person weights, with valid party-size expansion only as a fallback. sources: [trip_distance_by_purpose] - summary: This plot bins weighted trips by origin-to-destination distance. + summary: This plot bins weighted person-trips by origin-to-destination distance. source_filters: - Trips without a tour purpose or usable trip distance are excluded. + - Participant values of 995 or greater, missing values, and nonpositive values fall back to one; they are never used as literal multipliers. + details: + Important details: + - Runs with an explicitly configured observed linked-trip distance use that value; runs with an applicable network distance use the prepared origin-to-destination distance. trip_stop_distance.out_of_direction: method: binned_values - method_text: Intermediate stops are restricted to the selected tour purpose, assigned to a bin using their prepared out-of-direction distance, and summed with trip weights. + method_text: Intermediate stops are restricted to the selected tour purpose, assigned to a bin using their prepared out-of-direction distance, and summed as person-stops using participant-attributed weights for joint records. sources: [stop_out_of_direction_distance_by_tour_purpose] summary: This plot bins weighted intermediate stops by out-of-direction distance. source_filters: @@ -558,14 +597,31 @@ notes: trip_skims.summary_table: method: descriptive_statistics - method_text: For each skim component and mode, the summary scans the prepared values once to calculate valid count, missing and zero shares, minimum, maximum, mean, median, and standard deviation. + method_text: For each skim component and mode, the summary calculates valid count, missing and zero shares, minimum, maximum, mean, mean excluding zeros, median, mode, and standard deviation. sources: [skimjoin_trip_component_stats] summary: The table reports descriptive statistics for the selected trip skim family and scenario. + column_definitions: + - "Skim Name: Display name of the selected skim component, including its unit when known." + - "Trip Mode: Mode represented by the row." + - "Total: Total trip weight in weighted mode, or number of trip records in unweighted mode, before missing skim values are removed." + - "Valid: Trip weight or record count with a nonmissing numeric skim value. Zero is a valid value." + - "Mean: Weighted arithmetic mean of valid values, including zeros; in unweighted mode, the ordinary arithmetic mean." + - "Mean Non-Zero: Weighted arithmetic mean of valid values that are not zero; in unweighted mode, the ordinary arithmetic mean of non-zero values. Blank when no valid non-zero values exist." + - "Std Dev: Weighted population standard deviation of valid values; in unweighted mode, the population standard deviation." + - "Min / Max: Smallest and largest valid skim values." + - "Median: Weighted median of valid values; in unweighted mode, the ordinary median." + - "Mode: Skim value with the greatest total weight or record count; ties use the smaller value." + - "Zero Share: Among trips with a nonmissing numeric skim value, the share of their summed trip weight whose value equals zero in weighted mode, or the share of those trip records in unweighted mode." + - "Missing Share: Share of total trip weight or records whose skim value is missing." details: Aggregation: - - Prepared skim statistics provide the valid count, missing share, zero share, minimum, maximum, mean, median, and standard deviation for each component and mode. + - Prepared skim statistics provide the valid count, missing share, zero share, minimum, maximum, mean, mean excluding zeros, median, mode, and standard deviation for each component and mode. Filters: - Family and scenario selectors choose the skim components and whether chosen-mode or all-record values are summarized. + Important details: + - Zero values are included in Mean and excluded only from Mean Non-Zero. + - Walk distance, bike and e-bike distance, and bike logsum use one combined skimjoin output per mode and metric. The OMAZ-DMAZ lookup is attempted first; when it is unavailable, skimjoin falls back to the corresponding OTAZ-DTAZ OMX lookup. + - Missing share for these combined outputs represents records unresolved by both the MAZ lookup and its TAZ fallback. trip_skims.distribution: method: direct_records @@ -582,14 +638,31 @@ notes: tour_skims.summary_table: method: descriptive_statistics - method_text: For each directional skim component and mode, the summary calculates valid count, missing and zero shares, minimum, maximum, mean, median, and standard deviation from prepared tour values. + method_text: For each directional skim component and mode, the summary calculates valid count, missing and zero shares, minimum, maximum, mean, mean excluding zeros, median, mode, and standard deviation from prepared tour values. sources: [skimjoin_tour_component_stats] summary: The table reports descriptive statistics for the selected tour skim family, direction, and scenario. + column_definitions: + - "Skim Name: Display name of the selected directional skim component, including its unit when known." + - "Tour Mode: Mode represented by the row." + - "Total: Total tour weight in weighted mode, or number of tour records in unweighted mode, before missing skim values are removed." + - "Valid: Tour weight or record count with a nonmissing numeric skim value. Zero is a valid value." + - "Mean: Weighted arithmetic mean of valid values, including zeros; in unweighted mode, the ordinary arithmetic mean." + - "Mean Non-Zero: Weighted arithmetic mean of valid values that are not zero; in unweighted mode, the ordinary arithmetic mean of non-zero values. Blank when no valid non-zero values exist." + - "Std Dev: Weighted population standard deviation of valid values; in unweighted mode, the population standard deviation." + - "Min / Max: Smallest and largest valid skim values." + - "Median: Weighted median of valid values; in unweighted mode, the ordinary median." + - "Mode: Skim value with the greatest total weight or record count; ties use the smaller value." + - "Zero Share: Among tours with a nonmissing numeric skim value, the share of their summed tour weight whose value equals zero in weighted mode, or the share of those tour records in unweighted mode." + - "Missing Share: Share of total tour weight or records whose skim value is missing." details: Aggregation: - - Prepared skim statistics provide the valid count, missing share, zero share, minimum, maximum, mean, median, and standard deviation for each component and mode. + - Prepared skim statistics provide the valid count, missing share, zero share, minimum, maximum, mean, mean excluding zeros, median, mode, and standard deviation for each component and mode. Filters: - Family, direction, and scenario selectors choose the displayed summary slice. + Important details: + - Zero values are included in Mean and excluded only from Mean Non-Zero. + - Walk distance, bike and e-bike distance, and bike logsum use one combined skimjoin output per mode and metric. The directional MAZ lookup is attempted first; when it is unavailable, skimjoin falls back to the corresponding directional TAZ OMX lookup. + - Missing share for these combined outputs represents records unresolved by both the MAZ lookup and its TAZ fallback. tour_skims.distributions: method: direct_records @@ -607,8 +680,9 @@ notes: regional_validation.flows: method: aligned_comparison method_text: Duplicate origin-destination pairs are summed within the observed and modeled sources, then the two matrices are joined by origin and destination before the selected cell-by-cell comparison is calculated. - sources: [county_flows_validation_summary, county_flows_joja_validation_summary, commuting_flows] + sources: [district_commuting_flows_validation_summary, county_commuting_flows_validation_summary, commuting_flows] summary: Observed and modeled origin-destination flow matrices are aligned by geography pair and displayed beside a comparison matrix. + formula: difference = modeled - observed; % difference = (modeled - observed) / observed * 100; absolute % difference = absolute value of % difference source_filters: - Modeled flows include only workers with known home and workplace zones. - Flow rows without both an origin and destination geography are excluded. @@ -618,6 +692,7 @@ notes: - Duplicate origin-destination pairs are summed before the two matrices are joined. Comparison: - The selected metric is computed cell by cell from aligned observed and modeled values. + - Percent differences are blank when the observed flow is zero. - Include Totals adds origin, destination, and grand-total cells to the matrices. transit_validation.boardings: @@ -650,14 +725,20 @@ notes: traffic.facility_summary: method: aligned_comparison - method_text: Daily observed and modeled volumes are paired by count-location ID and grouped by facility type. Location count, percent RMSE, and R-squared are calculated from those paired daily values. + method_text: Daily observed and modeled volumes are paired by count-location ID and grouped by facility type. Location count, RMSE, RMSPE, and R-squared are calculated from those paired daily values. sources: [count_location_counts_validation_summary, count_location_volumes_validation_summary, count_location_scatter_validation_summary, count_location_fit_validation_summary] summary: The table summarizes daily count-location validation statistics by facility type. + formula: >- + % difference = (sum(modeled_i) - sum(observed_i)) / sum(observed_i) * 100; + RMSE = sqrt(sum((modeled_i - observed_i)²) / n); + RMSPE = sqrt(mean(((observed_i - modeled_i) / observed_i)²)) * 100; + R² = 1 - sum((modeled_i - fitted_i)²) / sum((modeled_i - mean(modeled_i))²) details: Aggregation: - Daily observed and modeled volumes are paired by count location and grouped by facility type. - - The table reports location count, percent RMSE, and R-squared using the prepared fit summary when available. + - RMSE and RMSPE are calculated from the paired points; R-squared uses the prepared fit summary when available and otherwise is calculated from the paired points. Important details: + - RMSPE is blank when any observed count in the facility group is zero. - This overview always uses unfiltered daily totals; the controls below apply to other traffic charts. traffic.count_locations: @@ -665,12 +746,19 @@ notes: method_text: After period and facility filters are applied, one observed count and one modeled volume are paired by count-location ID; each successfully paired location becomes one scatter point. sources: [count_location_counts_validation_summary, count_location_volumes_validation_summary, count_location_scatter_validation_summary, count_location_fit_validation_summary] summary: The scatter plot pairs observed traffic counts with modeled count-location volumes for the selected period and facility type. + formula: >- + x = observed count; y = modeled volume; + slope = sum((x_i - mean(x)) * (y_i - mean(y))) / sum((x_i - mean(x))²); + intercept = mean(y) - slope * mean(x); fitted_i = slope * x_i + intercept; + R² = 1 - sum((y_i - fitted_i)²) / sum((y_i - mean(y))²); + one-to-one reference = y = x details: Aggregation: - Observed and modeled values are joined by count-location identifier after applying the selected filters. - A one-to-one reference line is always shown; a fitted regression line is added when fit coefficients are available. Display: - Each point is one validated count location. + - Hovering over a fitted line shows its run, equation, R-squared, and number of locations without reducing the plot area. traffic.link_volume: method: supplied_aggregation @@ -698,16 +786,23 @@ notes: traffic.screenlines: method: aligned_comparison - method_text: Records are first summed by screenline, direction, and count period; observed and modeled totals with the same three-part key are then paired for the scatter plot. + method_text: Records are first summed by screenline, direction, count period, and facility type; observed and modeled totals with matching keys are then paired for the scatter plot and fitted with an ordinary least-squares trendline. sources: [screenline_flow_comparisons] summary: The scatter plot compares observed and modeled screenline flows. + formula: >- + x = observed flow; y = modeled flow; + slope = sum((x_i - mean(x)) * (y_i - mean(y))) / sum((x_i - mean(x))²); + intercept = mean(y) - slope * mean(x); fitted_i = slope * x_i + intercept; + R² = 1 - sum((y_i - fitted_i)²) / sum((y_i - mean(y))²); + one-to-one reference = y = x source_filters: - Screenline records without an identifier, direction, count period, or observed volume are excluded before matching modeled flows. details: Aggregation: - - Flow records are grouped by screenline, direction, and count period before observed and modeled volumes are paired. + - Flow records are grouped by screenline, direction, count period, and facility type before observed and modeled volumes are paired. Display: - - Each point represents one comparable screenline-direction-period record. + - The time-period and facility-type selectors filter the paired records before fitting. + - Hovering over a fitted line shows its run, equation, R-squared, and number of comparable records; the plot also includes a one-to-one reference line. vmt.overview: method: vmt @@ -725,13 +820,13 @@ notes: vmt.personal_auto: method: vmt - method_text: Each eligible auto trip contributes its distance times final trip weight, divided by vehicle occupancy so shared-ride passenger records do not each contribute a full vehicle trip. + method_text: Each eligible auto trip contributes its distance times final trip weight, divided by mode-appropriate vehicle occupancy so shared-ride passenger records do not each contribute a full vehicle trip. sources: [auto_vmt_by_home_geography_income_hhsize_time_period] summary: Personal auto VMT estimates vehicle miles traveled by weighted auto trips and allocates shared-ride travel across vehicle occupants. formula: trip distance × final trip weight ÷ vehicle occupancy source_filters: - When a prepared auto skim distance is available, records without that distance are excluded. - - Otherwise, only configured auto-mode trips with a usable origin-to-destination distance are included. + - Otherwise, configured non-drive-transit auto modes with a usable origin-to-destination distance are included; PNR and KNR are excluded because `od_dist` is the full linked transit-trip distance. details: Included records: - Trips with a usable auto distance and final trip weight. @@ -740,8 +835,14 @@ notes: - Results are grouped by the selected home geography, income segment, household size, time period, or mode. - When geography is the breakdown, the chart shows the 25 geographies with the highest VMT. Important details: - - Vehicle occupancy uses the number of tour participants when available and otherwise defaults to one. - - Daily summary rows are preferred when all time periods are selected. + - SOV, PNR, taxi, and single-party TNC occupancy is 1; HOV2 and KNR occupancy is 2; HOV3 occupancy is 3.33. Other auto modes use a valid participant count or default to 1. + - A household-level joint trip and an identified chauffeur escort event represent one vehicle and therefore use occupancy 1. + - Repeated records sharing a joint-trip identifier are already person-level; every row is retained and its HOV2/HOV3 mode occupancy is applied rather than treating every row as a separate full vehicle. + - Participant values of 995 or greater, missing values, and nonpositive values default to 1 and are never used literally. + - Drive-transit VMT is included only when `skim_auto_distance` supplies the auto access or egress leg; PNR and KNR are excluded from the full linked-trip `od_dist` fallback. + - When the prepared data do not expose a canonical driver-to-escortee relationship, no additional escortee duplicate-suppression adjustment is applied. + - When component time periods exist for a segment, its Daily row is recomputed from only those non-Daily rows; a stale pre-existing Daily row is not added again. Daily-only segments retain their source Daily row. + - A run without scenario zone assignments can use an explicitly configured observed linked-trip distance instead of a network-skim lookup. vmt.non_motorized: method: vmt @@ -758,7 +859,9 @@ notes: - Results are grouped by the selected home geography, income segment, household size, time period, or mode. - When geography is the breakdown, the chart shows the 25 geographies with the highest mileage. Important details: - - The distance source recorded in the prepared summary is used; this may be a trip distance or a configured skim distance. + - The combined walk skimjoin distance uses the MAZ walk lookup first and the TAZ walk OMX distance as its fallback; the configured prepared walk distance is used only if that combined output remains unavailable. + - The combined bicycle and e-bike skimjoin distance uses the MAZ bike lookup first and the purpose-appropriate TAZ bike OMX distance as its fallback; the configured prepared walk distance is used only if that combined output remains unavailable. + - A run without scenario zone assignments can use explicitly configured observed linked-trip distances for walk, bicycle, and e-bike travel. vmt.external_travel: method: supplied_aggregation @@ -819,87 +922,101 @@ notes: summary: This plot sums household weights by household size, combining households of five or more people into 5+. daily_activity.activity_pattern: - method_text: Prepared persons with a coordinated daily activity pattern are grouped by that pattern and summed with person weights. + method_text: Each person with a non-null coordinated daily activity pattern is grouped by that person-table value and person weight. Only persons without a usable person choice fall back to all eligible prepared day rows, using each day row's weight when present and the person weight otherwise. sources: [daily_activity_pattern_by_person_type] - summary: This plot sums person weights by coordinated daily activity pattern. + summary: This plot uses person-table activity patterns rowwise when available and day-table patterns for remaining persons without deduplicating day records. source_filters: - - Persons without a daily activity pattern are excluded. + - Records without a daily activity pattern are excluded. + - Person-table `cdap_activity` is authoritative when present. Otherwise the prepared day-table value mapped from `daily_activity_pattern` is used; the processor does not force these two source classifications to agree. + - The selected source classification is not reclassified from mandatory- or nonmandatory-tour-frequency choices. daily_activity.mandatory_tour_frequency: - method_text: Eligible persons are grouped by their mandatory-tour-frequency choice and summed with person weights; zero or missing choices are not included. + method_text: Each person with a non-null mandatory-tour-frequency choice is grouped by that person-table value and person weight. Only persons without a usable choice fall back to work and school tours counted for every person-day, mapped to the same alternatives, and summed with each day row's weight when present. sources: [mandatory_tour_frequency_by_person_type] - summary: This plot sums person weights by mandatory-tour-frequency choice. + summary: This plot sums person or person-day weights by mandatory-tour-frequency choice, deriving choices from observed mandatory tours when needed. source_filters: - - Only persons with a mandatory-tour-frequency choice greater than zero are included. + - Only person or person-day records with a mandatory-tour-frequency choice greater than zero are included. + - Derived counts outside the modeled alternatives (0 work/1 school, 0 work/2 school, 1 work/0 school, 2 work/0 school, and 1 work/1 school) are excluded. daily_activity.nonmandatory_tour_frequency: - method_text: Non-mandatory tours are counted for each person, that person-level tour count becomes the frequency category, and person weights are summed within the resulting categories. + method_text: Non-mandatory individual tours and joint-tour participations are counted for each person-day in genuinely multi-day data when every record has a safe day key; otherwise the one-day/person fallback is used. The resulting person or person-day weights are summed within frequency categories. sources: [nonmandatory_tour_frequency_by_person_type] - summary: This plot sums person weights by the number of individual non-mandatory tours. + summary: This plot sums person or person-day weights by the number of individual non-mandatory tours and joint-tour participations. source_filters: - - Only tours categorized as non-mandatory contribute to the tour count. + - Individual tours categorized as non-mandatory and joint-tour participant records contribute to the combined frequency. + - A multi-day source with unsafe or ambiguous joint-participant day keys uses the person-level fallback so no unmatched rows are silently discarded. daily_activity.tour_rate: method: rates_and_averages - method_text: Weighted tours are totaled by purpose and person type, then divided by the weighted number of people in that person-type group; combined groups are recalculated from totals. + method_text: Eligible tours excluding at-work subtours contribute person-tour weight. Joint tours are attributed to known participant person types or expanded by valid party size as a fallback, then purpose totals are divided by summed person-day weights. sources: [tour_rates_by_person_type_and_tour_purpose] - summary: This plot divides weighted tours of each purpose by the weighted person population. - formula: tour rate = weighted tours ÷ weighted persons + summary: This plot divides weighted person-tours by weighted person-days without deduplicating eligible tour or day records. + formula: tour rate = sum(eligible person-tour weight) / sum(person-day weight) source_filters: - Tours without a person or tour purpose and persons without a person type or weight are excluded. + - At-work subtours are excluded because the rate represents primary daily tours rather than tours nested within a work tour. + - Participant values of 995 or greater, missing values, and nonpositive values fall back to one. daily_activity.trip_rate: method: rates_and_averages - method_text: Weighted trips are totaled by purpose and person type, then divided by the weighted number of people in that person-type group; combined groups are recalculated from totals. + method_text: Every eligible trip contributes person-trip weight. Joint trips are attributed to known participant person types or expanded by valid party size as a fallback, then purpose totals are divided by summed person-day weights. sources: [trip_rates_by_person_type_and_trip_purpose] - summary: This plot divides weighted trips of each purpose by the weighted person population. - formula: trip rate = weighted trips ÷ weighted persons + summary: This plot divides weighted person-trips by weighted person-days without deduplicating trip or day records. + formula: trip rate = sum(person-trip weight) / sum(person-day weight) source_filters: - Trips without a person or trip purpose and persons without a person type are excluded. + - Participant values of 995 or greater, missing values, and nonpositive values fall back to one. joint_travel.household_size: - method_text: Households appearing on at least one joint tour are identified once, joined to household size, and summed with household weights so multiple joint tours do not duplicate a household. + method_text: Household-days appearing on at least one unique joint tour are identified once, joined to household size, and summed with household weights so multiple joint tours in a day do not duplicate an observation. sources: [joint_tours_by_household_size] - summary: This plot sums the weights of households with at least one joint tour by household size. + summary: This plot sums household-day weights with at least one joint tour by household size; one-day runs contribute one household-day per household. source_filters: - Only tours categorized as joint are used to identify participating households. joint_travel.party_size: - method_text: Participants are counted for each joint tour, party sizes of five or more are recoded to 5+, and tour weights are summed within each party-size category. + method_text: A valid participant count is resolved for each joint tour, party sizes of five or more are recoded to 5+, and household-tour representation weights are summed within each party-size category. sources: [joint_tour_party_size_distribution] summary: This plot sums joint-tour weights by number of participants, combining party sizes of five or more into 5+. source_filters: - Only tours categorized as joint are included. + - Party sizes of 995 or greater, missing values, and nonpositive values are excluded rather than placed in the 5+ bin. + - Verified survey participant rows are all retained at one divided by their represented party size, so together they contribute one household tour; model rows already stored once retain full weight. joint_travel.composition: - method_text: Each eligible joint tour retains its participant count and prepared party-composition category; tour weights are then summed for each party-size and composition combination. + method_text: Each eligible joint-tour row retains its participant count and party-composition category; missing composition is derived from linked participants using canonical person types 1--5 as adults and 6--8 as children, with age 18 as the fallback boundary for year-valued ages, before household-tour representation weights are summed for each party-size and composition combination. sources: [joint_tour_composition_by_party_size] summary: This plot sums joint-tour weights by party composition for the selected party size. source_filters: - Only joint tours with a non-missing composition and participant count are included. + - Party sizes of 995 or greater and nonpositive values are excluded. + - Verified survey participant rows are retained fractionally so the represented rows jointly contribute one household tour. joint_travel.person_participation: - method_text: People participating in one or more joint tours are identified once, joined to household size, and summed with person weights within each household-size category. + method_text: People participating in one or more joint tours are identified for each diary day, joined to household size, and summed with person-day weights within each household-size category. sources: [person_jtp_by_household_size] - summary: This plot compares the weighted number of people participating in one or more joint tours by household size. + summary: This plot compares weighted person-days with one or more joint tours by household size; one-day runs contribute one person-day per person. source_filters: - A participant must have a positive number of joint tours and a known household size. + - Multi-day runs use each prepared day row's weight when available and otherwise inherit the corresponding person weight. joint_travel.household_participation: method: rates_and_averages - method_text: Within each household-size group, weighted households are totaled by joint-tour-count category and divided by the weighted total number of eligible households in that size group. + method_text: Within each household-size group, weighted household-days are totaled by joint-tour-count category and divided by the weighted total number of eligible household-days in that size group. sources: [household_jtp_by_household_size_and_jtf] - summary: This plot shows the share of households in each joint-tour-count category by household size. - formula: household share = weighted households in category ÷ weighted households in household-size group + summary: This plot shows the share of household-days in each joint-tour-count category by household size; one-day runs contribute one household-day per household. + formula: household-day share = weighted household-days in category ÷ weighted household-days in household-size group source_filters: - Only households with at least two people are included. + - Multi-day diaries use `(household_id, day_num)` observations and repeat the household expansion weight once per eligible diary day; they are never pooled into a weekly household choice. + - Participant-row representations are collapsed to unique joint-tour IDs before each household-day is classified as 0, 1, or 2+ joint tours. individual_choices.license: - method_text: Persons age 16 or older with a known license status are grouped by that status and summed with person weights. + method_text: Persons eligible to drive with a known license status are grouped by that status and summed with person weights. sources: [license_holding_status_distribution] summary: This plot sums person weights by driver's-license status. source_filters: - - Only persons age 16 or older with a known license status are included. + - When age is expressed in years, only people age 16 or older are included. When age is categorical, a direct driving-eligibility response is used instead. individual_choices.bicycle_comfort: method_text: Persons with a bicycle-comfort response are grouped by the prepared comfort level and summed with person weights. @@ -979,7 +1096,9 @@ notes: sources: [work_location_distance_distribution_by_geography] summary: This plot bins weighted workers by distance from home to their assigned workplace. source_filters: - - Only workers assigned to a workplace zone greater than zero are included. + - Only workers assigned to an internal workplace zone greater than zero are included. + - Workers who work from home and workers with an external workplace flag or external workplace zone are excluded. + - Workers without a usable nonnegative home-to-work distance are excluded. mandatory_location.university_distance: method: binned_values @@ -987,7 +1106,7 @@ notes: sources: [university_location_distance_distribution_by_geography] summary: This plot bins weighted university students by distance from home to their assigned school. source_filters: - - Only students assigned to a school zone greater than zero and coded as person type 3 are included. + - Only students assigned to a school zone greater than zero, coded as person type 3, and with a usable nonnegative distance are included. mandatory_location.school_distance: method: binned_values @@ -995,7 +1114,7 @@ notes: sources: [school_location_distance_distribution_by_geography] summary: This plot bins weighted school students by distance from home to their assigned school. source_filters: - - Only students assigned to a school zone greater than zero and coded as person type 6 or higher are included. + - Only students assigned to a school zone greater than zero, coded as person type 6 or higher, and with a usable nonnegative distance are included. mandatory_location.worker_status_table: method: rates_and_averages @@ -1069,6 +1188,9 @@ notes: summary: This plot bins weighted chauffeur tours by tour distance for the selected direction. source_filters: - Tours must have a usable tour distance and an explicit escort label for the selected direction. + details: + Important details: + - The `0` bin contains only exact zeros; positive sub-mile distances use `>0-<1`. escorted_tours.trip_distance: method: binned_values @@ -1077,27 +1199,31 @@ notes: summary: This plot bins weighted chauffeur trips by trip distance for the selected direction. source_filters: - Trips must have a usable distance and belong to a tour with an explicit escort label for the selected direction. + details: + Important details: + - The `0` bin contains only exact zeros; positive sub-mile distances use `>0-<1`. tour_purpose.category: - method_text: Tours with a broad category are grouped by that category and summed with tour weights. + method_text: Tours with a broad category are grouped by that category and summed in household-tour units; verified participant-row joint tours are retained fractionally rather than collapsed. sources: [tour_category_distribution] summary: This plot sums tour weights by broad tour category. source_filters: - Tours without a tour category are excluded. tour_purpose.purpose: - method_text: Tours with a detailed purpose are grouped by that purpose and summed with tour weights. + method_text: Tours with a detailed purpose are grouped by that purpose and summed in household-tour units; verified participant-row joint tours are retained fractionally rather than collapsed. sources: [tour_purpose_distribution] summary: This plot sums tour weights by detailed tour purpose. source_filters: - Tours without a tour purpose are excluded. tour_mode.mode: - method_text: Tours are restricted to the purpose and auto-sufficiency segment named in the plot, grouped by tour mode, and summed with tour weights. + method_text: Tours are restricted to the purpose and auto-sufficiency segment named in the plot, grouped by tour mode, and summed as person-tours. Joint records use all known participant person weights, with valid household party-size expansion only as a fallback. sources: [tour_mode_by_tour_purpose_and_auto_sufficiency] summary: This plot sums tour weights by mode for the purpose and auto-sufficiency segment named in the plot title. source_filters: - Tours without a purpose, tour mode, or household auto-sufficiency value are excluded. + - Participant values of 995 or greater, missing values, and nonpositive values fall back to one. tour_mode.vehicle_age: method_text: Tours with a usable allocated-vehicle code are split by occupancy, the vehicle age is decoded from that allocation, and the associated weights are summed by age. @@ -1138,14 +1264,15 @@ notes: tour_time.duration: method: binned_values - method_text: Elapsed duration is calculated from each eligible tour's start and end periods, assigned to a duration bin, and summed with tour weights. + method_text: Duration is calculated from each eligible tour's canonical start and end periods using the inclusive occupied-period convention `end - start + 1`, assigned to a duration bin, and summed with tour weights. sources: [tour_time_of_day_by_tour_purpose] - summary: This plot sums tour weights by elapsed tour-duration bin. + summary: This plot sums tour weights by occupied tour-duration period bin. source_filters: - Tours without a purpose or with a time bin outside the configured model day are excluded. + - Raw `duration_minutes` is preserved as source information but is not substituted for the common occupied-period calculation. tour_stop_frequency.direction: - method_text: Tours are restricted to the selected purpose, grouped by the outbound or inbound stop count named in the plot, and summed with tour weights. + method_text: Tours are restricted to the selected purpose, grouped by the outbound or inbound stop count named in the plot, and summed in household-tour units with row-preserving fractional weights for verified participant-row joint tours. sources: [tour_stop_frequency_by_tour_purpose] summary: This plot sums tour weights by stop count for the direction named in the plot title. source_filters: diff --git a/dashboard/export/assets/export.css b/dashboard/export/assets/export.css index 56b80e8..c36a7d3 100644 --- a/dashboard/export/assets/export.css +++ b/dashboard/export/assets/export.css @@ -31,14 +31,59 @@ body { border-top: 8px solid var(--accent); } +.export-header-top { + display: flex; + align-items: center; + justify-content: space-between; + gap: 16px; +} + +.export-brand { + display: flex; + align-items: center; + gap: 18px; + min-width: 0; +} + +.export-logo { + display: block; + flex: 0 0 auto; + width: auto; + max-width: min(280px, 35vw); + max-height: 72px; + object-fit: contain; +} + .export-header h1 { - margin: 0 0 8px; + margin: 0; font-size: 30px; + min-width: 0; } .export-note { color: #4b5563; - margin: 0; + margin: 8px 0 0; +} + +.rail-toggle { + flex: 0 0 auto; + border: 1px solid var(--line); + background: var(--surface-soft); + color: #334155; + border-radius: 10px; + padding: 8px 12px; + font-weight: 600; + cursor: pointer; +} + +.rail-toggle:hover { + border-color: var(--accent); + color: var(--accent-dark); +} + +.rail-toggle:focus-visible { + outline: 3px solid rgba(78, 121, 167, 0.3); + outline-offset: 2px; } .export-layout { @@ -48,6 +93,14 @@ body { align-items: start; } +.export-layout.rail-collapsed { + grid-template-columns: minmax(0, 1fr); +} + +.export-layout.rail-collapsed .export-rail { + display: none; +} + .export-rail, .export-main { min-width: 0; @@ -163,6 +216,13 @@ body { color: #334155; } +.local-tab-button { + max-width: 260px; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + .page-tab-button.active, .local-tab-button.active { background: var(--accent); @@ -277,13 +337,22 @@ body { font-weight: 700; } -.widget-shell select { +.widget-shell select, +.widget-shell input[type="number"] { padding: 9px 12px; border-radius: 10px; border: 1px solid #cbd5e1; background: var(--surface-soft); } +.widget-shell input[type="number"]:disabled { + color: #94a3b8; + background: #e2e8f0; + border-color: #cbd5e1; + cursor: not-allowed; + opacity: 1; +} + .widget-checkbox { display: inline-flex; align-items: center; @@ -452,6 +521,13 @@ table.export-table thead th { text-align: left; } +.export-table-sort-label { + min-width: 0; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + .export-table-sort-indicator { color: var(--muted); font-size: 12px; @@ -526,6 +602,16 @@ table.export-table thead th { grid-template-columns: 1fr; } + .export-header-top { + align-items: flex-start; + flex-wrap: wrap; + } + + .export-logo { + max-width: min(220px, 45vw); + max-height: 56px; + } + .page-panel { padding: 16px; } diff --git a/dashboard/export/assets/export_runtime.js b/dashboard/export/assets/export_runtime.js index e6ed08a..d8441e3 100644 --- a/dashboard/export/assets/export_runtime.js +++ b/dashboard/export/assets/export_runtime.js @@ -91,6 +91,9 @@ }); button.type = "button"; button.disabled = !!config.disabled; + if (config.title) { + button.title = String(config.title); + } if (!config.disabled && typeof config.onClick === "function") { button.addEventListener("click", config.onClick); } @@ -940,7 +943,13 @@ getTraceFieldLength(trace && trace.x), getTraceFieldLength(trace && trace.y) ); - const traceName = trace && trace.name ? trace.name : "trace_" + String(traceIndex + 1); + const traceName = ( + trace && trace.meta && trace.meta.run_name + ? trace.meta.run_name + : trace && trace.name + ? trace.name + : "trace_" + String(traceIndex + 1) + ); for (let pointIndex = 0; pointIndex < pointCount; pointIndex += 1) { rows.push([ traceName, @@ -1359,6 +1368,7 @@ type: "button", "data-column": column, "aria-sort": "none", + title: (node.column_tooltips || {})[column] || column, }, }, [ el("span", { className: "export-table-sort-label", text: column }), @@ -1395,6 +1405,7 @@ tabRow.appendChild( makeButton({ label: tab.title, + title: tab.full_title || tab.title, active: index === activeIndex, onClick: () => { activeIndex = index; @@ -1430,16 +1441,24 @@ className: "plot-shell", attrs: { "data-plot-pending": "true" }, }); - if (node.height) { + const aspectRatio = Number(node.aspect_ratio); + const preserveAspectRatio = Number.isFinite(aspectRatio) && aspectRatio > 0; + if (preserveAspectRatio) { + plotElement.style.aspectRatio = String(aspectRatio); + } else if (node.height) { plotElement.style.minHeight = String(node.height) + "px"; } const baseFigure = node.figure || { data: [], layout: {} }; + const layout = Object.assign({}, baseFigure.layout || {}, { + autosize: true, + width: null, + }); + if (preserveAspectRatio) { + delete layout.height; + } const figure = { data: baseFigure.data || [], - layout: Object.assign({}, baseFigure.layout || {}, { - autosize: true, - width: null, - }), + layout: layout, }; context.plotManager.registerPlot(plotElement, figure); return plotElement; @@ -1978,8 +1997,53 @@ } function renderShell(context, actions) { + const railCollapsed = !!context.railCollapsed; + const rail = renderRail(context, actions); + rail.id = "export-rail"; + + const main = el("main", { className: "export-main" }, [ + renderPageTabs(context, actions), + renderPagePanel(context, actions), + ]); + const layout = el("div", { + className: "export-layout" + (railCollapsed ? " rail-collapsed" : ""), + }, [rail, main]); + const railToggle = el("button", { + className: "rail-toggle", + text: railCollapsed ? "Show sidebar" : "Hide sidebar", + attrs: { + "aria-controls": "export-rail", + "aria-expanded": String(!railCollapsed), + }, + }); + railToggle.type = "button"; + railToggle.addEventListener("click", () => { + context.railCollapsed = !context.railCollapsed; + layout.classList.toggle("rail-collapsed", context.railCollapsed); + railToggle.textContent = context.railCollapsed ? "Show sidebar" : "Hide sidebar"; + railToggle.setAttribute("aria-expanded", String(!context.railCollapsed)); + context.plotManager.scheduleResize(); + }); + + const brandChildren = []; + if (context.payload.logo) { + brandChildren.push( + el("img", { + className: "export-logo", + attrs: { + src: context.payload.logo, + alt: context.payload.title + " logo", + }, + }) + ); + } + brandChildren.push(el("h1", { text: context.payload.title })); + const headerChildren = [ - el("h1", { text: context.payload.title }), + el("div", { className: "export-header-top" }, [ + el("div", { className: "export-brand" }, brandChildren), + railToggle, + ]), ]; if (context.payload.client_export_note && String(context.payload.client_export_note).trim()) { headerChildren.push( @@ -1990,17 +2054,9 @@ ); } - const main = el("main", { className: "export-main" }, [ - renderPageTabs(context, actions), - renderPagePanel(context, actions), - ]); - return el("div", { className: "export-shell" }, [ el("div", { className: "export-header" }, headerChildren), - el("div", { className: "export-layout" }, [ - renderRail(context, actions), - main, - ]), + layout, ]); } @@ -2039,6 +2095,7 @@ plotManager: config.plotManager, app: config.app, renderedRegions: {}, + railCollapsed: false, }; } diff --git a/dashboard/export/js_runtime/dom.js b/dashboard/export/js_runtime/dom.js index 54ae314..394e475 100644 --- a/dashboard/export/js_runtime/dom.js +++ b/dashboard/export/js_runtime/dom.js @@ -75,6 +75,9 @@ }); button.type = "button"; button.disabled = !!config.disabled; + if (config.title) { + button.title = String(config.title); + } if (!config.disabled && typeof config.onClick === "function") { button.addEventListener("click", config.onClick); } diff --git a/dashboard/export/js_runtime/index.js b/dashboard/export/js_runtime/index.js index f637b2f..98af6f4 100644 --- a/dashboard/export/js_runtime/index.js +++ b/dashboard/export/js_runtime/index.js @@ -13,6 +13,7 @@ plotManager: config.plotManager, app: config.app, renderedRegions: {}, + railCollapsed: false, }; } diff --git a/dashboard/export/js_runtime/plotly_lifecycle.js b/dashboard/export/js_runtime/plotly_lifecycle.js index b69d6f3..5283c91 100644 --- a/dashboard/export/js_runtime/plotly_lifecycle.js +++ b/dashboard/export/js_runtime/plotly_lifecycle.js @@ -182,7 +182,13 @@ getTraceFieldLength(trace && trace.x), getTraceFieldLength(trace && trace.y) ); - const traceName = trace && trace.name ? trace.name : "trace_" + String(traceIndex + 1); + const traceName = ( + trace && trace.meta && trace.meta.run_name + ? trace.meta.run_name + : trace && trace.name + ? trace.name + : "trace_" + String(traceIndex + 1) + ); for (let pointIndex = 0; pointIndex < pointCount; pointIndex += 1) { rows.push([ traceName, diff --git a/dashboard/export/js_runtime/renderers/app.js b/dashboard/export/js_runtime/renderers/app.js index fa37674..18b7c52 100644 --- a/dashboard/export/js_runtime/renderers/app.js +++ b/dashboard/export/js_runtime/renderers/app.js @@ -253,8 +253,53 @@ } function renderShell(context, actions) { + const railCollapsed = !!context.railCollapsed; + const rail = renderRail(context, actions); + rail.id = "export-rail"; + + const main = el("main", { className: "export-main" }, [ + renderPageTabs(context, actions), + renderPagePanel(context, actions), + ]); + const layout = el("div", { + className: "export-layout" + (railCollapsed ? " rail-collapsed" : ""), + }, [rail, main]); + const railToggle = el("button", { + className: "rail-toggle", + text: railCollapsed ? "Show sidebar" : "Hide sidebar", + attrs: { + "aria-controls": "export-rail", + "aria-expanded": String(!railCollapsed), + }, + }); + railToggle.type = "button"; + railToggle.addEventListener("click", () => { + context.railCollapsed = !context.railCollapsed; + layout.classList.toggle("rail-collapsed", context.railCollapsed); + railToggle.textContent = context.railCollapsed ? "Show sidebar" : "Hide sidebar"; + railToggle.setAttribute("aria-expanded", String(!context.railCollapsed)); + context.plotManager.scheduleResize(); + }); + + const brandChildren = []; + if (context.payload.logo) { + brandChildren.push( + el("img", { + className: "export-logo", + attrs: { + src: context.payload.logo, + alt: context.payload.title + " logo", + }, + }) + ); + } + brandChildren.push(el("h1", { text: context.payload.title })); + const headerChildren = [ - el("h1", { text: context.payload.title }), + el("div", { className: "export-header-top" }, [ + el("div", { className: "export-brand" }, brandChildren), + railToggle, + ]), ]; if (context.payload.client_export_note && String(context.payload.client_export_note).trim()) { headerChildren.push( @@ -265,17 +310,9 @@ ); } - const main = el("main", { className: "export-main" }, [ - renderPageTabs(context, actions), - renderPagePanel(context, actions), - ]); - return el("div", { className: "export-shell" }, [ el("div", { className: "export-header" }, headerChildren), - el("div", { className: "export-layout" }, [ - renderRail(context, actions), - main, - ]), + layout, ]); } diff --git a/dashboard/export/js_runtime/renderers/plots.js b/dashboard/export/js_runtime/renderers/plots.js index 531c0e6..91e5812 100644 --- a/dashboard/export/js_runtime/renderers/plots.js +++ b/dashboard/export/js_runtime/renderers/plots.js @@ -7,16 +7,24 @@ className: "plot-shell", attrs: { "data-plot-pending": "true" }, }); - if (node.height) { + const aspectRatio = Number(node.aspect_ratio); + const preserveAspectRatio = Number.isFinite(aspectRatio) && aspectRatio > 0; + if (preserveAspectRatio) { + plotElement.style.aspectRatio = String(aspectRatio); + } else if (node.height) { plotElement.style.minHeight = String(node.height) + "px"; } const baseFigure = node.figure || { data: [], layout: {} }; + const layout = Object.assign({}, baseFigure.layout || {}, { + autosize: true, + width: null, + }); + if (preserveAspectRatio) { + delete layout.height; + } const figure = { data: baseFigure.data || [], - layout: Object.assign({}, baseFigure.layout || {}, { - autosize: true, - width: null, - }), + layout: layout, }; context.plotManager.registerPlot(plotElement, figure); return plotElement; diff --git a/dashboard/export/js_runtime/renderers/tables.js b/dashboard/export/js_runtime/renderers/tables.js index 6143057..1689886 100644 --- a/dashboard/export/js_runtime/renderers/tables.js +++ b/dashboard/export/js_runtime/renderers/tables.js @@ -114,6 +114,7 @@ type: "button", "data-column": column, "aria-sort": "none", + title: (node.column_tooltips || {})[column] || column, }, }, [ el("span", { className: "export-table-sort-label", text: column }), diff --git a/dashboard/export/js_runtime/renderers/tabs.js b/dashboard/export/js_runtime/renderers/tabs.js index cacbc1c..3629b77 100644 --- a/dashboard/export/js_runtime/renderers/tabs.js +++ b/dashboard/export/js_runtime/renderers/tabs.js @@ -14,6 +14,7 @@ tabRow.appendChild( makeButton({ label: tab.title, + title: tab.full_title || tab.title, active: index === activeIndex, onClick: () => { activeIndex = index; diff --git a/dashboard/export/payload.py b/dashboard/export/payload.py index 3673810..f6449e9 100644 --- a/dashboard/export/payload.py +++ b/dashboard/export/payload.py @@ -2,7 +2,10 @@ from __future__ import annotations +import base64 import json +import mimetypes +from pathlib import Path from typing import Any from runtime.logging import get_logger @@ -49,6 +52,17 @@ PAGE_WARNING_BYTES = 10 * 1024 * 1024 STATIC_REGION_WARNING_BYTES = 5 * 1024 * 1024 SELECTOR_REGION_WARNING_BYTES = 1 * 1024 * 1024 + + +def _dashboard_logo_data_uri(path: str | None) -> str | None: + if path is None: + return None + logo_path = Path(path) + media_type, _ = mimetypes.guess_type(logo_path.name) + encoded = base64.b64encode(logo_path.read_bytes()).decode("ascii") + return f"data:{media_type};base64,{encoded}" + + def _build_validation_page( page_def: DashboardPageDefinition, config: Config, @@ -103,6 +117,7 @@ def build_export_artifacts( payload: ExportPayload = { "schema_version": EXPORT_SCHEMA_VERSION, "title": config.dashboard_title, + "logo": _dashboard_logo_data_uri(config.dashboard_logo), "runs_loaded": run_legend_entries( RenderContext.from_dashboard(config, chrome_state) ), diff --git a/dashboard/export/serializer.py b/dashboard/export/serializer.py index f952361..398549e 100644 --- a/dashboard/export/serializer.py +++ b/dashboard/export/serializer.py @@ -142,29 +142,38 @@ def _container_css_classes(viewable: Any) -> list[str]: "css_classes": _container_css_classes(obj), } if isinstance(obj, pn.Tabs): + full_titles = tuple(getattr(obj, "_run_label_full_titles", ())) + serialized_tabs = [] + for index, (title, child) in enumerate(iter_tabs(obj)): + if _is_hidden_view(child): + continue + tab = { + "title": title, + "content": serialize_viewable( + child, + disable_widgets=disable_widgets, + widget_metadata=widget_metadata, + region_nodes_by_id=region_nodes_by_id, + hidden_widget_ids=hidden_widget_ids, + hidden_view_ids=hidden_view_ids, + ), + } + if index < len(full_titles) and full_titles[index] != title: + tab["full_title"] = full_titles[index] + serialized_tabs.append(tab) return { "kind": "tabs", - "tabs": [ - { - "title": title, - "content": serialize_viewable( - child, - disable_widgets=disable_widgets, - widget_metadata=widget_metadata, - region_nodes_by_id=region_nodes_by_id, - hidden_widget_ids=hidden_widget_ids, - hidden_view_ids=hidden_view_ids, - ), - } - for title, child in iter_tabs(obj) - if not _is_hidden_view(child) - ], + "tabs": serialized_tabs, } if isinstance(obj, pn.pane.Plotly): figure = obj.object.to_plotly_json() layout = figure.get("layout", {}) if isinstance(figure, dict) else {} height = layout.get("height") if isinstance(layout, dict) else None - return {"kind": "plotly", "figure": figure, "height": height} + node = {"kind": "plotly", "figure": figure, "height": height} + aspect_ratio = getattr(obj, "aspect_ratio", None) + if isinstance(aspect_ratio, (int, float)) and aspect_ratio > 0: + node["aspect_ratio"] = float(aspect_ratio) + return node if isinstance(obj, pn.widgets.Tabulator): frame = obj.value title_map = { @@ -174,7 +183,17 @@ def _container_css_classes(viewable: Any) -> list[str]: } columns = [str(column) for column in frame.columns] display_columns = [title_map.get(column, column) for column in columns] - return { + header_tooltips = { + str(column): str(tooltip) + for column, tooltip in (obj.header_tooltips or {}).items() + if tooltip is not None + } + column_tooltips = { + display_column: header_tooltips[column] + for column, display_column in zip(columns, display_columns) + if column in header_tooltips + } + table = { "kind": "table", "columns": display_columns, "rows": [ @@ -185,6 +204,9 @@ def _container_css_classes(viewable: Any) -> list[str]: for row in frame.to_dict(orient="records") ], } + if column_tooltips: + table["column_tooltips"] = column_tooltips + return table if isinstance(obj, pn.widgets.RadioButtonGroup): if id(obj) in hidden_widget_ids: return {"kind": "spacer", "height": 0, "width": 0} diff --git a/dashboard/export/types.py b/dashboard/export/types.py index efa4f24..86cf02a 100644 --- a/dashboard/export/types.py +++ b/dashboard/export/types.py @@ -91,7 +91,11 @@ class CardNode(TypedDict): children: list["ExportNode"] -class TabPayload(TypedDict): +class OptionalTabPayload(TypedDict, total=False): + full_title: str + + +class TabPayload(OptionalTabPayload): title: str content: "ExportNode" @@ -101,12 +105,21 @@ class TabsNode(TypedDict): tabs: list[TabPayload] -class PlotlyNode(TypedDict): +class OptionalPlotlyNode(TypedDict, total=False): + height: int | float | None + aspect_ratio: float + + +class PlotlyNode(OptionalPlotlyNode): kind: Literal["plotly"] figure: dict[str, Any] -class TableNode(TypedDict): +class OptionalTableNode(TypedDict, total=False): + column_tooltips: dict[str, str] + + +class TableNode(OptionalTableNode): kind: Literal["table"] columns: list[str] rows: list[dict[str, Any]] @@ -172,6 +185,7 @@ class PageExportSupportPayload(TypedDict): class ExportPayload(TypedDict): title: str + logo: str | None runs_loaded: list[dict[str, str]] chrome: ExportChrome dashboard_controls: DashboardControlsPayload diff --git a/dashboard/helpers/category_helpers.py b/dashboard/helpers/category_helpers.py index 667d73a..f83a48e 100644 --- a/dashboard/helpers/category_helpers.py +++ b/dashboard/helpers/category_helpers.py @@ -339,13 +339,15 @@ def normalize_category_strings( def numeric_like_sort_expr(column: str) -> pl.Expr: """Sort numeric-like string bins numerically while leaving text bins at the end.""" - base_value = ( - pl.col(column) - .cast(pl.Utf8) - .str.replace(r"\+$", "") - .cast(pl.Float64, strict=False) + label = pl.col(column).cast(pl.Utf8) + base_value = label.str.replace(r"\+$", "").cast(pl.Float64, strict=False) + return ( + pl.when(label == ">0-<1") + .then(pl.lit(0.5)) + .when(base_value.is_null()) + .then(pl.lit(float("inf"))) + .otherwise(base_value) ) - return pl.when(base_value.is_null()).then(pl.lit(float("inf"))).otherwise(base_value) def capped_numeric_category_expr( @@ -355,11 +357,14 @@ def capped_numeric_category_expr( target_col: str | None = None, ) -> pl.Expr: """Return a string category expression with values at or above ``cap_value`` capped.""" - numeric_value = pl.col(column).cast(pl.Int64, strict=False) + label = pl.col(column).cast(pl.Utf8) + numeric_value = label.str.replace(r"\+$", "").cast(pl.Float64, strict=False) return ( - pl.when(numeric_value >= cap_value) + pl.when(label == ">0-<1") + .then(label) + .when(numeric_value >= cap_value) .then(pl.lit(f"{cap_value}+")) - .otherwise(pl.col(column).cast(pl.Int64, strict=False).cast(pl.Utf8)) + .otherwise(numeric_value.cast(pl.Int64, strict=False).cast(pl.Utf8)) .alias(target_col or column) ) diff --git a/dashboard/helpers/distance_range.py b/dashboard/helpers/distance_range.py index 3d4c09a..bf4597f 100644 --- a/dashboard/helpers/distance_range.py +++ b/dashboard/helpers/distance_range.py @@ -24,7 +24,10 @@ def distance_bin_value(value: object) -> float | None: if isinstance(value, int | float): numeric = float(value) return numeric if isfinite(numeric) else None - match = _DISTANCE_NUMBER_RE.search(str(value).strip()) + label = str(value).strip() + if label == ">0-<1": + return 0.5 + match = _DISTANCE_NUMBER_RE.search(label) if match is None: return None numeric = float(match.group(0)) @@ -96,11 +99,13 @@ def fixed_distance_axis_ticks( step: int = 2, plus_label: bool = True, ) -> tuple[list[float], list[str]]: - """Return fixed whole-mile distance ticks, labeling the max as ``40+``.""" + """Return exact-zero, sub-mile, and fixed whole-mile distance ticks.""" tickvals = [float(value) for value in range(0, max_value + 1, step)] ticktext = [str(int(value)) for value in tickvals] if plus_label and ticktext: ticktext[-1] = f"{int(tickvals[-1])}+" + tickvals.insert(1, 0.5) + ticktext.insert(1, ">0-<1") return tickvals, ticktext diff --git a/dashboard/pages/daily_travel/_escorted_tours/contracts.py b/dashboard/pages/daily_travel/_escorted_tours/contracts.py index 5754b9a..d290ade 100644 --- a/dashboard/pages/daily_travel/_escorted_tours/contracts.py +++ b/dashboard/pages/daily_travel/_escorted_tours/contracts.py @@ -19,7 +19,6 @@ "households_with_school_escorting_by_student_count_and_direction", "schoolkids_per_escorted_tour_by_student_count_and_direction", ) -PAGE_SUMMARY_IDS = (*CORE_SUMMARY_IDS, *OPTIONAL_SUMMARY_IDS) STOP_SEGMENT_LABELS = { "outbound_before_dropoff": "Adult Escort Stops Before Dropoff - Outbound", "outbound_after_dropoff": "Adult Escort Stops After Dropoff - Outbound", @@ -28,7 +27,9 @@ } STUDENT_ESCORT_DESCRIPTION = ( "Student school tours by escort type. `Both Directions` means the same child " - "school tour is escorted in both outbound and inbound directions." + "school tour is escorted in both outbound and inbound directions. `Outbound` " + "means the home-to-school direction is escorted, and `Inbound` means the " + "school-to-home direction is escorted." ) HOUSEHOLD_ESCORT_DESCRIPTION = ( "Households with school escorting by number of students per household. " diff --git a/dashboard/pages/daily_travel/_escorted_tours/domains.py b/dashboard/pages/daily_travel/_escorted_tours/domains.py index 5d784a8..686f95f 100644 --- a/dashboard/pages/daily_travel/_escorted_tours/domains.py +++ b/dashboard/pages/daily_travel/_escorted_tours/domains.py @@ -30,7 +30,7 @@ def _direction_options(self) -> list[str]: "school_escorted_tours_by_escort_type_and_direction", "weighted", ) - if data is None: + if not data: return ["Both Directions"] return direction_options(data) diff --git a/dashboard/pages/daily_travel/_escorted_tours/features.py b/dashboard/pages/daily_travel/_escorted_tours/features.py index dec42e4..f45e5f1 100644 --- a/dashboard/pages/daily_travel/_escorted_tours/features.py +++ b/dashboard/pages/daily_travel/_escorted_tours/features.py @@ -166,7 +166,7 @@ def render_student_school_escort_section(self, summary_data): def render_student_school_escort_charts(self, summary_data): """Build the three student escort status charts when the summary is available.""" - if summary_data is None: + if not summary_data: return None escort_order = self.config.ordered_values("escort", STUDENT_ESCORT_TYPE_ORDER) @@ -240,7 +240,7 @@ def render_household_school_escort_charts( student_count_values: list[str], ): """Build household escort count/rate charts for each direction.""" - if denominator_summary is None or numerator_summary is None: + if not denominator_summary or not numerator_summary: return None charts: list[pn.viewable.Viewable] = [] @@ -321,7 +321,7 @@ def render_schoolkids_per_escorted_tour_charts( student_count_values: list[str], ): """Build average schoolkids-per-tour charts for each direction.""" - if summary_data is None: + if not summary_data: return None charts: list[pn.viewable.Viewable] = [] diff --git a/dashboard/pages/daily_travel/_escorted_tours_data.py b/dashboard/pages/daily_travel/_escorted_tours_data.py index a4976d5..1f58331 100644 --- a/dashboard/pages/daily_travel/_escorted_tours_data.py +++ b/dashboard/pages/daily_travel/_escorted_tours_data.py @@ -13,7 +13,7 @@ ) DIRECTION_COL = "direction" -DISTANCE_BINS = [str(i) for i in range(40)] + ["40+"] +DISTANCE_BINS = ["0", ">0-<1", *[str(i) for i in range(1, 40)], "40+"] def direction_options(data_list: list[tuple[str, pl.DataFrame]]) -> list[str]: diff --git a/dashboard/pages/daily_travel/daily_activity_pattern.py b/dashboard/pages/daily_travel/daily_activity_pattern.py index 305da91..af02580 100644 --- a/dashboard/pages/daily_travel/daily_activity_pattern.py +++ b/dashboard/pages/daily_travel/daily_activity_pattern.py @@ -67,7 +67,7 @@ def _person_type_source_data(self, weighting_key: str): """Use the first available person-type summary to seed the selector domain.""" for summary_id in PERSON_TYPE_SUMMARY_IDS: data = self.data.summary(summary_id, weighting_key) - if data is not None: + if data: return data return None @@ -330,7 +330,7 @@ def render_body(self): return [self.no_runs_message()] summaries = self._optional_summaries() - if not any(data is not None for data in summaries.values()): + if not any(summaries.values()): return [self.summary_only_unavailable_card()] display_person_type, raw_person_type = self._selected_person_type() diff --git a/dashboard/pages/daily_travel/escorted_tours.py b/dashboard/pages/daily_travel/escorted_tours.py index 15afb63..61fd4c5 100644 --- a/dashboard/pages/daily_travel/escorted_tours.py +++ b/dashboard/pages/daily_travel/escorted_tours.py @@ -15,7 +15,8 @@ title="Escorted Tours", group_id="daily_travel", order=29, - required_summary_ids=(*PAGE_SUMMARY_IDS,), + required_summary_ids=CORE_SUMMARY_IDS, + optional_summary_ids=OPTIONAL_SUMMARY_IDS, ) class EscortedToursPage( EscortedToursCompositionMixin, diff --git a/dashboard/pages/joint_travel.py b/dashboard/pages/joint_travel.py index 33d00b8..c430b2d 100644 --- a/dashboard/pages/joint_travel.py +++ b/dashboard/pages/joint_travel.py @@ -85,14 +85,14 @@ def _party_size_options(self) -> list[str]: "joint_tour_composition_by_party_size", self.weighting_key, ) - return party_size_options(data) if data is not None else ["All"] + return party_size_options(data) if data else ["All"] def _household_size_options(self) -> list[str]: data = self.data.summary( "household_jtp_by_household_size_and_jtf", self.weighting_key, ) - return household_size_options(data) if data is not None else ["All"] + return household_size_options(data) if data else ["All"] def _summaries(self): return self.data.summaries(*self.required_summary_ids) @@ -116,8 +116,12 @@ def render_frequency(self): if not self.state.run_labels: return [self.no_runs_message()] summaries = self._summaries() - if summaries is None: - return [self.summary_only_unavailable_card()] + if not summaries["jtf_distribution"]: + return [ + self.summary_only_unavailable_card( + summary_ids=("jtf_distribution",), + ) + ] return [ selector_row(self.hide_no_joint_tours, height=48), self.noted_view( @@ -128,8 +132,6 @@ def render_frequency(self): def render_joint_tour_detail(self): summaries = self._summaries() - if summaries is None: - return [] party_size = self.party_size_sel.value joint_tours_hhsize_data = [ (label, df.with_columns(pl.col("household_size").cast(pl.Utf8))) @@ -160,35 +162,53 @@ def render_joint_tour_detail(self): summaries["joint_tour_composition_by_party_size"], party_size ) ) + household_size_view = ( + self.render_household_size_chart( + complete_joint_household_size_data( + joint_tours_hhsize_data, + value_col="joint_tour_hh_count", + household_size_values=household_size_values, + ), + household_size_values, + ) + if summaries["joint_tours_by_household_size"] + else self.summary_only_unavailable_card( + summary_ids=("joint_tours_by_household_size",), + ) + ) + party_size_view = ( + self.render_party_size_chart(party_size_data, party_size_values) + if summaries["joint_tour_party_size_distribution"] + else self.summary_only_unavailable_card( + summary_ids=("joint_tour_party_size_distribution",), + ) + ) + composition_view = ( + self.render_composition_chart( + comp_party_data, + composition_label_values, + party_size, + ) + if summaries["joint_tour_composition_by_party_size"] + else self.summary_only_unavailable_card( + summary_ids=("joint_tour_composition_by_party_size",), + ) + ) return [ pn.Column( selector_row(self.party_size_sel), pn.Row( self.noted_view( "joint_travel.household_size", - self.render_household_size_chart( - complete_joint_household_size_data( - joint_tours_hhsize_data, - value_col="joint_tour_hh_count", - household_size_values=household_size_values, - ), - household_size_values, - ), + household_size_view, ), self.noted_view( "joint_travel.party_size", - self.render_party_size_chart( - party_size_data, - party_size_values, - ), + party_size_view, ), self.noted_view( "joint_travel.composition", - self.render_composition_chart( - comp_party_data, - composition_label_values, - party_size, - ), + composition_view, ), sizing_mode="stretch_width", ), @@ -198,8 +218,6 @@ def render_joint_tour_detail(self): def render_participation(self): summaries = self._summaries() - if summaries is None: - return [] hhsize = self.hhsize_sel.value person_participation = self.query( lambda: person_participation_data( @@ -228,6 +246,35 @@ def render_participation(self): ) ], ) + person_view = ( + self.render_person_participation_chart( + complete_joint_household_size_data( + person_participation, + value_col="person_value", + household_size_values=household_size_values, + ), + household_size_values, + ) + if summaries["person_jtp_by_household_size"] + else self.summary_only_unavailable_card( + summary_ids=("person_jtp_by_household_size",), + ) + ) + household_view = ( + self.render_household_participation_chart( + household_participation, + jtf_values, + hhsize, + ) + if any(not df.is_empty() for _, df in household_participation) + else self.summary_only_unavailable_card( + summary_ids=("household_jtp_by_household_size_and_jtf",), + detail=( + "The household joint-tour participation summary has no data " + f"for household size `{hhsize}`." + ), + ) + ) return [ pn.Column( pn.Row( @@ -238,22 +285,11 @@ def render_participation(self): pn.Row( self.noted_view( "joint_travel.person_participation", - self.render_person_participation_chart( - complete_joint_household_size_data( - person_participation, - value_col="person_value", - household_size_values=household_size_values, - ), - household_size_values, - ), + person_view, ), self.noted_view( "joint_travel.household_participation", - self.render_household_participation_chart( - household_participation, - jtf_values, - hhsize, - ), + household_view, ), sizing_mode="stretch_width", ), diff --git a/dashboard/pages/long_term_choices/_mandatory_location_choice/domains.py b/dashboard/pages/long_term_choices/_mandatory_location_choice/domains.py index 888f310..903f6df 100644 --- a/dashboard/pages/long_term_choices/_mandatory_location_choice/domains.py +++ b/dashboard/pages/long_term_choices/_mandatory_location_choice/domains.py @@ -2,19 +2,8 @@ from __future__ import annotations -import panel as pn -import polars as pl -from dashboard.helpers.distance_range import ( - DistanceRangeControls, - capped_distance_max_options, - distance_axis_bounds, - fixed_distance_axis_ticks, - with_distance_axis, -) from dashboard.helpers.geography_helpers import * -from dashboard.page_base import SectionContent -from dashboard.rendering import data_table, selector_row from dashboard.pages.long_term_choices._mandatory_location_choice_support import * @@ -200,7 +189,7 @@ def _collect_data(self) -> dict[str, object]: "average_mandatory_tour_distance_by_purpose_and_geography", ) - if not any(summary is not None for summary in summaries.values()): + if not any(summaries.values()): return { "mode": "unavailable", "geo_opts": [ALL_GEOGRAPHY_TYPES_LABEL], @@ -243,6 +232,7 @@ def _collect_data(self) -> dict[str, object]: geo_opts, geo_raw_by_label = geography_type_options( internal_external or None, work_from_home or None, + telecommute or None, work_distance or None, school_distance or None, university_distance or None, @@ -252,6 +242,7 @@ def _collect_data(self) -> dict[str, object]: ) geography_option_sources = ( internal_external or None, + telecommute or None, work_distance or None, school_distance or None, university_distance or None, diff --git a/dashboard/pages/long_term_choices/_mandatory_location_choice/features.py b/dashboard/pages/long_term_choices/_mandatory_location_choice/features.py index 45ace21..3777f89 100644 --- a/dashboard/pages/long_term_choices/_mandatory_location_choice/features.py +++ b/dashboard/pages/long_term_choices/_mandatory_location_choice/features.py @@ -45,18 +45,32 @@ def render_worker_geography_section(self) -> SectionContent: geography, ) ) - worker_views.append( - self.noted_view( - "mandatory_location.worker_status_table", - data_table( - [ - (label, self.render_internal_external_worker_table(df)) - for label, df in internal_external_table - ], - "Internal vs. External Workers", - ), + if any(not df.is_empty() for _, df in internal_external_table): + worker_views.append( + self.noted_view( + "mandatory_location.worker_status_table", + data_table( + [ + ( + label, + self.render_internal_external_worker_table(df), + ) + for label, df in internal_external_table + ], + "Internal vs. External Workers", + ), + ) + ) + else: + worker_views.append( + self.data_not_available_card( + detail=( + "No internal/external worker data is available for " + "the selected geography." + ), + missing_items=["internal_external_worker_by_geography"], + ) ) - ) else: worker_views.append( self.data_not_available_card( @@ -174,8 +188,9 @@ def render_external_workplace_chart( def render_distance_distribution_section(self) -> SectionContent: """Render the three mandatory distance distributions side by side.""" - if self._current_data["mode"] != "ready": - return [] + placeholder = self._render_ready_state() + if placeholder is not None: + return placeholder geo_level, geography = self._selected_geography() chart_specs = [ @@ -207,6 +222,7 @@ def render_distance_distribution_section(self) -> SectionContent: self.distance_distribution_chart_data( geo_level, geography, + summary_id=spec["summary_id"], summary_data=spec["summary_data"], ), ) @@ -264,19 +280,24 @@ def distance_distribution_chart_data( geo_level: str, geography: str, *, + summary_id: str, summary_data: list[tuple[str, pl.DataFrame]] | None, ) -> list[tuple[str, pl.DataFrame]] | None: """Return chart-ready mandatory distance data for one summary.""" if summary_data is None: return None + summary_query_input = (summary_id, summary_data) filtered_summary = self.query( lambda: filter_selected_geography( - summary_data, + summary_query_input[1], geo_level, geography, ) ) - return self.query(lambda: distance_distribution_chart_data(filtered_summary)) + chart_query_input = (summary_id, filtered_summary) + return self.query( + lambda: distance_distribution_chart_data(chart_query_input[1]) + ) def render_distance_distribution_chart( self, @@ -321,8 +342,9 @@ def render_distance_distribution_chart( def render_remote_work_section(self) -> SectionContent: """Render work-from-home and telecommute summaries.""" - if self._current_data["mode"] != "ready": - return [] + placeholder = self._render_ready_state() + if placeholder is not None: + return placeholder geo_level, geography = self._selected_geography() return [ @@ -359,6 +381,13 @@ def render_work_from_home_chart( geography, ) ) + if not any(not df.is_empty() for _, df in wfh_data): + return self.data_not_available_card( + detail=( + "No work-from-home data is available for the selected geography." + ), + missing_items=["work_from_home_rate_by_geography"], + ) return self.plot.bar( wfh_data, x="geography_label", @@ -426,8 +455,9 @@ def render_telecommute_chart( def render_mandatory_distance_table_section(self) -> SectionContent: """Render the percent-difference table for average mandatory tour distance.""" - if self._current_data["mode"] != "ready": - return [] + placeholder = self._render_ready_state() + if placeholder is not None: + return placeholder geo_level, geography = self._selected_geography() average_distance = self._current_data["average_distance"] @@ -468,6 +498,11 @@ def render_mandatory_distance_table_section(self) -> SectionContent: data_table( comparison_tables, "Average Mandatory Tour Distance vs Base Run", + numeric_decimal_places_by_column={ + "Average Mandatory Tour Distance": 1, + "Base Run Average Mandatory Tour Distance": 1, + "Difference": 1, + }, ), ) ] diff --git a/dashboard/pages/long_term_choices/individual_choices.py b/dashboard/pages/long_term_choices/individual_choices.py index 1e4b48a..57cbe50 100644 --- a/dashboard/pages/long_term_choices/individual_choices.py +++ b/dashboard/pages/long_term_choices/individual_choices.py @@ -113,7 +113,7 @@ def _summary_or_placeholder( *, detail: str, ) -> list[tuple[str, pl.DataFrame]] | pn.Card: - summary = self.data.summary(summary_name, required=False) + summary = self.data.summary(summary_name) if summary: return summary return self.data_not_available_card(detail=detail, missing_items=[summary_name]) diff --git a/dashboard/pages/long_term_choices/shadow_pricing.py b/dashboard/pages/long_term_choices/shadow_pricing.py index 3446160..a85cfd5 100644 --- a/dashboard/pages/long_term_choices/shadow_pricing.py +++ b/dashboard/pages/long_term_choices/shadow_pricing.py @@ -227,20 +227,16 @@ def _collect_data(self) -> dict[str, object]: } workplace_summary = normalize_geography_data( - self.data.summary("workplace_shadow_pricing_residuals", required=False) + self.data.summary("workplace_shadow_pricing_residuals") ) school_summary = normalize_geography_data( - self.data.summary("school_shadow_pricing_residuals", required=False) + self.data.summary("school_shadow_pricing_residuals") ) workplace_hist = normalize_geography_data( - self.data.summary( - "workplace_shadow_pricing_residual_histogram", required=False - ) + self.data.summary("workplace_shadow_pricing_residual_histogram") ) school_hist = normalize_geography_data( - self.data.summary( - "school_shadow_pricing_residual_histogram", required=False - ) + self.data.summary("school_shadow_pricing_residual_histogram") ) geo_opts, geo_raw_by_label = geography_type_options( workplace_hist or school_hist or workplace_summary or school_summary, @@ -310,12 +306,17 @@ def render_workplace_plot_section(self) -> SectionContent: def render_workplace_table_section(self) -> SectionContent: """Render the workplace residual table.""" - if self._current_data["mode"] != "ready": - return [] + if self._current_data["mode"] == "no_runs": + return [self.no_runs_message()] workplace_summary = self._current_data["workplace_summary"] if workplace_summary is None: - return [] + return [ + self.data_not_available_card( + detail="The workplace employment residual summary is unavailable.", + missing_items=["workplace_shadow_pricing_residuals"], + ) + ] if self._maz_tables_disabled(): return [ self.data_not_available_card( @@ -360,8 +361,8 @@ def render_workplace_table(self, df: pl.DataFrame) -> pl.DataFrame: def render_school_plot_section(self) -> SectionContent: """Render the school residual distribution for one student type.""" - if self._current_data["mode"] != "ready": - return [] + if self._current_data["mode"] == "no_runs": + return [self.no_runs_message()] school_hist = self._current_data["school_hist"] if school_hist is None: @@ -408,12 +409,17 @@ def render_school_plot_section(self) -> SectionContent: def render_school_table_section(self) -> SectionContent: """Render school residuals for the selected geography level and student type.""" - if self._current_data["mode"] != "ready": - return [] + if self._current_data["mode"] == "no_runs": + return [self.no_runs_message()] school_summary = self._current_data["school_summary"] if school_summary is None: - return [] + return [ + self.data_not_available_card( + detail="The school enrollment residual summary is unavailable.", + missing_items=["school_shadow_pricing_residuals"], + ) + ] if self._maz_tables_disabled(): return [ self.data_not_available_card( diff --git a/dashboard/pages/long_term_choices/vehicle_ownership_type.py b/dashboard/pages/long_term_choices/vehicle_ownership_type.py index fce65d2..c399a9b 100644 --- a/dashboard/pages/long_term_choices/vehicle_ownership_type.py +++ b/dashboard/pages/long_term_choices/vehicle_ownership_type.py @@ -161,7 +161,7 @@ def render_ownership_summary(self): def render_vehicle_mix(self): if not self.state.run_labels: - return [] + return [self.no_runs_message()] summaries = self._optional_summaries() vehicle_views: list[pn.viewable.Viewable] = [] @@ -216,11 +216,17 @@ def render_auto_ownership_chart(self, summary_data): missing_items=["auto_ownership_distribution"], ) household_size = str(self.hhsize_sel.value) + chart_data = _auto_ownership_chart_data(summary_data, household_size) + if not any(not df.is_empty() for _, df in chart_data): + return self.data_not_available_card( + detail=( + "The auto ownership summary has no data for household size " + f"`{household_size}`." + ), + missing_items=["auto_ownership_distribution"], + ) return self.plot.bar( - _auto_ownership_chart_data( - summary_data, - household_size, - ), + chart_data, x="household_vehicle_count", y="household_count", title=f"Auto Ownership by Household Size - {household_size}", diff --git a/dashboard/pages/overview.py b/dashboard/pages/overview.py index 16b7e88..6dd2681 100644 --- a/dashboard/pages/overview.py +++ b/dashboard/pages/overview.py @@ -7,10 +7,10 @@ from dashboard.rendering import ( to_pandas, - column_titles, drop_index_columns, format_numeric_frame, ) +from dashboard.rendering.tables import column_title_metadata from dashboard.helpers.comparison_helpers import ( build_base_run_percent_difference_table, ) @@ -19,8 +19,8 @@ from dashboard.page_base import SectionContent KPI_METRICS = [ - ("person_count", "Population"), - ("household_count", "Households"), + ("person_count", "Population (person-days)"), + ("household_count", "Households (HH-days)"), ("auto_vmt", "VMT"), ("tour_count", "Tours"), ("trip_count", "Trips"), @@ -182,11 +182,13 @@ def render_percent_difference_table( drop_index_columns(pct_df), numeric_precision=2, ) + titles, header_tooltips = column_title_metadata(display_df.columns) return pn.widgets.Tabulator( to_pandas(display_df), sizing_mode="stretch_width", height=260, - titles=column_titles(display_df.columns), + titles=titles, + header_tooltips=header_tooltips, show_index=False, ) @@ -249,12 +251,12 @@ def render_kpis(self) -> SectionContent: self._kpi_card( totals_list, metric="person_count", - label="Population", + label="Population (person-days)", ), self._kpi_card( totals_list, metric="household_count", - label="Households", + label="Households (HH-days)", ), vmt_box, sizing_mode="stretch_width", @@ -303,7 +305,7 @@ def render_kpis(self) -> SectionContent: def render_demographics(self) -> SectionContent: """Render the demographic distribution charts.""" if not self.state.run_labels: - return [] + return [self.no_runs_message()] ptype_result, hhsize_result = self._demographic_results() return [ diff --git a/dashboard/pages/skim_summaries/_shared.py b/dashboard/pages/skim_summaries/_shared.py index e980e6e..9590de0 100644 --- a/dashboard/pages/skim_summaries/_shared.py +++ b/dashboard/pages/skim_summaries/_shared.py @@ -27,14 +27,15 @@ ) SKIM_FAMILY_MODE_MAP = { "Auto Skims": ("SOV", "HOV2", "HOV3"), - "Transit Skims": ("WALK_TRANSIT", "PNR_TRANSIT", "KNR_TRANSIT"), + "Transit Skims": ("WALK_TRANSIT", "BIKE_TRANSIT", "PNR_TRANSIT", "KNR_TRANSIT"), "Walk Skims": ("WALK",), - "Bike Skims": ("BIKE", "EBIKE", "ESCOOTER", "BIKE_TRANSIT"), + "Bike Skims": ("BIKE", "EBIKE", "ESCOOTER"), } SUMMARY_METRIC_COLUMNS = [ "n_total", "n_valid", "mean", + "mean_nonzero", "std", "min", "max", @@ -118,7 +119,7 @@ def component_display_name( "skim_auto_time": "Drive Time (min)", "skim_auto_distance": "Drive Distance (mi)", "skim_auto_cost": "Drive Cost ($)", - "skim_walk_distance": "TAZ Skim Walk Distance (mi)", + "skim_walk_distance": "Walk Distance (mi)", "skim_walk_time": "Total Walk Access/Egress Time (min)", "skim_walk_maz_distance": "MAZ Network Walk Distance (mi)", "skim_walk_maz_actual": "MAZ Actual Walk Time (min)", @@ -146,8 +147,15 @@ def component_display_name( "skim_walk_time_inbound": "Total Walk Access/Egress Time (min)", "skim_transit_tiv_outbound": "Transit In-Vehicle Time (min)", "skim_transit_tiv_inbound": "Transit In-Vehicle Time (min)", - "skim_bike_distance": "TAZ Skim Bike Distance (mi)", + "skim_bike_distance": "Bike Distance (mi)", "skim_bike_maz_distance": "MAZ Network Bike Distance (mi)", + "skim_bike_transit_distance_bus": ( + "Total Bike Distance - Local Bus (mi) (Estimated from Walk Skims)" + ), + "skim_bike_transit_distance_premium": ( + "Total Bike Distance - Premium Transit (mi) " + "(Estimated from Walk Skims)" + ), } if value in special_labels: return special_labels[value] @@ -283,9 +291,34 @@ def _ignored_modes_for_series( def _configured_outputs_by_mode( config: Config, series: DashboardSummarySeries, + df: pl.DataFrame, *, + mode_column: str, target_table: str, ) -> dict[str, set[str]]: + required_columns = {"skim_scenario", mode_column, "component"} + if required_columns.issubset(df.columns): + summarized_pairs = ( + df.select( + pl.col("skim_scenario").cast(pl.Utf8), + pl.col(mode_column).cast(pl.Utf8), + pl.col("component").cast(pl.Utf8), + ) + .filter( + (pl.col("skim_scenario") == ALL_RECORDS_SCENARIO) + & pl.col(mode_column).is_not_null() + & (pl.col(mode_column) != ALL_MODES) + & pl.col("component").is_not_null() + ) + .select(mode_column, "component") + .unique() + ) + if not summarized_pairs.is_empty(): + outputs_by_mode: dict[str, set[str]] = {} + for mode, component in summarized_pairs.iter_rows(): + outputs_by_mode.setdefault(str(mode), set()).add(str(component)) + return outputs_by_mode + skimjoin_settings = _skimjoin_settings_for_series(config, series) normalized = getattr(skimjoin_settings, "normalized_config", None) if normalized is None: @@ -310,6 +343,8 @@ def _skim_family_definitions( configured_outputs = _configured_outputs_by_mode( config, series, + df, + mode_column=mode_column, target_table=target_table, ) ignored_modes = _ignored_modes_for_series(config, series) @@ -334,11 +369,6 @@ def _skim_family_definitions( for mode in configured_outputs if mode != ALL_MODES and mode in available_modes ] - all_modes.extend( - mode - for mode in available_modes - if mode not in all_modes and mode != ALL_MODES - ) elif configured_outputs: all_modes = [mode for mode in configured_outputs if mode != ALL_MODES] else: @@ -463,7 +493,7 @@ def family_stats_table( direction_suffix = None if direction is None else f"_{direction.lower()}" target_columns = ["skim_name", mode_column, *SUMMARY_METRIC_COLUMNS] filtered_list: list[tuple[str, pl.DataFrame]] = [] - for label, _, df in nonempty_series(data_list): + for label, series, df in nonempty_series(data_list): family_definition = family_definitions_by_label.get(label, {}).get(family) family_modes = family_definition.get("modes", ()) if family_definition else () configured_outputs = ( @@ -486,8 +516,25 @@ def family_stats_table( & (pl.col(mode_column) != ALL_MODES) ) if configured_outputs: - filtered = filtered.filter( - pl.col("component").is_in(list(configured_outputs)) + outputs_by_mode = _configured_outputs_by_mode( + config, + series, + df, + mode_column=mode_column, + target_table=target_table, + ) + mode_output_filters = [] + for mode in family_modes: + mode_outputs = outputs_by_mode.get(mode) + if mode_outputs: + mode_output_filters.append( + (pl.col(mode_column) == mode) + & pl.col("component").is_in(sorted(mode_outputs)) + ) + filtered = ( + filtered.filter(pl.any_horizontal(mode_output_filters)) + if mode_output_filters + else filtered.head(0) ) if direction_suffix is not None: filtered = filtered.filter( @@ -510,10 +557,19 @@ def family_stats_table( return filtered_list -def skim_summary_precision_overrides() -> dict[str, int]: +def skim_summary_decimal_places() -> dict[str, int]: return { "n_total": 0, "n_valid": 0, + "mean": 1, + "mean_nonzero": 1, + "std": 1, + "min": 1, + "max": 1, + "median": 1, + "mode": 1, + "zero_share": 3, + "missing_share": 3, } diff --git a/dashboard/pages/skim_summaries/tour_skims.py b/dashboard/pages/skim_summaries/tour_skims.py index deb8206..6cc467e 100644 --- a/dashboard/pages/skim_summaries/tour_skims.py +++ b/dashboard/pages/skim_summaries/tour_skims.py @@ -18,7 +18,7 @@ skim_direction_options, skim_family_options, skim_scenario_available, - skim_summary_precision_overrides, + skim_summary_decimal_places, tour_component_base_options, tour_mode_options, ) @@ -313,7 +313,7 @@ def render_summary_table(self): family = self.tour_family_sel.value direction = self.tour_direction_sel.value - if tour_stats is None: + if not tour_stats: return self.data_not_available_card( detail="Tour skim summaries require the precomputed skim tour statistics table.", missing_items=[TOUR_STATS_SUMMARY_ID], @@ -348,8 +348,7 @@ def render_summary_table(self): stats_data, title=f"Tour Summary Statistics - {family} / {direction}", height=280, - numeric_precision=2, - numeric_precision_by_column=skim_summary_precision_overrides(), + numeric_decimal_places_by_column=skim_summary_decimal_places(), ) def render_summary_section(self): diff --git a/dashboard/pages/skim_summaries/trip_skims.py b/dashboard/pages/skim_summaries/trip_skims.py index 349cba1..ec66bb1 100644 --- a/dashboard/pages/skim_summaries/trip_skims.py +++ b/dashboard/pages/skim_summaries/trip_skims.py @@ -18,7 +18,7 @@ skim_scenario_available, resolve_distribution_range, skim_family_options, - skim_summary_precision_overrides, + skim_summary_decimal_places, ) from dashboard.pages.skim_summaries._page_controls import ( repair_selector_options, @@ -263,7 +263,7 @@ def render_summary_section(self): return [self.no_runs_message()] trip_stats = self._trip_summaries() - if trip_stats is None: + if not trip_stats: return [ self.data_not_available_card( detail="Trip skim summaries require the precomputed skim trip statistics table.", @@ -363,8 +363,7 @@ def render_summary_table( trip_stats_data, title=f"Trip Summary Statistics - {family}", height=280, - numeric_precision=2, - numeric_precision_by_column=skim_summary_precision_overrides(), + numeric_decimal_places_by_column=skim_summary_decimal_places(), ) def render_distribution_chart( diff --git a/dashboard/pages/tour_summaries/_tour_mode/transforms.py b/dashboard/pages/tour_summaries/_tour_mode/transforms.py index 5bf53c6..9d13b41 100644 --- a/dashboard/pages/tour_summaries/_tour_mode/transforms.py +++ b/dashboard/pages/tour_summaries/_tour_mode/transforms.py @@ -24,21 +24,24 @@ def auto_sufficiency_display_label(auto_sufficiency: str, config) -> str: _, label_noun = _auto_sufficiency_basis_terms(config) return { "All": "All", - "Zero Auto": "Zero Auto", - "Auto Deficient": f"Fewer Vehicles Than {label_noun}", - "Auto Sufficient": f"At Least As Many Vehicles as {label_noun}", + "Zero Auto": "Zero Auto Households", + "Auto Deficient": f"Households with Fewer Vehicles Than {label_noun}", + "Auto Sufficient": ( + f"Households with At Least as Many Vehicles as {label_noun}" + ), }[auto_sufficiency] def auto_sufficiency_definitions_markdown(config) -> str: """Describe the configured household basis behind the auto sufficiency split.""" basis_noun, _ = _auto_sufficiency_basis_terms(config) + zero_auto_label = auto_sufficiency_display_label("Zero Auto", config) deficient_label = auto_sufficiency_display_label("Auto Deficient", config) sufficient_label = auto_sufficiency_display_label("Auto Sufficient", config) return f""" **Auto sufficiency definitions** - - **Zero Auto**: household has no vehicles. + - **{zero_auto_label}**: household has no vehicles. - **{deficient_label}**: household has fewer vehicles than {basis_noun}. - **{sufficient_label}**: household has at least as many vehicles as {basis_noun}. """ diff --git a/dashboard/pages/tour_summaries/internal_external_tours.py b/dashboard/pages/tour_summaries/internal_external_tours.py index 06c3805..ad26d6d 100644 --- a/dashboard/pages/tour_summaries/internal_external_tours.py +++ b/dashboard/pages/tour_summaries/internal_external_tours.py @@ -176,7 +176,7 @@ def render_internal_external_table( table_data = self.query(lambda: filter_geography_level(summary_data, geo_level)) return data_table( - self._display_geography_table(table_data, geography_col="home_geography"), + self._display_geography_table(table_data), "Internal vs. External Non-Mandatory Tour Frequency", ) diff --git a/dashboard/pages/tour_summaries/park_and_ride_location.py b/dashboard/pages/tour_summaries/park_and_ride_location.py index 43b3305..1855ca5 100644 --- a/dashboard/pages/tour_summaries/park_and_ride_location.py +++ b/dashboard/pages/tour_summaries/park_and_ride_location.py @@ -109,12 +109,10 @@ def _collect_data(self) -> dict[str, object]: } residuals = normalize_geography_data( - self.data.summary("park_and_ride_location_residuals", required=False) + self.data.summary("park_and_ride_location_residuals") ) histogram = normalize_geography_data( - self.data.summary( - "park_and_ride_location_residual_histogram", required=False - ) + self.data.summary("park_and_ride_location_residual_histogram") ) geo_opts, geo_raw_by_label = geography_type_options( histogram or residuals, @@ -167,12 +165,17 @@ def render_plot_section(self) -> SectionContent: def render_table_section(self) -> SectionContent: """Render the residual table for the selected geography level.""" - if self._current_data["mode"] != "ready": - return [] + if self._current_data["mode"] == "no_runs": + return [self.no_runs_message()] residuals = self._current_data["residuals"] if residuals is None: - return [] + return [ + self.data_not_available_card( + detail="The park-and-ride residual summary is unavailable.", + missing_items=["park_and_ride_location_residuals"], + ) + ] if self._maz_tables_disabled(): return [ self.data_not_available_card( diff --git a/dashboard/pages/tour_summaries/tour_distance.py b/dashboard/pages/tour_summaries/tour_distance.py index be9c146..ddacb6d 100644 --- a/dashboard/pages/tour_summaries/tour_distance.py +++ b/dashboard/pages/tour_summaries/tour_distance.py @@ -211,8 +211,6 @@ def _summaries(self) -> dict[str, object] | None: def _distance_sources(self): summaries = self._summaries() - if not summaries: - return None, None, None nonmandatory_average = normalize_geography_data( summaries["average_nonmandatory_tour_distance_by_purpose_and_geography"] ) @@ -282,8 +280,13 @@ def render_distance_section(self) -> SectionContent: return [self.no_runs_message()] summaries = self._summaries() - if summaries is None: - return [self.summary_only_unavailable_card()] + distance_summary = summaries["tour_distance_by_tour_purpose"] + if not distance_summary: + return [ + self.summary_only_unavailable_card( + summary_ids=("tour_distance_by_tour_purpose",), + ) + ] selected_purpose = str(self.tour_purpose_sel.value) raw_purpose = self._tour_purpose_to_raw.get( @@ -291,7 +294,7 @@ def render_distance_section(self) -> SectionContent: ) distance_data = self.query( lambda: tour_distance_chart_data( - summaries["tour_distance_by_tour_purpose"], + distance_summary, str(raw_purpose), ) ) @@ -344,11 +347,20 @@ def render_distance_chart( def render_average_section(self) -> SectionContent: """Render the average non-mandatory distance comparison table.""" summaries = self._summaries() - if summaries is None: - return [] + nonmandatory_summary = summaries[ + "average_nonmandatory_tour_distance_by_purpose_and_geography" + ] + if not nonmandatory_summary: + return [ + self.summary_only_unavailable_card( + summary_ids=( + "average_nonmandatory_tour_distance_by_purpose_and_geography", + ), + ) + ] nonmandatory_average = normalize_geography_data( - summaries["average_nonmandatory_tour_distance_by_purpose_and_geography"] + nonmandatory_summary ) geo_level = self.selected_geography_level_raw() geography = self.selected_geography_raw() diff --git a/dashboard/pages/tour_summaries/tour_time.py b/dashboard/pages/tour_summaries/tour_time.py index d4ef3a6..3fb0647 100644 --- a/dashboard/pages/tour_summaries/tour_time.py +++ b/dashboard/pages/tour_summaries/tour_time.py @@ -35,6 +35,7 @@ def tour_time_chart_data( def clock_profile(value_col: str) -> list[tuple[str, pl.DataFrame]]: return ( filtered.select("time_bin", value_col) + .map(lambda frame: frame.filter(pl.col("time_bin") > 0)) .sort("time_bin") .with_columns( pl.col("time_bin") @@ -98,7 +99,7 @@ def _purpose_options(self) -> list[str]: "tour_time_of_day_by_tour_purpose", self.weighting_key, ) - if data is None: + if not data: self._purpose_to_raw = {self.TOTAL_PURPOSE_LABEL: "all_tour_purposes"} return [self.TOTAL_PURPOSE_LABEL] options, self._purpose_to_raw = column_options( diff --git a/dashboard/pages/validation/_traffic/composition.py b/dashboard/pages/validation/_traffic/composition.py index 230e17f..338f8c1 100644 --- a/dashboard/pages/validation/_traffic/composition.py +++ b/dashboard/pages/validation/_traffic/composition.py @@ -12,6 +12,7 @@ class TrafficPageCompositionMixin: def build_page(self) -> pn.viewable.Viewable: self.demo_facility_raw_by_label = {"All": "All"} + self.screenline_facility_raw_by_label = {"All": "All"} self.demo_period_sel = self.selector( "demo_period", widget=pn.widgets.Select( @@ -44,6 +45,20 @@ def build_page(self) -> pn.viewable.Viewable: ), label="Top N by Modeled Volume", ) + self.screenline_period_sel = self.selector( + "screenline_period", + widget=pn.widgets.Select( + name="Time Period", + options=list(DEMO_TRAFFIC_TIME_PERIODS), + value="Day", + ), + label="Time Period", + ) + self.screenline_facility_sel = self.select( + "screenline_facility_type", + "Facility Type", + options=self._screenline_facility_options, + ) observed_fit = self.feature("observed_model_fit") facility = self.feature("facility_summaries") links = self.feature("link_tables") @@ -76,6 +91,7 @@ def build_page(self) -> pn.viewable.Viewable: ) self._screenline_body = screenlines.section( "body", + selectors=("screenline_period", "screenline_facility_type"), render=self.render_screenline_flow_section, ) return self.new_section( @@ -104,6 +120,10 @@ def build_page(self) -> pn.viewable.Viewable: self._external_top_body, ), pn.pane.Markdown("### Screenline Flow Summaries"), + selector_row( + self.screenline_period_sel, + self.screenline_facility_sel, + ), self.noted_section("traffic.screenlines", self._screenline_body), sizing_mode="stretch_width", ) diff --git a/dashboard/pages/validation/_traffic/features.py b/dashboard/pages/validation/_traffic/features.py index 916fc10..e6dd29b 100644 --- a/dashboard/pages/validation/_traffic/features.py +++ b/dashboard/pages/validation/_traffic/features.py @@ -13,45 +13,45 @@ class TrafficFeatureMixin: - def render_validation_chart( - self, - data_list: list[tuple[str, pl.DataFrame]] | None, - *, - title: str, - detail: str, - missing_summary_id: str, - ) -> pn.viewable.Viewable: - if data_list is None: - return self.data_not_available_card( - detail=detail, - missing_items=[missing_summary_id], - ) - chart_data = self.query(lambda: validation_chart_data(data_list)) - return self.plot.scatter( - chart_data, - x="observed_volume", - y="modeled_volume", - title=title, - x_title="Observed Traffic Volume", - y_title="Modeled Traffic Volume", - ) - def render_screenline_flow_section(self): if not self.state.run_labels: return [self.no_runs_message()] - + data = self.data.summary("screenline_flow_comparisons", self.weighting_key) + if not data: + return [ + self.data_not_available_card( + detail="Screenline flow comparisons are unavailable.", + missing_items=["screenline_flow_comparisons"], + ) + ] + period = str(self.screenline_period_sel.value) + facility_type = self.selected_screenline_facility_type_raw() + scatter_data = self.query( + lambda: screenline_scatter_data( + data, + period=period, + facility_type=facility_type, + ) + ) + fit_data = self.query(lambda: screenline_fit_line_data(scatter_data)) return [ - self.render_validation_chart( - self.data.summary("screenline_flow_comparisons", self.weighting_key), - title="Screenline Flow Comparisons", - detail="Screenline flow comparisons are unavailable.", - missing_summary_id="screenline_flow_comparisons", + self.plot.scatter( + scatter_data, + x="observed_volume", + y="modeled_volume", + title=f"Screenline Observed vs Modeled - {period}", + x_title="Observed Screenline Flow (vehicles)", + y_title="Modeled Screenline Flow (vehicles)", + fit_overlays=fit_data, + one_to_one=True, + legend_on_right=True, + panel_aspect_ratio=1.0, ) ] def render_demo_facility_summary_section(self) -> list[pn.viewable.Viewable]: if not self.state.run_labels: - return [] + return [self.no_runs_message()] count_list = self.data.summary( "count_location_counts_validation_summary", self.weighting_key @@ -66,7 +66,15 @@ def render_demo_facility_summary_section(self) -> list[pn.viewable.Viewable]: "count_location_fit_validation_summary", self.weighting_key ) if not any((count_list, volume_list, scatter_list, fit_list)): - return [] + return [ + self.data_not_available_card( + detail="Count-location facility summaries are unavailable.", + missing_items=[ + "count_location_counts_validation_summary", + "count_location_volumes_validation_summary", + ], + ) + ] # Keep this overview on unfiltered daily totals. The controls below it # belong only to the Traffic Volume Summaries sections. @@ -110,8 +118,8 @@ def render_demo_facility_summary_section(self) -> list[pn.viewable.Viewable]: data_table( facility_comparison, title="Count Location Summary by Facility Type", - numeric_precision_by_column={"RMSE": 3, "R^2": 3}, - column_sorters={"n": "number", "RMSE": "number", "R^2": "number"}, + numeric_precision_by_column={"RMSE": 3, "R²": 3}, + column_sorters={"n": "number", "RMSE": "number", "R²": "number"}, ) ] @@ -131,8 +139,6 @@ def render_demo_traffic_section(self) -> list[pn.viewable.Viewable]: fit_list = self.data.summary( "count_location_fit_validation_summary", self.weighting_key ) - if not any((count_list, volume_list, scatter_list, fit_list)): - return [] period = self.demo_period_sel.value volume_col = DEMO_TRAFFIC_TIME_PERIODS[str(period)] facility_type = self.selected_facility_type_raw() @@ -158,10 +164,11 @@ def render_demo_traffic_section(self) -> list[pn.viewable.Viewable]: x="observed_volume", y="modeled_volume", title=f"Count Location Observed vs Modeled - {period}", - x_title="Observed Count", - y_title="Modeled Volume", + x_title="Observed Count (vehicles)", + y_title="Modeled Volume (vehicles)", fit_overlays=fit_data, one_to_one=True, + legend_on_right=True, panel_aspect_ratio=1.0, ) ) @@ -180,9 +187,10 @@ def render_demo_traffic_section(self) -> list[pn.viewable.Viewable]: x="observed_volume", y="modeled_volume", title=f"Count Location Observed vs Modeled - {period}", - x_title="Observed Count", - y_title="Modeled Volume", + x_title="Observed Count (vehicles)", + y_title="Modeled Volume (vehicles)", one_to_one=True, + legend_on_right=True, panel_aspect_ratio=1.0, ) ) @@ -203,10 +211,10 @@ def render_demo_traffic_section(self) -> list[pn.viewable.Viewable]: def render_demo_link_volume_section(self) -> list[pn.viewable.Viewable]: if not self.state.run_labels: - return [] + return [self.no_runs_message()] link_list = self.data.summary("link_validation_summary", self.weighting_key) - if link_list is None: + if not link_list: return [ self.data_not_available_card( detail="Link validation summaries are unavailable.", @@ -251,15 +259,12 @@ def render_demo_top_count_section(self) -> list[pn.viewable.Viewable]: volume_list = self.data.summary( "count_location_volumes_validation_summary", self.weighting_key ) - if not any((link_list, count_list, volume_list)): - return [] - facility_type = self.selected_facility_type_raw() top_period = self.demo_top_period_sel.value top_volume_col = DEMO_TRAFFIC_TIME_PERIODS[str(top_period)] top_n = int(self.demo_top_n_sel.value) - if count_list is not None and volume_list is not None: + if count_list and volume_list: volume_comparison = self.query( lambda: label_category_data( demo_volume_comparison_table( @@ -286,17 +291,15 @@ def render_demo_top_count_section(self) -> list[pn.viewable.Viewable]: column_sorters={"Difference": "number"}, ), ] - if link_list is not None: - return [ - self.data_not_available_card( - detail=( - "Count-location validation counts and volumes are both " - "required for this comparison table." - ), - missing_items=[ - "count_location_counts_validation_summary", - "count_location_volumes_validation_summary", - ], - ) - ] - return [] + return [ + self.data_not_available_card( + detail=( + "Count-location validation counts and volumes are both " + "required for this comparison table." + ), + missing_items=[ + "count_location_counts_validation_summary", + "count_location_volumes_validation_summary", + ], + ) + ] diff --git a/dashboard/pages/validation/_traffic/selector_domains.py b/dashboard/pages/validation/_traffic/selector_domains.py index 112e4bd..b09adc0 100644 --- a/dashboard/pages/validation/_traffic/selector_domains.py +++ b/dashboard/pages/validation/_traffic/selector_domains.py @@ -37,3 +37,15 @@ def selected_facility_type_raw(self) -> str: selected = str(self.demo_facility_sel.value) raw_value = self.demo_facility_raw_by_label.get(selected, selected) return "All" if raw_value is None else str(raw_value) + + def _screenline_facility_options(self) -> list[str]: + options, self.screenline_facility_raw_by_label = demo_facility_options( + self.data.summary("screenline_flow_comparisons", self.weighting_key), + config=self.config, + ) + return options + + def selected_screenline_facility_type_raw(self) -> str: + selected = str(self.screenline_facility_sel.value) + raw_value = self.screenline_facility_raw_by_label.get(selected, selected) + return "All" if raw_value is None else str(raw_value) diff --git a/dashboard/pages/validation/_traffic/transforms.py b/dashboard/pages/validation/_traffic/transforms.py index 5c5e6c4..ec4a860 100644 --- a/dashboard/pages/validation/_traffic/transforms.py +++ b/dashboard/pages/validation/_traffic/transforms.py @@ -16,28 +16,45 @@ from .contracts import * -def validation_chart_data( +def screenline_scatter_data( data_list: list[tuple[str, pl.DataFrame]], + *, + period: str, + facility_type: str, ) -> list[tuple[str, pl.DataFrame]]: - """Aggregate one validation summary list to one observed/modeled point per id.""" - out = [] + """Filter screenline observed/modeled points for one period and facility.""" + out: list[tuple[str, pl.DataFrame]] = [] + required = { + "screenline_id", + "count_period", + "observed_volume", + "modeled_volume", + } for label, df in nonempty(data_list): - filtered = df - id_col = None - if "count_location_id" in filtered.columns: - id_col = "count_location_id" - elif "screenline_id" in filtered.columns: - id_col = "screenline_id" - if id_col is not None: - filtered = ( - filtered.group_by(id_col) - .agg( - observed_volume=pl.col("observed_volume").sum(), - modeled_volume=pl.col("modeled_volume").sum(), - ) - .sort(id_col) + if not required.issubset(df.columns): + continue + filtered = df.with_columns( + pl.col("count_period").cast(pl.Utf8), + ( + pl.col("facility_type").cast(pl.Utf8) + if "facility_type" in df.columns + else pl.lit("All") + ).alias("facility_type"), + ).filter(pl.col("count_period") == period) + if facility_type != "All": + filtered = filtered.filter(pl.col("facility_type") == facility_type) + out.append( + ( + label, + filtered.select( + "screenline_id", + "facility_type", + "count_period", + "observed_volume", + "modeled_volume", + ).sort("screenline_id"), ) - out.append((label, filtered)) + ) return out @@ -146,6 +163,26 @@ def demo_count_scatter_data_from_sources( return out +def _fit_line_frame( + *, + observed_min: float, + observed_max: float, + slope: float, + intercept: float, + annotation: str, +) -> pl.DataFrame: + point_count = 101 + step = (observed_max - observed_min) / (point_count - 1) + observed = [observed_min + step * index for index in range(point_count)] + return pl.DataFrame( + { + "observed_volume": observed, + "modeled_volume": [slope * value + intercept for value in observed], + "annotation": [annotation] * point_count, + } + ) + + def demo_count_fit_line_data( fit_list: list[tuple[str, pl.DataFrame]] | None, *, @@ -197,22 +234,21 @@ def demo_count_fit_line_data( out.append( ( label, - pl.DataFrame( - { - "observed_volume": [observed_min, observed_max], - "modeled_volume": [ - slope * observed_min + intercept, - slope * observed_max + intercept, - ], - "annotation": [annotation, annotation], - } + _fit_line_frame( + observed_min=observed_min, + observed_max=observed_max, + slope=slope, + intercept=intercept, + annotation=annotation, ), ) ) return out -def _r_squared_from_points(points: pl.DataFrame) -> float | None: +def _linear_fit_from_points( + points: pl.DataFrame, +) -> tuple[float, float, float] | None: if points.height < 2: return None x = [float(value) for value in points["observed_volume"].to_list()] @@ -229,8 +265,48 @@ def _r_squared_from_points(points: pl.DataFrame) -> float | None: sse = sum((yi - yhat) ** 2 for yi, yhat in zip(y, fitted)) ss_yy = sum((yi - y_mean) ** 2 for yi in y) if math.isclose(ss_yy, 0.0): - return 1.0 if math.isclose(sse, 0.0) else 0.0 - return max(0.0, min(1.0, 1.0 - sse / ss_yy)) + r_squared = 1.0 if math.isclose(sse, 0.0) else 0.0 + else: + r_squared = max(0.0, min(1.0, 1.0 - sse / ss_yy)) + return slope, intercept, r_squared + + +def _r_squared_from_points(points: pl.DataFrame) -> float | None: + fit = _linear_fit_from_points(points) + return None if fit is None else fit[2] + + +def screenline_fit_line_data( + scatter_data: list[tuple[str, pl.DataFrame]], +) -> list[tuple[str, pl.DataFrame]]: + """Build regression lines and annotations from filtered screenline points.""" + out: list[tuple[str, pl.DataFrame]] = [] + for label, df in nonempty(scatter_data): + points = df.select("observed_volume", "modeled_volume").drop_nulls() + fit = _linear_fit_from_points(points) + if fit is None: + continue + slope, intercept, r_squared = fit + observed_min = float(points["observed_volume"].min()) + observed_max = float(points["observed_volume"].max()) + sign = "+" if intercept >= 0 else "-" + annotation = ( + f"{label}
    y = {slope:.2f}x {sign} {abs(intercept):.2f}" + f"
    R² = {r_squared:.2f}
    n = {points.height}" + ) + out.append( + ( + label, + _fit_line_frame( + observed_min=observed_min, + observed_max=observed_max, + slope=slope, + intercept=intercept, + annotation=annotation, + ), + ) + ) + return out def _fit_r_squared_lookup( @@ -330,6 +406,17 @@ def demo_facility_comparison_table( rmse = math.sqrt( sum(difference**2 for difference in differences) / len(differences) ) + if any(value == 0.0 for value in observed): + rmspe = "" + else: + squared_percentage_errors = [ + ((observe - model) / observe) ** 2 + for observe, model in zip(observed, modeled) + ] + rmspe_value = math.sqrt( + sum(squared_percentage_errors) / len(squared_percentage_errors) + ) * 100.0 + rmspe = f"{rmspe_value:.2f}%" percent_value = ( None if total_observed == 0.0 @@ -349,7 +436,8 @@ def demo_facility_comparison_table( "Total Modeled Count": total_modeled, "% Difference": percent_difference, "RMSE": rmse, - "R^2": r_squared_lookup.get(raw_facility_type) + "RMSPE": rmspe, + "R²": r_squared_lookup.get(raw_facility_type) if raw_facility_type in r_squared_lookup else _r_squared_from_points(facility_points), } diff --git a/dashboard/pages/validation/_vmt/features.py b/dashboard/pages/validation/_vmt/features.py index e645f37..cc3a37c 100644 --- a/dashboard/pages/validation/_vmt/features.py +++ b/dashboard/pages/validation/_vmt/features.py @@ -43,7 +43,17 @@ def render_vmt_overview_section(self) -> list[pn.viewable.Viewable]: ), ) if not overview_data: - return [] + return [ + self.data_not_available_card( + detail="VMT overview summaries are unavailable.", + missing_items=[ + PERSONAL_AUTO_VMT_SUMMARY_ID, + NON_MOTORIZED_VMT_SUMMARY_ID, + EXTERNAL_VMT_SUMMARY_ID, + COMMERCIAL_VMT_SUMMARY_ID, + ], + ) + ] return [ data_table( overview_data, @@ -64,7 +74,7 @@ def render_bicycle_chart(self) -> pn.viewable.Viewable: "bicycle_vmt_by_facility_type", self.weighting_key, ) - if bicycle_vmt is None: + if not bicycle_vmt: return self.data_not_available_card( detail="Bicycle VMT summaries are unavailable.", missing_items=["bicycle_vmt_by_facility_type"], @@ -87,7 +97,7 @@ def render_bicycle_section(self): class SegmentedVmtFeatureMixin: def render_personal_auto_vmt_section(self) -> list[pn.viewable.Viewable]: if not self.state.run_labels: - return [] + return [self.no_runs_message()] personal_vmt = self.data.summary( PERSONAL_AUTO_VMT_SUMMARY_ID, columns=PERSONAL_AUTO_VMT_REQUIRED_COLUMNS, @@ -189,7 +199,7 @@ def render_personal_auto_vmt_section(self) -> list[pn.viewable.Viewable]: def render_non_motorized_vmt_section(self) -> list[pn.viewable.Viewable]: if not self.state.run_labels: - return [] + return [self.no_runs_message()] non_motorized_vmt = self.data.summary( NON_MOTORIZED_VMT_SUMMARY_ID, columns=NON_MOTORIZED_VMT_REQUIRED_COLUMNS, @@ -301,15 +311,6 @@ def render_body(self): def render_commercial_vmt_section(self): if not self.state.run_labels: return [self.no_runs_message()] - summary_ids = [ - "commercial_vehicle_validation_summary", - "commercial_vehicle_vmt_validation_summary", - ] - if not any( - self.data.summary(summary_id, self.weighting_key) - for summary_id in summary_ids - ): - return [] return [self.render_demo_commercial_chart()] def render_demo_commercial_chart(self) -> pn.viewable.Viewable: @@ -319,7 +320,7 @@ def render_demo_commercial_chart(self) -> pn.viewable.Viewable: else "commercial_vehicle_validation_summary" ) data = self.data.summary(summary_id, self.weighting_key) - if data is None: + if not data: return self.data_not_available_card( detail="Commercial vehicle summaries are unavailable.", missing_items=[summary_id], @@ -386,7 +387,7 @@ def render_external_travel_chart(self) -> pn.viewable.Viewable: else "external_trip_validation_summary" ) data = self.data.summary(summary_id, self.weighting_key) - if data is None: + if not data: return self.data_not_available_card( detail="External travel summaries are unavailable.", missing_items=[summary_id], @@ -445,15 +446,6 @@ def render_external_travel_chart(self) -> pn.viewable.Viewable: ) def render_external_vmt_section(self) -> list[pn.viewable.Viewable]: - summary_ids = [ - "external_trip_validation_summary", - "external_vmt_validation_summary", - ] - if not any( - self.data.summary(summary_id, self.weighting_key) - for summary_id in summary_ids - ): - return [] content: list[pn.viewable.Viewable] = [ self.render_external_travel_chart(), ] diff --git a/dashboard/pages/validation/regional.py b/dashboard/pages/validation/regional.py index f6e7e14..e39a200 100644 --- a/dashboard/pages/validation/regional.py +++ b/dashboard/pages/validation/regional.py @@ -10,22 +10,27 @@ from dashboard.rendering import selector_row from dashboard.helpers.category_helpers import nonempty +from dashboard.rendering.labels import ( + attach_full_tab_titles, + display_label_map, + hover_label, +) from dashboard import DashboardPage, dashboard_page TOTAL_FLOW_LABELS = {"total", "all", "all_geographies"} FLOW_COMPARISON_OPTIONS = [ + "Modeled", "Observed", "Difference", - "Percent Difference", - "Absolute Percent Difference", - "Modeled", + "% Difference", + "Absolute % Difference", ] FLOW_VALUE_COLUMNS = { "Modeled": "modeled", "Observed": "observed", "Difference": "difference", - "Percent Difference": "percent_difference", - "Absolute Percent Difference": "absolute_percent_difference", + "% Difference": "percent_difference", + "Absolute % Difference": "absolute_percent_difference", } @@ -37,11 +42,11 @@ class FlowOption: FLOW_OPTIONS = { "District flows": FlowOption( - summary_id="county_flows_validation_summary", + summary_id="district_commuting_flows_validation_summary", modeled_geography_types=("district", "home_district"), ), "County flows": FlowOption( - summary_id="county_flows_joja_validation_summary", + summary_id="county_commuting_flows_validation_summary", modeled_geography_types=("county", "home_county"), ), } @@ -325,7 +330,11 @@ def flow_heatmap( title: str, ) -> pn.viewable.Viewable: tabs = pn.Tabs() - for label, df in nonempty(data_list): + runs = nonempty(data_list) + full_labels = [str(label) for label, _ in runs] + display_labels = display_label_map(full_labels) + for label, df in runs: + full_label = str(label) matrix = normalize_flow_matrix(df, include_totals=include_totals) destinations = [column for column in matrix.columns if column != "Origin"] z = matrix.select(destinations).to_numpy().tolist() @@ -340,7 +349,8 @@ def flow_heatmap( y=matrix["Origin"].cast(pl.Utf8).to_list(), colorscale="Blues", hovertemplate=( - "Origin: %{y}
    Destination: %{x}
    Flow: %{z:,.0f}" + f"{hover_label(full_label)}
    Origin: %{{y}}
    " + "Destination: %{x}
    Flow: %{z:,.0f}" ), ) ) @@ -352,7 +362,13 @@ def flow_heatmap( margin=dict(l=70, r=20, t=80, b=70), font=dict(family="Inter, Segoe UI, Arial, sans-serif", size=12), ) - tabs.append((label, pn.pane.Plotly(fig, sizing_mode="stretch_width"))) + tabs.append( + ( + display_labels[full_label], + pn.pane.Plotly(fig, sizing_mode="stretch_width"), + ) + ) + attach_full_tab_titles(tabs, full_labels) return tabs @@ -365,7 +381,11 @@ def flow_comparison_heatmap( """Render aligned observed/modeled flow comparisons as heatmaps.""" value_col = FLOW_VALUE_COLUMNS[metric] tabs = pn.Tabs() - for label, df in nonempty(data_list): + runs = nonempty(data_list) + full_labels = [str(label) for label, _ in runs] + display_labels = display_label_map(full_labels) + for label, df in runs: + full_label = str(label) if df.is_empty(): continue origins = _flow_label_order(df["Origin"].to_list(), include_totals=True) @@ -381,7 +401,7 @@ def flow_comparison_heatmap( [lookup.get((origin, destination)) for destination in destinations] for origin in origins ] - if metric in {"Percent Difference", "Absolute Percent Difference"}: + if metric in {"% Difference", "Absolute % Difference"}: text = [ ["" if value is None else f"{float(value):,.1f}%" for value in row] for row in z @@ -392,7 +412,7 @@ def flow_comparison_heatmap( for row in z ] colorscale = ( - "RdBu_r" if metric in {"Difference", "Percent Difference"} else "Blues" + "RdBu_r" if metric in {"Difference", "% Difference"} else "Blues" ) z_values = [ abs(float(value)) for row in z for value in row if value is not None @@ -407,11 +427,12 @@ def flow_comparison_heatmap( "y": origins, "colorscale": colorscale, "hovertemplate": ( - "Origin: %{y}
    Destination: %{x}
    " + f"{hover_label(full_label)}
    Origin: %{{y}}
    " + "Destination: %{x}
    " f"{metric}: %{{text}}" ), } - if metric in {"Difference", "Percent Difference"} and zmax is not None: + if metric in {"Difference", "% Difference"} and zmax is not None: heatmap_kwargs.update(zmid=0, zmin=-zmax, zmax=zmax) fig = go.Figure(data=go.Heatmap(**heatmap_kwargs)) fig.update_layout( @@ -422,7 +443,13 @@ def flow_comparison_heatmap( margin=dict(l=70, r=20, t=80, b=70), font=dict(family="Inter, Segoe UI, Arial, sans-serif", size=12), ) - tabs.append((label, pn.pane.Plotly(fig, sizing_mode="stretch_width"))) + tabs.append( + ( + display_labels[full_label], + pn.pane.Plotly(fig, sizing_mode="stretch_width"), + ) + ) + attach_full_tab_titles(tabs, full_labels) return tabs @@ -433,8 +460,8 @@ def flow_comparison_heatmap( order=55, default_enabled=False, optional_summary_ids=( - "county_flows_validation_summary", - "county_flows_joja_validation_summary", + "district_commuting_flows_validation_summary", + "county_commuting_flows_validation_summary", "commuting_flows", ), ) @@ -499,7 +526,7 @@ def render_flow_section(self) -> pn.viewable.Viewable: flow_option.summary_id, self.weighting_key, ) - if observed_data is None: + if not observed_data: return self.data_not_available_card( detail="External regional flow summaries are unavailable.", missing_items=[flow_option.summary_id], @@ -523,7 +550,7 @@ def render_flow_section(self) -> pn.viewable.Viewable: modeled_data, flow_option.modeled_geography_types, ) - if modeled_data is None or geography_type is None: + if not modeled_data or geography_type is None: return self.data_not_available_card( detail=( "Modeled commuting flows are unavailable for the selected " diff --git a/dashboard/pages/validation/transit.py b/dashboard/pages/validation/transit.py index 0dcaf3d..268bb8e 100644 --- a/dashboard/pages/validation/transit.py +++ b/dashboard/pages/validation/transit.py @@ -8,7 +8,6 @@ from dashboard.rendering import selector_row from dashboard.data_access import RunTables from dashboard.helpers.category_helpers import ( - common_column_options, column_options, nonempty, ) @@ -84,18 +83,19 @@ def build_page(self) -> pn.viewable.Viewable: ) self._transfer_body = self.section( "transit_transfer_body", - selectors=("technology", "access_mode"), + selectors=("access_mode",), render=self.render_transfer_section, ) return self.new_section( pn.pane.Markdown("## Transit Validation"), pn.pane.Markdown("### Transit Boardings"), - selector_row(self.technology_sel, self.access_mode_sel), + selector_row(self.technology_sel), self._boardings_body, self.section_note( "transit_validation.boardings", self._boardings_body ), pn.pane.Markdown("### Transfer Rate"), + selector_row(self.access_mode_sel), self.noted_section( "transit_validation.transfer_rate", self._transfer_body ), @@ -103,11 +103,11 @@ def build_page(self) -> pn.viewable.Viewable: ) def _technology_options(self) -> list[str]: - options, _ = common_column_options( + options, _ = column_options( self.data.summary( "transit_boardings_by_operator_and_technology", self.weighting_key - ), - self.data.summary("transit_transfer_rate", self.weighting_key), + ) + or [], column="technology", total_raw="All", total_label="All", @@ -147,7 +147,7 @@ def render_boardings_chart( "transit_boardings_by_operator_and_technology", self.weighting_key, ) - if boarding_list is None: + if not boarding_list: return self.data_not_available_card( detail="Transit boarding summaries are unavailable.", missing_items=["transit_boardings_by_operator_and_technology"], @@ -171,21 +171,20 @@ def render_transfer_chart(self, operator_values: list[str]) -> pn.viewable.Viewa "transit_transfer_rate", self.weighting_key, ) - if transfer_list is None: + if not transfer_list: return self.data_not_available_card( detail="Transit transfer summaries are unavailable.", missing_items=["transit_transfer_rate"], ) - technology = self.technology_sel.value access_mode = self.access_mode_sel.value transfer_data = self.query( - lambda: filter_transit_data(transfer_list, technology, access_mode) + lambda: filter_transit_data(transfer_list, "All", access_mode) ) return self.plot.bar( transfer_data, x="operator", y="transfer_rate", - title=f"Transit Transfer Rate - {technology}, {access_mode}", + title=f"Transit Transfer Rate - {access_mode}", x_title="Operator", y_title="Boardings per Linked Trip", value_mode="count", @@ -199,23 +198,15 @@ def render_boardings_section(self): "transit_boardings_by_operator_and_technology", self.weighting_key, ) - transfer_list = self.data.summary( - "transit_transfer_rate", - self.weighting_key, - ) - operator_values = self._operator_values(boarding_list, transfer_list) + operator_values = self._operator_values(boarding_list) return [self.render_boardings_chart(operator_values)] def render_transfer_section(self): if not self.state.run_labels: return [self.no_runs_message()] - boarding_list = self.data.summary( - "transit_boardings_by_operator_and_technology", - self.weighting_key, - ) transfer_list = self.data.summary( "transit_transfer_rate", self.weighting_key, ) - operator_values = self._operator_values(boarding_list, transfer_list) + operator_values = self._operator_values(transfer_list) return [self.render_transfer_chart(operator_values)] diff --git a/dashboard/rendering/figures.py b/dashboard/rendering/figures.py index 250b9f8..773b1a9 100644 --- a/dashboard/rendering/figures.py +++ b/dashboard/rendering/figures.py @@ -11,6 +11,7 @@ from dashboard.data_access import RunTableData, RunTables from dashboard.rendering.context import RenderContext +from dashboard.rendering.labels import display_label_map, hover_label ChartTables = RunTables | RunTableData ChartValueMode = Literal["dashboard", "count", "share"] @@ -81,11 +82,20 @@ def _point_hover( share: bool, ) -> str: return ( - f"{label}
    {x_title or x}: {x_value}" + f"{hover_label(label)}
    {x_title or x}: {x_value}" f"
    {_y_title(y_title or y, share)}: {_hover_value(y_value, y_title, share)}" ) +def _run_labels(context: RenderContext, data: ChartTables) -> list[str]: + labels = list(context.run_labels) + for label, _ in data: + text = str(label) + if text not in labels: + labels.append(text) + return labels + + def bar_figure( context: RenderContext, data: ChartTables, @@ -103,9 +113,11 @@ def bar_figure( show_legend: bool | None = None, ) -> go.Figure: """Build a grouped/stacked categorical figure.""" + data = list(data) _require_columns(data, "bar", x, y) share = _share_mode(context, value_mode) figure = go.Figure() + legend_labels = display_label_map(_run_labels(context, data)) observed_order: list[object] = [] for index, (label, frame) in enumerate(data): if frame.is_empty(): @@ -135,8 +147,9 @@ def bar_figure( ] figure.add_trace( go.Bar( - name=str(label), x=x_values, y=y_values, + name=legend_labels[str(label)], x=x_values, y=y_values, marker_color=context.color(str(label), index), + meta={"run_name": str(label)}, hovertemplate="%{customdata}", customdata=hover, ) ) @@ -166,16 +179,29 @@ def line_figure( value_mode: ChartValueMode = "dashboard", height: int = 350, ) -> go.Figure: + data = list(data) _require_columns(data, "line", x, y) share = _share_mode(context, value_mode) figure = go.Figure() + legend_labels = display_label_map(_run_labels(context, data)) for index, (label, frame) in enumerate(data): + x_values = frame[x].to_list() values = np.asarray(frame[y].to_list(), dtype=float) if share and values.sum() > 0: values = values / values.sum() * 100.0 + y_values = values.tolist() + hover = [ + _point_hover( + str(label), x_value, y_value, x=x, y=y, + x_title=x_title, y_title=y_title, share=share, + ) + for x_value, y_value in zip(x_values, y_values) + ] figure.add_trace(go.Scatter( - name=str(label), x=frame[x].to_list(), y=values.tolist(), mode="lines", + name=legend_labels[str(label)], x=x_values, y=y_values, mode="lines", line=dict(color=context.color(str(label), index), width=2), + meta={"run_name": str(label)}, + hovertemplate="%{customdata}", customdata=hover, )) _layout(figure, title=title, x_title=x_title, y_title=_y_title(y_title, share), height=height) return figure @@ -212,9 +238,11 @@ def density_figure( tick_text: list[str] | None = None, hover_x_title: str | None = None, ) -> go.Figure: + data = list(data) _require_columns(data, "density", x, y) share = _share_mode(context, value_mode) figure = go.Figure() + legend_labels = display_label_map(_run_labels(context, data)) observed_x: list[object] = [] for index, (label, frame) in enumerate(data): x_values = frame[x].to_list() @@ -230,8 +258,9 @@ def density_figure( for xv, yv in zip(x_values, y_values) ] figure.add_trace(go.Scatter( - name=str(label), x=x_values, y=y_values, mode="lines", + name=legend_labels[str(label)], x=x_values, y=y_values, mode="lines", line=dict(color=color, width=2), fill="tozeroy", + meta={"run_name": str(label)}, hovertemplate="%{customdata}", customdata=hover, )) _layout(figure, title=title, x_title=x_title, y_title=_y_title(y_title, share), height=height) @@ -277,9 +306,19 @@ def scatter_figure( fit_overlays: ChartTables | None = None, fit_annotation: str = "annotation", one_to_one: bool = False, + legend_on_right: bool = False, ) -> go.Figure: + data = list(data) + fit_overlays = list(fit_overlays or []) _require_columns(data, "scatter", x, y) figure = go.Figure() + all_run_labels = _run_labels(context, data) + for label, _ in fit_overlays: + if str(label) not in all_run_labels: + all_run_labels.append(str(label)) + legend_labels = display_label_map( + [*all_run_labels, *(f"{label} fit" for label in all_run_labels)] + ) label_indices = {str(label): index for index, (label, _) in enumerate(data)} axis_values: list[float] = [] for index, (label, frame) in enumerate(data): @@ -291,30 +330,66 @@ def scatter_figure( if one_to_one: axis_values.extend(_finite([*x_values, *y_values])) figure.add_trace(go.Scatter( - name=str(label), x=x_values, y=y_values, mode="markers", + name=legend_labels[str(label)], x=x_values, y=y_values, mode="markers", marker=dict(color=context.color(str(label), index), size=8, line=dict(width=0.4)), + legendgroup=str(label), + meta={"run_name": str(label)}, + hovertemplate=( + f"{hover_label(label)}
    {x_title or x}: %{{x}}
    " + f"{y_title or y}: %{{y}}" + ), )) - for index, (label, frame) in enumerate(fit_overlays or []): + for index, (label, frame) in enumerate(fit_overlays): if frame.is_empty() or x not in frame.columns or y not in frame.columns: continue color = context.color(str(label), label_indices.get(str(label), index)) + trace_name = legend_labels[f"{label} fit"] + annotation = "" + if fit_annotation in frame.columns: + annotation = str(frame[fit_annotation][0] or "").strip() + prefix = f"{label}
    " + if annotation.startswith(prefix): + annotation = f"{hover_label(label)}
    {annotation[len(prefix):]}" figure.add_trace(go.Scatter( - name=f"{label} fit", x=frame[x].to_list(), y=frame[y].to_list(), + name=trace_name, x=frame[x].to_list(), y=frame[y].to_list(), mode="lines", line=dict(color=color, width=2), + legendgroup=str(label), + meta={"run_name": f"{label} fit"}, + hovertemplate=(f"{annotation}" if annotation else None), )) - if fit_annotation in frame.columns and str(frame[fit_annotation][0] or "").strip(): - figure.add_annotation( - text=str(frame[fit_annotation][0]), xref="paper", yref="paper", - x=0.02, y=max(0.05, 0.98 - 0.12 * index), showarrow=False, - font=dict(color=color, size=12), bgcolor="rgba(255,255,255,0.75)", - bordercolor=color, borderwidth=1, - ) if one_to_one: - maximum = max([value for value in axis_values if value >= 0], default=1.0) or 1.0 + if axis_values: + minimum = min(axis_values) + maximum = max(axis_values) + if minimum == maximum: + padding = max(abs(minimum) * 0.05, 1.0) + minimum -= padding + maximum += padding + else: + minimum, maximum = 0.0, 1.0 figure.add_trace(go.Scatter( - name="1:1 line", x=[0.0, maximum], y=[0.0, maximum], mode="lines", + name="1:1 line", x=[minimum, maximum], y=[minimum, maximum], mode="lines", line=dict(color="#BDBDBD", width=1.5, dash="dash"), - hoverinfo="skip", showlegend=False, + hoverinfo="skip", showlegend=True, )) _layout(figure, title=title, x_title=x_title, y_title=y_title, height=height) + if one_to_one: + figure.update_xaxes(range=[minimum, maximum], constrain="domain") + figure.update_yaxes( + range=[minimum, maximum], + constrain="domain", + scaleanchor="x", + scaleratio=1.0, + ) + if legend_on_right: + figure.update_layout( + legend=dict( + orientation="v", + x=1.02, + xanchor="left", + y=1.0, + yanchor="top", + ), + margin=dict(l=60, r=180, t=90, b=90), + ) return figure diff --git a/dashboard/rendering/labels.py b/dashboard/rendering/labels.py new file mode 100644 index 0000000..3e6635f --- /dev/null +++ b/dashboard/rendering/labels.py @@ -0,0 +1,111 @@ +"""Presentation-only shortening and wrapping for run labels.""" + +from __future__ import annotations + +import html +import textwrap +from collections.abc import Iterable +from typing import Any + + +MAX_DISPLAY_LABEL_LENGTH = 30 +HOVER_LABEL_LINE_LENGTH = 36 + + +def _truncate_middle(label: str, max_length: int) -> str: + if len(label) <= max_length: + return label + if max_length <= 1: + return "…"[:max_length] + + words = label.split() + if len(words) >= 3 and len(words[0]) + len(words[-1]) + 1 <= max_length: + leading_words = [words[0]] + trailing_words = [words[-1]] + leading_index = 1 + trailing_index = len(words) - 2 + while leading_index <= trailing_index: + changed = False + leading_candidate = ( + " ".join([*leading_words, words[leading_index]]) + + "…" + + " ".join(trailing_words) + ) + if len(leading_candidate) <= max_length: + leading_words.append(words[leading_index]) + leading_index += 1 + changed = True + trailing_candidate = ( + " ".join(leading_words) + + "…" + + " ".join([words[trailing_index], *trailing_words]) + ) + if leading_index <= trailing_index and len(trailing_candidate) <= max_length: + trailing_words.insert(0, words[trailing_index]) + trailing_index -= 1 + changed = True + if not changed: + break + return " ".join(leading_words) + "…" + " ".join(trailing_words) + + available = max_length - 1 + leading_length = (available * 2 + 2) // 3 + trailing_length = available - leading_length + leading = label[:leading_length].rstrip() + trailing = label[-trailing_length:].lstrip() if trailing_length else "" + return f"{leading}…{trailing}" + + +def display_label_map( + labels: Iterable[object], + *, + max_length: int = MAX_DISPLAY_LABEL_LENGTH, +) -> dict[str, str]: + """Return stable, unique display labels without changing full identities.""" + full_labels = list(dict.fromkeys(str(label) for label in labels)) + candidates = { + label: _truncate_middle(label, max_length) + for label in full_labels + } + groups: dict[str, list[str]] = {} + for label, candidate in candidates.items(): + groups.setdefault(candidate, []).append(label) + + output: dict[str, str] = {} + used: set[str] = set() + for label in full_labels: + candidate = candidates[label] + duplicates = groups[candidate] + if len(duplicates) == 1 and candidate not in used: + output[label] = candidate + used.add(candidate) + continue + + index = duplicates.index(label) + 1 + while True: + suffix = f" [{index}]" + unique_candidate = ( + _truncate_middle(label, max_length - len(suffix)) + suffix + ) + if unique_candidate not in used: + output[label] = unique_candidate + used.add(unique_candidate) + break + index += 1 + return output + + +def hover_label(label: object) -> str: + """Return the escaped full label with line breaks suitable for Plotly.""" + lines = textwrap.wrap( + str(label), + width=HOVER_LABEL_LINE_LENGTH, + break_long_words=True, + break_on_hyphens=False, + ) or [""] + return "
    ".join(html.escape(line) for line in lines) + + +def attach_full_tab_titles(tabs: Any, labels: Iterable[object]) -> None: + """Attach full titles for the standalone-export serializer.""" + tabs._run_label_full_titles = tuple(str(label) for label in labels) diff --git a/dashboard/rendering/tables.py b/dashboard/rendering/tables.py index 362ec46..e9ad3cf 100644 --- a/dashboard/rendering/tables.py +++ b/dashboard/rendering/tables.py @@ -2,6 +2,7 @@ from __future__ import annotations +import html import math import numpy as np @@ -9,6 +10,7 @@ import polars as pl from dashboard.data_access import RunTableData, RunTables +from dashboard.rendering.labels import attach_full_tab_titles, display_label_map TableData = RunTables | RunTableData @@ -45,6 +47,43 @@ def format_numeric(value, *, precision: int | None = 2): return value +def format_fixed_decimal(value, *, decimal_places: int): + if value is None: + return None + if isinstance(value, (bool, np.bool_)): + return value + if isinstance(value, (int, np.integer, float, np.floating)): + number = float(value) + if not math.isfinite(number): + return None + return f"{number:.{max(decimal_places, 0)}f}" + return value + + +def format_fixed_decimal_frame( + frame: pl.DataFrame, + *, + decimal_places_by_column: dict[str, int] | None = None, +) -> pl.DataFrame: + if not decimal_places_by_column: + return frame + expressions = [] + for column, decimal_places in decimal_places_by_column.items(): + dtype = frame.schema.get(column) + if dtype is None or not getattr(dtype, "is_numeric", lambda: False)(): + continue + expressions.append( + pl.col(column).map_elements( + lambda value, places=decimal_places: format_fixed_decimal( + value, + decimal_places=places, + ), + return_dtype=pl.Utf8, + ).alias(column) + ) + return frame.with_columns(expressions) if expressions else frame + + def format_numeric_frame( frame: pl.DataFrame, *, @@ -93,6 +132,7 @@ def drop_index_columns(frame: pl.DataFrame) -> pl.DataFrame: _NAME_OVERRIDES = { "% Diff": "% Diff", "FACTYPE": "Facility Type", "From_Node": "From Node", "To_Node": "To Node", + "mean_nonzero": "Mean Non-Zero", } @@ -125,6 +165,24 @@ def column_titles(columns: list[object] | tuple[object, ...]) -> dict[str, str]: return titles +def column_title_metadata( + columns: list[object] | tuple[object, ...], +) -> tuple[dict[str, str], dict[str, str]]: + """Return compact column titles and full tooltips for truncated titles.""" + full_titles = column_titles(columns) + display_titles = display_label_map(full_titles.values()) + titles = { + column: display_titles[full_title] + for column, full_title in full_titles.items() + } + tooltips = { + column: full_title + for column, full_title in full_titles.items() + if titles[column] != full_title + } + return titles, tooltips + + def data_table( data: TableData, title: str = "", @@ -132,13 +190,21 @@ def data_table( numeric_precision: int | None = 2, numeric_precision_by_column: dict[str, int] | None = None, column_sorters: dict[str, str] | None = None, + numeric_decimal_places_by_column: dict[str, int] | None = None, ) -> pn.viewable.Viewable: + data = list(data) tabs = pn.Tabs() + full_labels = [str(label) for label, frame in data if not frame.is_empty()] + display_labels = display_label_map(full_labels) for label, frame in data: if frame.is_empty(): continue + full_label = str(label) display = format_numeric_frame( - drop_index_columns(frame), + format_fixed_decimal_frame( + drop_index_columns(frame), + decimal_places_by_column=numeric_decimal_places_by_column, + ), numeric_precision=numeric_precision, numeric_precision_by_column=numeric_precision_by_column, ) @@ -149,11 +215,23 @@ def data_table( for column, sorter in column_sorters.items() if str(column) in display.columns ] - tabs.append((label, pn.widgets.Tabulator( + titles, header_tooltips = column_title_metadata(display.columns) + table = pn.widgets.Tabulator( to_pandas(display), height=height, sizing_mode="stretch_width", - theme="simple", titles=column_titles(display.columns), + theme="simple", titles=titles, header_tooltips=header_tooltips, show_index=False, configuration=configuration, - ))) + ) + tab_content: pn.viewable.Viewable = table + if display_labels[full_label] != full_label: + tab_content = pn.Column( + pn.pane.HTML( + f'
    Run: ' + f"{html.escape(full_label)}
    " + ), + table, + ) + tabs.append((display_labels[full_label], tab_content)) + attach_full_tab_titles(tabs, full_labels) return pn.Column(pn.pane.Markdown(f"### {title}"), tabs) if title else tabs diff --git a/processor/cache_identity.py b/processor/cache_identity.py index a633086..13fe17c 100644 --- a/processor/cache_identity.py +++ b/processor/cache_identity.py @@ -33,24 +33,33 @@ def build_run_fingerprint( label: str, run_dir: str | None, skim_file: str | None, + raw_file_identities: dict[str, dict[str, object] | None] | None = None, + skim_file_identity: dict[str, object] | None = None, skimjoin: dict[str, object] | None = None, file_map: dict[str, str] | None = None, fallback_file_map: dict[str, str] | None = None, hh_weight_col: str | None, person_weight_col: str | None, trip_weight_col: str | None, + day_weight_col: str | None = "day_weight", ) -> dict[str, object]: """Return the run inputs that determine whether cache data is reusable.""" return { "label": label, "run_dir": str(run_dir) if run_dir is not None else None, "skim_file": str(skim_file) if skim_file is not None else None, + "skim_file_identity": skim_file_identity, + "raw_file_identities": { + key: value + for key, value in sorted((raw_file_identities or {}).items()) + }, "skimjoin": dict(sorted((skimjoin or {}).items())) if skimjoin else None, "file_map": dict(sorted((file_map or {}).items())), "fallback_file_map": dict(sorted((fallback_file_map or {}).items())), "hh_weight_col": hh_weight_col, "person_weight_col": person_weight_col, "trip_weight_col": trip_weight_col, + "day_weight_col": day_weight_col, } @@ -65,4 +74,20 @@ def file_identity(path: str | Path) -> dict[str, object]: } -__all__ = ["build_run_fingerprint", "build_run_keys", "file_identity", "slugify"] +def optional_file_identity(path: str | Path | None) -> dict[str, object] | None: + """Return a file identity when the configured input currently exists.""" + if path is None: + return None + resolved = Path(path).expanduser().resolve() + if not resolved.is_file(): + return None + return file_identity(resolved) + + +__all__ = [ + "build_run_fingerprint", + "build_run_keys", + "file_identity", + "optional_file_identity", + "slugify", +] diff --git a/processor/models.py b/processor/models.py index b034ce5..2eb8143 100644 --- a/processor/models.py +++ b/processor/models.py @@ -60,7 +60,7 @@ class RunData: - tours: ``tour_purpose``, ``tour_mode``, ``tour_category``, ``start_hour``, ``end_hour``, ``tourdur``, ``num_ob_stops``, ``num_ib_stops``, - ``num_tot_stops``, ``SKIMDIST``, ``AUTOSUFF``, ``NUMBER_HH``, + ``num_tot_stops``, ``tour_distance``, ``SKIMDIST``, ``AUTOSUFF``, ``NUMBER_HH``, ``finalweight`` - trips: ``tour_purpose``, ``trip_purpose``, ``tour_mode``, ``trip_mode``, ``depart_hour``, ``stops``, ``out_dir_dist``, ``od_dist``, @@ -85,6 +85,7 @@ class RunData: hh_weight_col: Optional[str] = None person_weight_col: Optional[str] = None trip_weight_col: Optional[str] = None + day_weight_col: Optional[str] = "day_weight" day: pl.DataFrame = field(default_factory=pl.DataFrame) vehicles: pl.DataFrame = field(default_factory=pl.DataFrame) trip_hypothetical_skims: pl.DataFrame = field(default_factory=pl.DataFrame) @@ -161,6 +162,7 @@ def mapped(table_name: str, frame: pl.DataFrame) -> pl.DataFrame: hh_weight_col=None if clear_weight_columns else run.hh_weight_col, person_weight_col=None if clear_weight_columns else run.person_weight_col, trip_weight_col=None if clear_weight_columns else run.trip_weight_col, + day_weight_col=None if clear_weight_columns else run.day_weight_col, table_availability_metadata=TableAvailabilityMetadata( states=dict(run.table_availability_metadata.states), diagnostics=dict(run.table_availability_metadata.diagnostics), @@ -210,6 +212,7 @@ def prune_prepared_run( hh_weight_col=prepared_run.hh_weight_col, person_weight_col=prepared_run.person_weight_col, trip_weight_col=prepared_run.trip_weight_col, + day_weight_col=prepared_run.day_weight_col, table_availability_metadata=TableAvailabilityMetadata( states=dict(prepared_run.table_availability_metadata.states), diagnostics=dict(prepared_run.table_availability_metadata.diagnostics), diff --git a/processor/prepare/cache.py b/processor/prepare/cache.py index f8c743a..c9c2934 100644 --- a/processor/prepare/cache.py +++ b/processor/prepare/cache.py @@ -20,6 +20,7 @@ build_run_fingerprint, build_run_keys, file_identity, + optional_file_identity, slugify, ) from processor.models import RunData @@ -32,8 +33,8 @@ from processor.prepare.writer import write_all from runtime.config import Config -SCHEMA_VERSION = 9 -SUPPORTED_SCHEMA_VERSIONS = {2, 3, 4, 5, 6, 7, 8, 9} +SCHEMA_VERSION = 11 +SUPPORTED_SCHEMA_VERSIONS = {2, 3, 4, 5, 6, 7, 8, 9, 10, 11} SUPPORTED_FILE_FORMATS = ("parquet", "csv") PREPARED_TABLE_ATTRS: tuple[tuple[str, str, str], ...] = ( ("hh", "households", "households"), @@ -90,9 +91,14 @@ def build_prepared_manifest_identity( } if source_type == "custom_prepared_table_map": normalized_table_map = dict(sorted((prepared_table_map or {}).items())) + sidecar_table_ids = {attr_name for attr_name, _ in SIDECAR_TABLE_ATTRS} identity["prepared_table_map"] = normalized_table_map identity["prepared_table_fingerprints"] = { - table_id: file_identity(path) + table_id: ( + optional_file_identity(path) + if table_id in sidecar_table_ids + else file_identity(path) + ) for table_id, path in normalized_table_map.items() } else: @@ -213,20 +219,24 @@ def _write_sidecar_tables( *, file_format: str, ) -> dict[str, str]: - sidecar_frames = { - attr_name: getattr(rd, attr_name) - for attr_name, _ in SIDECAR_TABLE_ATTRS - if isinstance(getattr(rd, attr_name), pl.DataFrame) - and not getattr(rd, attr_name).is_empty() - } + filenames = _sidecar_file_map(file_format) + sidecar_frames: dict[str, pl.DataFrame] = {} + for attr_name, _ in SIDECAR_TABLE_ATTRS: + frame = getattr(rd, attr_name) + if isinstance(frame, pl.DataFrame) and not frame.is_empty(): + sidecar_frames[attr_name] = frame + continue + + stale_path = cache_dir / filenames[attr_name] + if stale_path.is_file(): + stale_path.unlink() + if not sidecar_frames: return {} - sidecar_dir = cache_dir / "prepared_tables" - sidecar_dir.mkdir(parents=True, exist_ok=True) - filenames = _sidecar_file_map(file_format) + cache_dir.mkdir(parents=True, exist_ok=True) for attr_name, frame in sidecar_frames.items(): - path = sidecar_dir / filenames[attr_name] + path = cache_dir / filenames[attr_name] if file_format == "parquet": frame.write_parquet(path) elif file_format == "csv": @@ -244,6 +254,8 @@ def write_prepared_run_cache( output_root: str | Path | None = None, run_fingerprint: dict[str, object] | None = None, file_format: str | None = None, + cache_name: str = "prepared_tables", + prepare_config_digest: str | None = None, ) -> PreparedRunCacheEntry: """Write one prepared run's canonical tables and manifest.""" if file_format is None: @@ -259,7 +271,7 @@ def write_prepared_run_cache( ) output_root.mkdir(parents=True, exist_ok=True) - cache_dir = output_root / run_key / "prepared_tables" + cache_dir = output_root / run_key / cache_name cache_dir.mkdir(parents=True, exist_ok=True) tables_to_write: dict[str, pl.DataFrame] = {} @@ -277,7 +289,7 @@ def write_prepared_run_cache( tables_to_write[stem] = table write_all(tables_to_write, cache_dir, file_format=file_format) - sidecar_files = _write_sidecar_tables(cache_dir.parent, rd, file_format=file_format) + sidecar_files = _write_sidecar_tables(cache_dir, rd, file_format=file_format) _write_skimjoin_outputs(cache_dir, rd, config) manifest = { @@ -288,10 +300,10 @@ def write_prepared_run_cache( "run_key": run_key, "source_run_dir": rd.run_dir, "config_path": config.config_path, - "prepare_config_digest": config.prepare_config_digest, + "prepare_config_digest": prepare_config_digest or config.prepare_config_digest, "table_format": file_format, - "table_root": "prepared_tables", - "sidecar_root": "prepared_tables", + "table_root": cache_name, + "sidecar_root": cache_name, "table_files": _table_file_map(file_format), "sidecar_files": sidecar_files, "table_states": { @@ -335,7 +347,20 @@ def write_prepared_run_cache( "hh_weight_col": rd.hh_weight_col, "person_weight_col": rd.person_weight_col, "trip_weight_col": rd.trip_weight_col, + "day_weight_col": rd.day_weight_col, "run_fingerprint": run_fingerprint or {}, + "identity": { + "raw_inputs": dict((run_fingerprint or {}).get("raw_file_identities", {})), + "prepare_config": prepare_config_digest or config.prepare_config_digest, + "skimjoin_config": ( + dict((run_fingerprint or {}).get("skimjoin") or {}).get("config_digest") + ), + "skim_inputs": ( + dict((run_fingerprint or {}).get("skimjoin") or {}).get( + "resolved_skim_file_identities", [] + ) + ), + }, "prepare_diagnostics": dict(rd.prepare_diagnostics), "skimjoin_enabled": bool(rd.skimjoin_manifest.get("skimjoin_enabled", False)), "skimjoin_config_digest": rd.skimjoin_manifest.get("skimjoin_config_digest"), @@ -504,6 +529,7 @@ def load_prepared_run_cache( hh_weight_col=manifest.get("hh_weight_col"), person_weight_col=manifest.get("person_weight_col"), trip_weight_col=manifest.get("trip_weight_col"), + day_weight_col=manifest.get("day_weight_col", "day_weight"), prepare_diagnostics=dict(manifest.get("prepare_diagnostics", {})), skimjoin_manifest={ "skimjoin_enabled": bool(manifest.get("skimjoin_enabled", False)), @@ -547,6 +573,53 @@ def load_prepared_run_cache( ) +def inspect_prepared_run_cache( + cache_dir: str | Path, + *, + expected_prepare_config_digest: str | None = None, + expected_run_fingerprint: dict[str, object] | None = None, + expected_label: str | None = None, + expected_run_key: str | None = None, +) -> dict[str, object]: + """Validate a prepared cache identity without loading its tables.""" + cache_dir = Path(cache_dir) + manifest = read_manifest(cache_dir, error_cls=PreparedCacheError) + validate_schema_version( + cache_dir=cache_dir, + manifest=manifest, + supported_versions=SUPPORTED_SCHEMA_VERSIONS, + error_factory=lambda message: PreparedCacheError( + message.replace( + "Unsupported cache schema_version", + "Unsupported prepared cache schema_version", + ) + ), + ) + if expected_label is not None and manifest.get("label") != expected_label: + raise PreparedCacheError( + f"Prepared cache label mismatch in {cache_dir}: expected {expected_label!r}, found {manifest.get('label')!r}" + ) + if expected_run_key is not None and manifest.get("run_key") != expected_run_key: + raise PreparedCacheError( + f"Prepared cache run key mismatch in {cache_dir}: expected {expected_run_key!r}, found {manifest.get('run_key')!r}" + ) + if ( + expected_prepare_config_digest is not None + and manifest.get("prepare_config_digest") != expected_prepare_config_digest + ): + raise PreparedCacheError( + f"Prepared cache config digest mismatch in {cache_dir}; tables were built from a different preparation configuration." + ) + if ( + expected_run_fingerprint is not None + and manifest.get("run_fingerprint") != expected_run_fingerprint + ): + raise PreparedCacheError( + f"Prepared cache run fingerprint mismatch in {cache_dir}; tables were built from different run inputs." + ) + return manifest + + def discover_cache_dirs(root: str | Path) -> list[Path]: """Return child prepared-cache directories that contain a manifest.""" root = Path(root) @@ -575,6 +648,7 @@ def load_custom_prepared_tables( ) loaded_tables: dict[str, pl.DataFrame] = {} + sidecar_tables: dict[str, pl.DataFrame] = {} table_states: dict[str, str] = {} table_reasons: dict[str, str] = {} for attr_name, table_id, _ in PREPARED_TABLE_ATTRS: @@ -608,6 +682,29 @@ def load_custom_prepared_tables( loaded_tables[attr_name] = table table_states[table_id] = "empty" if table.width == 0 else "available" + for attr_name, _ in SIDECAR_TABLE_ATTRS: + configured_path = prepared_table_map.get(attr_name) + if configured_path is None: + sidecar_tables[attr_name] = pl.DataFrame() + continue + + path = Path(configured_path) + if not path.exists(): + sidecar_tables[attr_name] = pl.DataFrame() + table_states[attr_name] = "unavailable" + table_reasons[attr_name] = f"Missing prepared sidecar file: {path}" + continue + try: + sidecar_tables[attr_name] = _read_table_file(path) + except Exception as exc: + sidecar_tables[attr_name] = pl.DataFrame() + table_states[attr_name] = "failed" + table_reasons[attr_name] = str(exc) + continue + table_states[attr_name] = ( + "empty" if sidecar_tables[attr_name].width == 0 else "available" + ) + return attach_table_availability( RunData( label=label, @@ -619,6 +716,8 @@ def load_custom_prepared_tables( tours=loaded_tables["tours"], trips=loaded_tables["trips"], vehicles=loaded_tables["vehicles"], + trip_hypothetical_skims=sidecar_tables["trip_hypothetical_skims"], + tour_hypothetical_skims=sidecar_tables["tour_hypothetical_skims"], joint_participants=loaded_tables["joint_participants"], land_use=loaded_tables["land_use"], skim_matrix=None, diff --git a/processor/prepare/enrichment/canonicalize.py b/processor/prepare/enrichment/canonicalize.py index 6dba8a9..393a574 100644 --- a/processor/prepare/enrichment/canonicalize.py +++ b/processor/prepare/enrichment/canonicalize.py @@ -14,6 +14,164 @@ from runtime.config import Config +_CATEGORY_MAPPING_TABLE_ATTRIBUTES = { + "households": "hh", + "persons": "per", + "day": "day", + "tours": "tours", + "trips": "trips", + "vehicles": "vehicles", + "joint_tour_participants": "joint_participants", + "land_use": "land_use", +} + + +def _apply_prepare_category_mappings( + state: _PrepareState, config: Config +) -> _PrepareState: + run_mapping = config.prepare_category_mappings.mapping_for_run(state.label) + if run_mapping is None: + return state + + for table_name, column_mappings in run_mapping.items(): + attribute = _CATEGORY_MAPPING_TABLE_ATTRIBUTES[table_name] + table = getattr(state, attribute) + for target_column, spec in column_mappings.items(): + metric_id = f"category_mappings.{table_name}.{target_column}" + if spec.source_column not in table.columns: + state.prepare_diagnostics[metric_id] = { + "total": 0, + "unresolved": 0, + "status": "source_column_missing", + "source_column": spec.source_column, + } + continue + + source_text = pl.col(spec.source_column).cast(pl.Utf8) + if spec.output_type == "boolean": + preserve_source = ( + spec.preserve_unmapped + and table.schema.get(spec.source_column) == pl.Boolean + ) + default = ( + pl.col(spec.source_column) + if preserve_source + else pl.lit(None, dtype=pl.Boolean) + ) + mapped = source_text.replace_strict( + spec.mapping, + default=default, + return_dtype=pl.Boolean, + ) + else: + default = source_text if spec.preserve_unmapped else pl.lit(None) + mapped = source_text.replace_strict( + spec.mapping, + default=default, + return_dtype=pl.Utf8, + ) + + source_values = table.select( + source_text.alias("_source_value") + ).filter(pl.col("_source_value").is_not_null()) + unmatched = source_values.filter( + ~pl.col("_source_value").is_in(list(spec.mapping)) + ) + unmatched_values = ( + unmatched.select("_source_value") + .unique(maintain_order=True) + .head(20) + .to_series() + .to_list() + ) + state.prepare_diagnostics[metric_id] = { + "total": source_values.height, + "unresolved": unmatched.height, + "unresolved_share": ( + float(unmatched.height) / float(source_values.height) + if source_values.height + else 0.0 + ), + "source_column": spec.source_column, + "unmatched_values": unmatched_values, + } + table = table.with_columns(mapped.alias(target_column)) + setattr(state, attribute, table) + return state + + +def _toc_model_period_expr(hour_column: str, minute_column: str) -> pl.Expr: + hour = pl.col(hour_column).cast(pl.Int64, strict=False) + minute = pl.col(minute_column).cast(pl.Int64, strict=False) + minutes_after_3_am = (hour * 60 + minute - 180 + 1440) % 1440 + return ( + pl.when(hour.is_between(0, 23) & minute.is_between(0, 59)) + .then((minutes_after_3_am // 30) + 1) + .otherwise(None) + .cast(pl.Int32) + ) + + +def _is_toc_raw_survey(state: _PrepareState) -> bool: + return ( + {"num_people", "num_vehicles"}.issubset(state.hh.columns) + and {"can_drive", "person_type"}.issubset(state.per.columns) + and { + "tour_start_hour", + "tour_start_minute", + "tour_end_hour", + "tour_end_minute", + "tour_mode", + "tour_purpose", + "duration_minutes", + "distance_miles", + }.issubset(state.tours.columns) + and { + "linked_trip_mode", + "d_purpose_category", + "depart_hour", + "depart_minute", + "distance_miles", + }.issubset(state.trips.columns) + ) + + +def _normalize_toc_raw_survey( + state: _PrepareState, config: Config +) -> _PrepareState: + if not _is_toc_raw_survey(state): + return state + + state.hh = state.hh.with_columns( + pl.col("num_people").alias("hhsize"), + pl.col("num_vehicles").alias("auto_ownership"), + ) + state.tours = state.tours.with_columns( + _toc_model_period_expr("tour_start_hour", "tour_start_minute").alias("start"), + _toc_model_period_expr("tour_start_hour", "tour_start_minute").alias( + "start_hour" + ), + _toc_model_period_expr("tour_end_hour", "tour_end_minute").alias("end"), + _toc_model_period_expr("tour_end_hour", "tour_end_minute").alias("end_hour"), + pl.col("distance_miles").cast(pl.Float64, strict=False).alias("SKIMDIST"), + ) + if "joint_num_participants" in state.tours.columns: + state.tours = state.tours.with_columns( + pl.col("joint_num_participants") + .cast(pl.Int64, strict=False) + .alias("number_of_participants") + ) + + state.trips = state.trips.with_columns( + _toc_model_period_expr("depart_hour", "depart_minute").alias("depart"), + pl.col("distance_miles").cast(pl.Float64, strict=False).alias("od_dist"), + pl.col("distance_miles") + .cast(pl.Float64, strict=False) + .alias("prepared_non_motorized_distance"), + ) + return state + + def _canonicalize_households(hh: pl.DataFrame, config: Config) -> pl.DataFrame: hh = _materialize_column( hh, @@ -355,6 +513,7 @@ def _canonicalize_land_use(land_use: pl.DataFrame, config: Config) -> pl.DataFra def _canonicalize_identifiers_and_core_columns( state: _PrepareState, config: Config ) -> _PrepareState: + state = _apply_prepare_category_mappings(state, config) state.hh = _canonicalize_households(state.hh, config) state.per = _canonicalize_persons(state.per, config) state.day = _canonicalize_day(state.day, config) @@ -365,4 +524,4 @@ def _canonicalize_identifiers_and_core_columns( state.joint_participants, config ) state.land_use = _canonicalize_land_use(state.land_use, config) - return state + return _normalize_toc_raw_survey(state, config) diff --git a/processor/prepare/enrichment/finalize.py b/processor/prepare/enrichment/finalize.py index bf9893d..5839c9b 100644 --- a/processor/prepare/enrichment/finalize.py +++ b/processor/prepare/enrichment/finalize.py @@ -105,6 +105,7 @@ def _cast_tours(tours: pl.DataFrame) -> pl.DataFrame: "pnr_taz": pl.Int32, "income_segment": pl.Int64, "vot_bin": pl.Utf8, + "tour_distance": pl.Float64, "SKIMDIST": pl.Float64, "NUMBER_HH": pl.Int32, "AUTOSUFF": pl.Int32, diff --git a/processor/prepare/enrichment/households_persons.py b/processor/prepare/enrichment/households_persons.py index cc41db0..ac5f167 100644 --- a/processor/prepare/enrichment/households_persons.py +++ b/processor/prepare/enrichment/households_persons.py @@ -5,7 +5,10 @@ from runtime.logging import get_logger import polars as pl -from processor.prepare.enrichment.autosuff import derive_household_autosuff_counts +from processor.prepare.enrichment.autosuff import ( + _truthy_expr, + derive_household_autosuff_counts, +) from processor.prepare.enrichment.columns import _has_columns from processor.prepare.enrichment.types import _PrepareState, _ZoneContext from processor.prepare.enrichment.zones import ( @@ -23,6 +26,24 @@ NATIVE_HOME_GEOGRAPHY_COLUMNS = ("home_county", "home_mpo") +def _derive_external_worker_flag(state: _PrepareState) -> None: + """Derive the canonical external-worker flag from existing worker fields.""" + required = {"is_worker", "work_from_home", "external_worker_identification"} + if "is_external_worker" in state.per.columns or not required.issubset( + state.per.columns + ): + return + state.per = state.per.with_columns( + ( + _truthy_expr("is_worker") + & ~_truthy_expr("work_from_home") + & _truthy_expr("external_worker_identification") + ) + .fill_null(False) + .alias("is_external_worker") + ) + + def _add_configured_geographies( df: pl.DataFrame, *, @@ -150,6 +171,7 @@ def _enrich_persons( state: _PrepareState, config: Config, zone_context: _ZoneContext ) -> None: _derive_num_joint_tours(state) + _derive_external_worker_flag(state) if ( "home_zone_id" not in state.per.columns diff --git a/processor/prepare/enrichment/tours.py b/processor/prepare/enrichment/tours.py index c6ea30c..b60c9fa 100644 --- a/processor/prepare/enrichment/tours.py +++ b/processor/prepare/enrichment/tours.py @@ -25,10 +25,174 @@ LOGGER = get_logger("processor.prepare") +def _identifier_expr(column: str) -> pl.Expr: + return ( + pl.col(column) + .cast(pl.Float64, strict=False) + .cast(pl.Int64, strict=False) + .cast(pl.Utf8) + ) + + +def _canonical_tour_origins(tours: pl.DataFrame, hh: pl.DataFrame) -> pl.DataFrame: + if tours.is_empty() or "origin" not in tours.columns: + return tours + + result = tours + if {"household_id", "home_zone_id"}.issubset(hh.columns) and "household_id" in result.columns: + result = result.join( + hh.select( + "household_id", + pl.col("home_zone_id").alias("_home_tour_origin"), + ).unique("household_id"), + on="household_id", + how="left", + ) + else: + result = result.with_columns(pl.lit(None).alias("_home_tour_origin")) + + parent_origin_columns: list[str] = [] + for parent_id, tour_id in ( + ("parent_tour_id", "tour_id"), + ("survey_parent_tour_id", "survey_tour_id"), + ): + if not {parent_id, tour_id, "destination"}.issubset(tours.columns): + continue + key_col = f"_parent_key_{len(parent_origin_columns)}" + origin_col = f"_parent_tour_origin_{len(parent_origin_columns)}" + parent_destinations = ( + tours.filter(pl.col(tour_id).is_not_null()) + .select( + _identifier_expr(tour_id).alias(key_col), + pl.col("destination").alias(origin_col), + ) + .filter(pl.col(key_col).is_not_null()) + .unique(key_col) + ) + result = ( + result.with_columns(_identifier_expr(parent_id).alias(key_col)) + .join(parent_destinations, on=key_col, how="left") + .drop(key_col) + ) + parent_origin_columns.append(origin_col) + + category = ( + pl.col("tour_category").cast(pl.Utf8).str.strip_chars().str.to_lowercase() + if "tour_category" in result.columns + else pl.lit("") + ) + parent_origins = [pl.col(column) for column in parent_origin_columns] + parent_origins.append(pl.col("origin")) + result = result.with_columns( + pl.when(category.is_in(["atwork", "at_work"])) + .then(pl.coalesce(parent_origins)) + .otherwise(pl.coalesce("_home_tour_origin", "origin")) + .alias("origin") + ) + return result.drop(["_home_tour_origin", *parent_origin_columns]) + + +def _derive_joint_composition( + tours: pl.DataFrame, + joint_participants: pl.DataFrame, + persons: pl.DataFrame, +) -> pl.DataFrame: + person_type_col = next( + ( + column + for column in ("person_type", "ptype", "PERTYPE") + if column in persons.columns + ), + None, + ) + age_col = next( + (column for column in ("age", "age_app", "age_fv") if column in persons.columns), + None, + ) + if ( + tours.is_empty() + or joint_participants.is_empty() + or (person_type_col is None and age_col is None) + or "person_id" not in persons.columns + or "person_id" not in joint_participants.columns + ): + return tours + + adult_sources: list[pl.Expr] = [] + if person_type_col is not None: + person_type = pl.col(person_type_col).cast(pl.Int64, strict=False) + adult_sources.append( + pl.when(person_type.is_between(1, 5)) + .then(pl.lit(True)) + .when(person_type.is_between(6, 8)) + .then(pl.lit(False)) + .otherwise(None) + ) + if age_col is not None: + age = pl.col(age_col).cast(pl.Float64, strict=False) + adult_sources.append( + pl.when(age.is_between(0, 994)).then(age >= 18).otherwise(None) + ) + + participant_adulthood = joint_participants.join( + persons.select( + "person_id", + pl.coalesce(adult_sources).alias("_participant_is_adult"), + ).unique("person_id"), + on="person_id", + how="left", + ).filter(pl.col("_participant_is_adult").is_not_null()) + + result = tours + derived_columns: list[str] = [] + for identifier in ("joint_tour_id", "tour_id"): + if identifier not in result.columns or identifier not in participant_adulthood.columns: + continue + key_col = f"_composition_key_{len(derived_columns)}" + composition_col = f"_derived_composition_{len(derived_columns)}" + compositions = ( + participant_adulthood.filter(pl.col(identifier).is_not_null()) + .with_columns(_identifier_expr(identifier).alias(key_col)) + .filter(pl.col(key_col).is_not_null()) + .unique([key_col, "person_id"]) + .group_by(key_col) + .agg( + pl.col("_participant_is_adult").min().alias("_all_adults"), + pl.col("_participant_is_adult").max().alias("_any_adults"), + ) + .with_columns( + pl.when(pl.col("_all_adults")) + .then(pl.lit("adults")) + .when(~pl.col("_any_adults")) + .then(pl.lit("children")) + .otherwise(pl.lit("mixed")) + .alias(composition_col) + ) + .select(key_col, composition_col) + ) + result = ( + result.with_columns(_identifier_expr(identifier).alias(key_col)) + .join(compositions, on=key_col, how="left") + .drop(key_col) + ) + derived_columns.append(composition_col) + + if not derived_columns: + return result + sources = [] + if "composition" in result.columns: + sources.append(pl.col("composition").cast(pl.Utf8)) + sources.extend(pl.col(column) for column in derived_columns) + return result.with_columns(pl.coalesce(sources).alias("composition")).drop( + derived_columns + ) + + def _enrich_tours( state: _PrepareState, config: Config, zone_context: _ZoneContext ) -> _PrepareState: autosuff_ref_col = autosuff_reference_column(config) + state.tours = _canonical_tour_origins(state.tours, state.hh) hh_income_source = _resolve_source_column(state.hh, config.col_income_segment) if hh_income_source is not None: hh_income = state.hh.select( @@ -188,26 +352,51 @@ def _enrich_tours( state.label, ) + if "SKIMDIST" in state.tours.columns: + if "tour_distance" in state.tours.columns: + state.tours = state.tours.with_columns( + pl.coalesce( + pl.col("tour_distance").cast(pl.Float64, strict=False), + pl.col("SKIMDIST"), + ).alias("tour_distance") + ) + + number_hh_sources: list[pl.Expr] = [] + participant_count_col = "_joint_participant_count" if ( "tour_id" in state.tours.columns and "tour_id" in state.joint_participants.columns and state.joint_participants.schema.get("tour_id") != pl.Null ): party_size = state.joint_participants.group_by("tour_id").agg( - pl.len().alias("NUMBER_HH") + pl.len().cast(pl.Int64).alias(participant_count_col) ) state.tours = state.tours.join(party_size, on="tour_id", how="left") - if "NUMBER_HH" not in state.tours.columns: - state.tours = state.tours.with_columns(pl.lit(1).alias("NUMBER_HH")) - state.tours = state.tours.with_columns(pl.col("NUMBER_HH").fill_null(1)) + number_hh_sources.append(pl.col(participant_count_col)) + if "number_of_participants" in state.tours.columns: + number_hh_sources.append( + pl.col("number_of_participants").cast(pl.Int64, strict=False) + ) + if "NUMBER_HH" in state.tours.columns: + number_hh_sources.append(pl.col("NUMBER_HH").cast(pl.Int64, strict=False)) + number_hh_sources.append(pl.lit(1, dtype=pl.Int64)) + state.tours = state.tours.with_columns( + pl.coalesce(number_hh_sources).alias("NUMBER_HH") + ) + if participant_count_col in state.tours.columns: + state.tours = state.tours.drop(participant_count_col) - if ( - "start_hour" in state.tours.columns - and "end_hour" in state.tours.columns - and "tourdur" not in state.tours.columns - ): + state.tours = _derive_joint_composition( + state.tours, + state.joint_participants, + state.per, + ) + + if "start_hour" in state.tours.columns and "end_hour" in state.tours.columns: state.tours = state.tours.with_columns( - (pl.col("end_hour") - pl.col("start_hour")).alias("tourdur") + (pl.col("end_hour") - pl.col("start_hour") + 1) + .clip(1, 48) + .alias("tourdur") ) state.tours = _attach_first_inbound_trip_depart(state.tours, state.trips) diff --git a/processor/prepare/enrichment/trips.py b/processor/prepare/enrichment/trips.py index f5806bd..cd84f7e 100644 --- a/processor/prepare/enrichment/trips.py +++ b/processor/prepare/enrichment/trips.py @@ -233,7 +233,7 @@ def _escort_link_targets( child_trip_purpose: str, target_col: str, ) -> pl.DataFrame: - if escorted_ids_col not in tours.columns: + if escorted_ids_col not in tours.columns or target_col not in trips.columns: return pl.DataFrame( schema={"tour_id": pl.Int64, "_target_values": pl.List(pl.Int64)} ) @@ -394,6 +394,7 @@ def _select_direction_event( has_targets = ( "_target_values" in candidates.columns and candidates.schema.get("_target_values") == pl.List(pl.Int64) + and location_col in candidates.columns ) location_match_expr = ( pl.col(location_col).cast(pl.Int64, strict=False).is_in(pl.col("_target_values")) @@ -609,6 +610,45 @@ def _enrich_trips( config=config, metric_id="trips.AUTOSUFF", ) + + boundary_required = {"tour_id", "origin", "destination", "outbound", "trip_num"} + if boundary_required.issubset(state.trips.columns) and { + "tour_id", + "origin", + }.issubset(state.tours.columns): + boundaries = state.trips.group_by(["tour_id", "outbound"]).agg( + pl.col("trip_num").min().alias("_first_trip_num"), + pl.col("trip_num").max().alias("_last_trip_num"), + ) + state.trips = ( + state.trips.join(boundaries, on=["tour_id", "outbound"], how="left") + .join( + state.tours.select( + "tour_id", + pl.col("origin").alias("_tour_origin"), + ).unique("tour_id"), + on="tour_id", + how="left", + ) + .with_columns( + pl.when( + pl.col("outbound").cast(pl.Boolean, strict=False) + & (pl.col("trip_num") == pl.col("_first_trip_num")) + ) + .then(pl.coalesce("_tour_origin", "origin")) + .otherwise(pl.col("origin")) + .alias("origin"), + pl.when( + ~pl.col("outbound").cast(pl.Boolean, strict=False) + & (pl.col("trip_num") == pl.col("_last_trip_num")) + ) + .then(pl.coalesce("_tour_origin", "destination")) + .otherwise(pl.col("destination")) + .alias("destination"), + ) + .drop("_first_trip_num", "_last_trip_num", "_tour_origin") + ) + if config.use_maz and {"origin", "destination"}.issubset(state.trips.columns): state.trips = state.trips.with_columns( pl.col("origin").alias("o_maz"), diff --git a/processor/prepare/enrichment/types.py b/processor/prepare/enrichment/types.py index 0012dec..cb35cbe 100644 --- a/processor/prepare/enrichment/types.py +++ b/processor/prepare/enrichment/types.py @@ -33,6 +33,7 @@ class _PrepareState: hh_weight_col: str | None person_weight_col: str | None trip_weight_col: str | None + day_weight_col: str | None table_states: dict[str, str] table_reasons: dict[str, str] prepare_diagnostics: dict[str, object] @@ -57,6 +58,7 @@ def from_run(cls, run: RunData) -> "_PrepareState": hh_weight_col=run.hh_weight_col, person_weight_col=run.person_weight_col, trip_weight_col=run.trip_weight_col, + day_weight_col=run.day_weight_col, table_states=table_availability(run), table_reasons=table_unavailable_reasons(run), prepare_diagnostics=dict(run.prepare_diagnostics), @@ -82,6 +84,7 @@ def to_run(self) -> RunData: hh_weight_col=self.hh_weight_col, person_weight_col=self.person_weight_col, trip_weight_col=self.trip_weight_col, + day_weight_col=self.day_weight_col, prepare_diagnostics=dict(self.prepare_diagnostics), ), table_states=self.table_states, diff --git a/processor/prepare/enrichment/weights.py b/processor/prepare/enrichment/weights.py index 4c437e3..63d2457 100644 --- a/processor/prepare/enrichment/weights.py +++ b/processor/prepare/enrichment/weights.py @@ -137,6 +137,8 @@ def _compute_day_weights( day: pl.DataFrame, per: pl.DataFrame, hh: pl.DataFrame, + *, + day_weight_col: str | None, ) -> pl.DataFrame: result = day if _has_columns(day, "person_id") and _has_columns(per, "person_id", "finalweight"): @@ -153,11 +155,11 @@ def _compute_day_weights( how="left", ) - if "day_weight" in day.columns: + if day_weight_col and day_weight_col in day.columns: result = result.with_columns( pl.coalesce( [ - pl.col("day_weight").cast(pl.Float64), + pl.col(day_weight_col).cast(pl.Float64), pl.col("_pw") if "_pw" in result.columns else pl.lit(None), pl.col("_hw") if "_hw" in result.columns else pl.lit(None), pl.lit(1.0), @@ -265,6 +267,7 @@ def compute_weights( hh_weight_col: Optional[str] = None, person_weight_col: Optional[str] = None, trip_weight_col: Optional[str] = None, + day_weight_col: Optional[str] = "day_weight", ) -> tuple[pl.DataFrame, pl.DataFrame, pl.DataFrame, pl.DataFrame, pl.DataFrame, pl.DataFrame]: """Compute and attach ``finalweight`` to prepared tables.""" explicit_weight_supplied = any([hh_weight_col, person_weight_col, trip_weight_col]) @@ -285,7 +288,12 @@ def compute_weights( hh, trip_weight_col=trip_weight_col, ) - day = _compute_day_weights(day, per, hh) + day = _compute_day_weights( + day, + per, + hh, + day_weight_col=day_weight_col, + ) tours = _compute_tour_weights( tours, trips, @@ -309,6 +317,7 @@ def _apply_weights(state: _PrepareState, config: Config) -> _PrepareState: hh_weight_col=state.hh_weight_col, person_weight_col=state.person_weight_col, trip_weight_col=state.trip_weight_col, + day_weight_col=state.day_weight_col, ) LOGGER.info("[prepare_data] Weights ready for '%s'", state.label) return state diff --git a/processor/prepare/reader.py b/processor/prepare/reader.py index 2418d0f..c2c4cd8 100644 --- a/processor/prepare/reader.py +++ b/processor/prepare/reader.py @@ -49,6 +49,34 @@ def resolve_run_file_map( return effective +def resolve_run_file_paths( + run_dir: str | Path, + config: Config, + run_file_map: dict[str, str] | None = None, +) -> dict[str, str | None]: + """Resolve the concrete raw input files the reader would use.""" + root = Path(run_dir).expanduser() + resolved: dict[str, str | None] = {} + for table_id, configured in resolve_run_file_map(config, run_file_map).items(): + configured_path = Path(configured) + suffix = configured_path.suffix.lower() + candidates = ( + [root / configured_path] + if suffix in {".csv", ".parquet"} + else [ + root / f"{configured_path.name}.parquet", + root / f"{configured_path.name}.csv", + ] + ) + selected = next((candidate for candidate in candidates if candidate.is_file()), None) + if selected is None: + fallback = config.fallback_files.get(table_id) + fallback_path = Path(fallback).expanduser() if fallback else None + selected = fallback_path if fallback_path is not None and fallback_path.is_file() else None + resolved[table_id] = str(selected.resolve()) if selected is not None else None + return resolved + + def _find_and_read(run_dir: Path, configured: str) -> pl.DataFrame: """Read a table from run_dir, resolving file format.""" path = Path(configured) @@ -96,6 +124,7 @@ def read_run( hh_weight_col: Optional[str] = None, person_weight_col: Optional[str] = None, trip_weight_col: Optional[str] = None, + day_weight_col: Optional[str] = "day_weight", ) -> RunData: """Read ActivitySim outputs and optionally the OMX skim for one run.""" run_dir = Path(run_dir) @@ -211,10 +240,17 @@ def _read(key: str) -> pl.DataFrame: hh_weight_col=hh_weight_col or None, person_weight_col=person_weight_col or None, trip_weight_col=trip_weight_col or None, + day_weight_col=day_weight_col, ), table_states=table_states, table_reasons=table_reasons, ) -__all__ = ["RunData", "read_run", "resolve_run_file_map", "resolve_skim_path"] +__all__ = [ + "RunData", + "read_run", + "resolve_run_file_map", + "resolve_run_file_paths", + "resolve_skim_path", +] diff --git a/processor/segmentation.py b/processor/segmentation.py index aa27be4..9ce7113 100644 --- a/processor/segmentation.py +++ b/processor/segmentation.py @@ -534,6 +534,7 @@ def _slice_run_data_from_source_subset( hh_weight_col=prepared_run.hh_weight_col, person_weight_col=prepared_run.person_weight_col, trip_weight_col=prepared_run.trip_weight_col, + day_weight_col=prepared_run.day_weight_col, table_availability_metadata=prepared_run.table_availability_metadata, skimjoin_artifacts=prepared_run.skimjoin_artifacts, skimjoin_manifest=prepared_run.skimjoin_manifest, @@ -608,6 +609,7 @@ def _copy_run_data( hh_weight_col=prepared_run.hh_weight_col, person_weight_col=prepared_run.person_weight_col, trip_weight_col=prepared_run.trip_weight_col, + day_weight_col=prepared_run.day_weight_col, table_availability_metadata=prepared_run.table_availability_metadata, skimjoin_artifacts=prepared_run.skimjoin_artifacts, skimjoin_manifest=prepared_run.skimjoin_manifest, diff --git a/processor/skimjoin/annotate/engine.py b/processor/skimjoin/annotate/engine.py index db87e76..0923aeb 100644 --- a/processor/skimjoin/annotate/engine.py +++ b/processor/skimjoin/annotate/engine.py @@ -11,6 +11,7 @@ _apply_output_columns, _apply_rule_sentinel_values, _build_lookup_summary_frame, + _combined_output_values, _concat_missing_frames, _empty_lookup_results_frame, _missing_report_schema, @@ -27,6 +28,32 @@ from processor.skimjoin.skimstore.base import SkimStore +def lookup_output_values( + source_table: pl.DataFrame, + *, + rules: list[NormalizedLookupRule], + normalized: NormalizedConfig, + inventory: pl.DataFrame, + mode_column: str, + skim_store: SkimStore | None = None, +) -> pl.DataFrame: + skim_store = skim_store or SkimStore() + _plan_csv_tables(skim_store, inventory, normalized, rules) + if "_row_id" not in source_table.columns: + raise ValueError("source_table must include _row_id.") + if mode_column not in source_table.columns: + return _combined_output_values(pl.DataFrame()) + + resolved_results = _resolved_lookup_values_without_reports( + rules=rules, + mode_subsets=_partition_trips_by_mode(source_table, mode_column), + inventory=inventory, + normalized=normalized, + skim_store=skim_store, + ) + return _combined_output_values(resolved_results) + + def annotate_lookup_table( base_table: pl.DataFrame, *, @@ -37,11 +64,14 @@ def annotate_lookup_table( mode_column: str, skim_store: SkimStore | None = None, include_fallback_report: bool = False, + collect_reports: bool = True, table_name: str = "trips", ) -> tuple[pl.DataFrame, pl.DataFrame, pl.DataFrame] | tuple[ pl.DataFrame, pl.DataFrame, pl.DataFrame, pl.DataFrame ]: skim_store = skim_store or SkimStore() + _plan_csv_tables(skim_store, inventory, normalized, rules) + collect_reports = collect_reports or include_fallback_report if "_row_id" not in base_table.columns: base_table = base_table.with_row_index("_row_id") source_table = source_table if source_table is not None else base_table @@ -55,6 +85,17 @@ def annotate_lookup_table( ) mode_subsets = _partition_trips_by_mode(source_table, mode_column) + if not collect_reports: + resolved_results = _resolved_lookup_values_without_reports( + rules=rules, + mode_subsets=mode_subsets, + inventory=inventory, + normalized=normalized, + skim_store=skim_store, + ) + annotated = _apply_output_columns(base_table, resolved_results) + return annotated.drop("_row_id"), pl.DataFrame(), pl.DataFrame() + work_item_frames, missing_frames, attempt_failure_frames, final_failure_frames = ( _rule_work_items_and_errors( rules=rules, @@ -84,14 +125,14 @@ def annotate_lookup_table( resolved_results = _select_resolved_chain_results(results) annotated = _apply_output_columns(base_table, resolved_results) missing_report = _concat_missing_frames(missing_frames) - fallback_report = _build_fallback_lookup_report( - resolved_results=resolved_results, - attempt_failure_frames=attempt_failure_frames, - final_failure_frames=final_failure_frames, - rule_by_name=rule_by_name, - table_name=table_name, - ) if include_fallback_report: + fallback_report = _build_fallback_lookup_report( + resolved_results=resolved_results, + attempt_failure_frames=attempt_failure_frames, + final_failure_frames=final_failure_frames, + rule_by_name=rule_by_name, + table_name=table_name, + ) return ( annotated.drop("_row_id"), lookup_summary, @@ -101,6 +142,69 @@ def annotate_lookup_table( return annotated.drop("_row_id"), lookup_summary, missing_report +def _plan_csv_tables( + skim_store: SkimStore, + inventory: pl.DataFrame, + normalized: NormalizedConfig, + rules: list[NormalizedLookupRule], +) -> None: + all_rules = [*normalized.trip_lookups, *normalized.tour_lookups, *rules] + skim_store.plan_csv_tables( + inventory, + matrix_templates={rule.matrix for rule in all_rules}, + ) + + +def _resolved_lookup_values_without_reports( + *, + rules: list[NormalizedLookupRule], + mode_subsets: dict[str, pl.DataFrame], + inventory: pl.DataFrame, + normalized: NormalizedConfig, + skim_store: SkimStore, +) -> pl.DataFrame: + metadata = _inventory_metadata_frame(inventory, normalized) + rule_by_name = {rule.name: rule for rule in rules} + resolved_frames: list[pl.DataFrame] = [] + + for chain_rules in _group_rules_by_chain(rules): + chain_work_items, _, _ = _resolve_chain_work_items( + chain_rules=chain_rules, + mode_subsets=mode_subsets, + collect_reports=False, + ) + if not chain_work_items: + continue + chain_queue = ( + chain_work_items[0] + if len(chain_work_items) == 1 + else pl.concat(chain_work_items, how="vertical") + ) + chain_results, _, _ = _execute_chain_queue( + chain_queue=chain_queue, + metadata=metadata, + rule_by_name=rule_by_name, + normalized=normalized, + skim_store=skim_store, + collect_reports=False, + ) + if not chain_results: + continue + resolved = _select_resolved_chain_results( + chain_results[0] + if len(chain_results) == 1 + else pl.concat(chain_results, how="vertical_relaxed") + ) + if not resolved.is_empty(): + resolved_frames.append( + resolved.select("_row_id", "output", "combine_method", "value") + ) + + if not resolved_frames: + return pl.DataFrame() + return pl.concat(resolved_frames, how="vertical_relaxed") + + def _missing_mode_response( *, base_table: pl.DataFrame, @@ -131,6 +235,7 @@ def _rule_work_items_and_errors( chain_work_items, chain_missing, chain_failures = _resolve_chain_work_items( chain_rules=chain_rules, mode_subsets=mode_subsets, + collect_reports=True, ) work_item_frames.extend(chain_work_items) attempt_failure_frames.extend(chain_failures) @@ -172,6 +277,7 @@ def _executed_lookup_results( rule_by_name=rule_by_name, normalized=normalized, skim_store=skim_store, + collect_reports=True, ) result_frames.extend(chain_results) attempt_failure_frames.extend(chain_failures) @@ -197,6 +303,7 @@ def _resolve_chain_work_items( *, chain_rules: list[NormalizedLookupRule], mode_subsets: dict[str, pl.DataFrame], + collect_reports: bool, ) -> tuple[list[pl.DataFrame], pl.DataFrame, list[pl.DataFrame]]: if not chain_rules: return [], _empty_missing_detail_report(), [] @@ -213,17 +320,22 @@ def _resolve_chain_work_items( for rule in chain_rules: missing_columns = _missing_trip_columns_for_rule(subset, rule) if missing_columns: - failure_history.append( - _chain_failure_frame_for_rows( - subset, - rule=rule, - reason=f"missing_trip_column:{missing_columns[0]}", + if collect_reports: + failure_history.append( + _chain_failure_frame_for_rows( + subset, + rule=rule, + reason=f"missing_trip_column:{missing_columns[0]}", + ) ) - ) continue - valid_items, resolution_errors = _resolve_rule_work_items(rule, subset) - if not resolution_errors.is_empty(): + valid_items, resolution_errors = _resolve_rule_work_items( + rule, + subset, + include_errors=collect_reports, + ) + if collect_reports and not resolution_errors.is_empty(): failure_history.append( _decorate_failure_frame(resolution_errors, rule=rule) ) @@ -231,7 +343,11 @@ def _resolve_chain_work_items( continue work_item_frames.append(valid_items) - planned_row_ids.append(valid_items.select("_row_id").unique()) + if collect_reports: + planned_row_ids.append(valid_items.select("_row_id").unique()) + + if not collect_reports: + return work_item_frames, _empty_missing_detail_report(), [] unresolved = ( subset.join( @@ -257,6 +373,7 @@ def _execute_chain_queue( rule_by_name: dict[str, NormalizedLookupRule], normalized: NormalizedConfig, skim_store: SkimStore, + collect_reports: bool, ) -> tuple[list[pl.DataFrame], pl.DataFrame, list[pl.DataFrame]]: result_frames: list[pl.DataFrame] = [] failure_history: list[pl.DataFrame] = [] @@ -277,22 +394,33 @@ def _execute_chain_queue( queued = step_queue.join(metadata, on="matrix_name", how="left") - missing_matrix = queued.filter(pl.col("file_path").is_null()) - if not missing_matrix.is_empty(): - for row in missing_matrix.select( - ["rule_name", "matrix_name"] - ).unique(maintain_order=True).iter_rows(named=True): - rule = rule_by_name[str(row["rule_name"])] - group = missing_matrix.filter( - (pl.col("rule_name") == row["rule_name"]) - & (pl.col("matrix_name") == row["matrix_name"]) - ) - failure_history.append( - _decorate_failure_frame( - _missing_matrix_frame(rule, str(row["matrix_name"]), group), - rule=rule, + ambiguous = queued.filter(pl.col("source_kind") == "ambiguous") + if not ambiguous.is_empty(): + row = ambiguous.select( + ["matrix_name", "ambiguous_sources"] + ).unique(maintain_order=True).row(0, named=True) + raise ValueError( + f"Ambiguous matrix reference {row['matrix_name']!r}; qualify it with one of: " + f"{row['ambiguous_sources']}" + ) + + if collect_reports: + missing_matrix = queued.filter(pl.col("file_path").is_null()) + if not missing_matrix.is_empty(): + for row in missing_matrix.select( + ["rule_name", "matrix_name"] + ).unique(maintain_order=True).iter_rows(named=True): + rule = rule_by_name[str(row["rule_name"])] + group = missing_matrix.filter( + (pl.col("rule_name") == row["rule_name"]) + & (pl.col("matrix_name") == row["matrix_name"]) + ) + failure_history.append( + _decorate_failure_frame( + _missing_matrix_frame(rule, str(row["matrix_name"]), group), + rule=rule, + ) ) - ) executable = queued.filter(pl.col("file_path").is_not_null()) if executable.is_empty(): @@ -310,9 +438,10 @@ def _execute_chain_queue( step_results = _apply_rule_sentinel_values(step_results, rule_by_name) result_frames.append(step_results) - invalid = step_results.filter(~pl.col("valid")) - if not invalid.is_empty(): - failure_history.append(_invalid_chain_failure_frame(invalid)) + if collect_reports: + invalid = step_results.filter(~pl.col("valid")) + if not invalid.is_empty(): + failure_history.append(_invalid_chain_failure_frame(invalid)) resolved_row_ids = ( step_results.filter(pl.col("valid")).select("_row_id").unique() @@ -320,6 +449,8 @@ def _execute_chain_queue( if not resolved_row_ids.is_empty(): unresolved = unresolved.join(resolved_row_ids, on="_row_id", how="anti") + if not collect_reports: + return result_frames, _empty_missing_detail_report(), [] return ( result_frames, _finalize_chain_failures(failure_history, unresolved), diff --git a/processor/skimjoin/annotate/tours.py b/processor/skimjoin/annotate/tours.py index f29106f..df9ec00 100644 --- a/processor/skimjoin/annotate/tours.py +++ b/processor/skimjoin/annotate/tours.py @@ -2,9 +2,12 @@ import polars as pl -from processor.skimjoin.annotate.engine import annotate_lookup_table +from processor.skimjoin.annotate.engine import ( + annotate_lookup_table, + lookup_output_values, +) from processor.skimjoin.config.normalize import TOUR_DIRECTION_COLUMN -from processor.skimjoin.config.schema import NormalizedConfig +from processor.skimjoin.config.schema import NormalizedConfig, NormalizedLookupRule from processor.skimjoin.skimstore.base import SkimStore @@ -15,6 +18,9 @@ def annotate_tours( trips: pl.DataFrame | None = None, skim_store: SkimStore | None = None, include_fallback_report: bool = False, + *, + rules: list[NormalizedLookupRule] | None = None, + collect_reports: bool = True, ) -> ( tuple[pl.DataFrame, pl.DataFrame, pl.DataFrame] | tuple[pl.DataFrame, pl.DataFrame, pl.DataFrame, pl.DataFrame] @@ -40,16 +46,43 @@ def annotate_tours( return annotate_lookup_table( tours_with_ids, source_table=contexts, - rules=normalized.tour_lookups, + rules=rules if rules is not None else normalized.tour_lookups, normalized=normalized, inventory=inventory, mode_column=normalized.activitysim.tour_mode_column, skim_store=skim_store, include_fallback_report=include_fallback_report, + collect_reports=collect_reports, table_name="tours", ) +def lookup_tour_output_values( + tours: pl.DataFrame, + normalized: NormalizedConfig, + inventory: pl.DataFrame, + *, + rules: list[NormalizedLookupRule], + skim_store: SkimStore | None = None, +) -> pl.DataFrame: + tours_with_ids = tours.with_row_index("_row_id") + contexts = pl.concat( + [ + _directional_tour_context(tours_with_ids, normalized, outbound=True), + _directional_tour_context(tours_with_ids, normalized, outbound=False), + ], + how="vertical_relaxed", + ) + return lookup_output_values( + contexts, + rules=rules, + normalized=normalized, + inventory=inventory, + mode_column=normalized.activitysim.tour_mode_column, + skim_store=skim_store, + ) + + def _directional_tour_context( tours: pl.DataFrame, normalized: NormalizedConfig, diff --git a/processor/skimjoin/annotate/trip_lookup_execution.py b/processor/skimjoin/annotate/trip_lookup_execution.py index 1bcd735..e141843 100644 --- a/processor/skimjoin/annotate/trip_lookup_execution.py +++ b/processor/skimjoin/annotate/trip_lookup_execution.py @@ -1,8 +1,11 @@ from __future__ import annotations +from collections import Counter + import polars as pl from processor.skimjoin.config.schema import NormalizedConfig, NormalizedLookupRule +from processor.skimjoin.inventory import qualified_matrix_reference from processor.skimjoin.skimstore.base import SkimStore from processor.skimjoin.annotate.trip_lookup_reports import _row_trip_id @@ -23,11 +26,48 @@ def _inventory_metadata_frame( "destination_column_name", ] ) - rows = selected.to_dicts() - for row in rows: + inventory_rows = selected.to_dicts() + name_counts = Counter(str(row["matrix_name"]) for row in inventory_rows) + rows: list[dict[str, object]] = [] + ambiguous_sources: dict[str, list[str]] = {} + for inventory_row in inventory_rows: + row = dict(inventory_row) row["lookup_name"] = normalized.zone_mapping.resolve_lookup_name( str(row["file_path"]) ) + row["ambiguous_sources"] = None + qualified = dict(row) + qualified["matrix_name"] = qualified_matrix_reference( + str(row["file_path"]), str(row["matrix_name"]) + ) + rows.append(qualified) + matrix_name = str(row["matrix_name"]) + if name_counts[matrix_name] == 1: + rows.append(row) + else: + ambiguous_sources.setdefault(matrix_name, []).append( + str(row["file_path"]) + ) + for matrix_name, file_paths in ambiguous_sources.items(): + rows.append( + { + "matrix_name": matrix_name, + "file_path": None, + "matrix_path": None, + "source_kind": "ambiguous", + "key_column_name": None, + "value_column_name": None, + "origin_column_name": None, + "destination_column_name": None, + "lookup_name": None, + "ambiguous_sources": ", ".join( + sorted( + qualified_matrix_reference(file_path, matrix_name) + for file_path in file_paths + ) + ), + } + ) return pl.DataFrame(rows, infer_schema_length=None) diff --git a/processor/skimjoin/annotate/trip_lookup_reports.py b/processor/skimjoin/annotate/trip_lookup_reports.py index a7783b1..3f98f5f 100644 --- a/processor/skimjoin/annotate/trip_lookup_reports.py +++ b/processor/skimjoin/annotate/trip_lookup_reports.py @@ -49,16 +49,7 @@ def _apply_output_columns( if results.is_empty(): return trips - outputs_long = ( - results.group_by(["_row_id", "output", "combine_method"], maintain_order=True) - .agg( - pl.when(pl.col("combine_method").first() == "sum") - .then(pl.col("value").sum()) - .otherwise(pl.col("value").first()) - .alias("value") - ) - .select("_row_id", "output", "value") - ) + outputs_long = _combined_output_values(results) outputs_wide = ( outputs_long.pivot( @@ -84,6 +75,23 @@ def _apply_output_columns( return trips.join(outputs_wide, on="_row_id", how="left") +def _combined_output_values(results: pl.DataFrame) -> pl.DataFrame: + if results.is_empty(): + return pl.DataFrame( + schema={"_row_id": pl.Int64, "output": pl.String, "value": pl.Float64} + ) + return ( + results.group_by(["_row_id", "output", "combine_method"], maintain_order=True) + .agg( + pl.when(pl.col("combine_method").first() == "sum") + .then(pl.col("value").sum()) + .otherwise(pl.col("value").first()) + .alias("value") + ) + .select("_row_id", "output", "value") + ) + + def _empty_lookup_results_frame() -> pl.DataFrame: return pl.DataFrame( schema={ diff --git a/processor/skimjoin/annotate/trip_work_items.py b/processor/skimjoin/annotate/trip_work_items.py index c52c9bd..6618e1a 100644 --- a/processor/skimjoin/annotate/trip_work_items.py +++ b/processor/skimjoin/annotate/trip_work_items.py @@ -12,6 +12,8 @@ def _resolve_rule_work_items( rule: NormalizedLookupRule, subset: pl.DataFrame, + *, + include_errors: bool = True, ) -> tuple[pl.DataFrame, pl.DataFrame]: work = _base_rule_work_frame(rule, subset) missing_reason_expr = pl.lit(None, dtype=pl.String) @@ -36,14 +38,20 @@ def _resolve_rule_work_items( _matrix_name_expr(rule.matrix, rule.dimensions_used).alias("matrix_name") ) - errors = work.filter(pl.col("__missing_reason").is_not_null()).select( - pl.col("_row_id").cast(pl.Int64), - pl.col("rule_name"), - pl.col("trip_id"), - pl.col("lookup_origin").cast(pl.Int64, strict=False).alias("origin"), - pl.col("lookup_destination").cast(pl.Int64, strict=False).alias("destination"), - pl.lit(None, dtype=pl.String).alias("matrix_name"), - pl.col("__missing_reason").alias("reason"), + errors = ( + work.filter(pl.col("__missing_reason").is_not_null()).select( + pl.col("_row_id").cast(pl.Int64), + pl.col("rule_name"), + pl.col("trip_id"), + pl.col("lookup_origin").cast(pl.Int64, strict=False).alias("origin"), + pl.col("lookup_destination") + .cast(pl.Int64, strict=False) + .alias("destination"), + pl.lit(None, dtype=pl.String).alias("matrix_name"), + pl.col("__missing_reason").alias("reason"), + ) + if include_errors + else pl.DataFrame() ) valid = work.filter(pl.col("__missing_reason").is_null()).select( diff --git a/processor/skimjoin/annotate/trips.py b/processor/skimjoin/annotate/trips.py index acc5585..ac4e219 100644 --- a/processor/skimjoin/annotate/trips.py +++ b/processor/skimjoin/annotate/trips.py @@ -2,8 +2,11 @@ import polars as pl -from processor.skimjoin.annotate.engine import annotate_lookup_table -from processor.skimjoin.config.schema import NormalizedConfig +from processor.skimjoin.annotate.engine import ( + annotate_lookup_table, + lookup_output_values, +) +from processor.skimjoin.config.schema import NormalizedConfig, NormalizedLookupRule from processor.skimjoin.skimstore.base import SkimStore @@ -13,6 +16,9 @@ def annotate_trips( inventory: pl.DataFrame, skim_store: SkimStore | None = None, include_fallback_report: bool = False, + *, + rules: list[NormalizedLookupRule] | None = None, + collect_reports: bool = True, ) -> tuple[pl.DataFrame, pl.DataFrame, pl.DataFrame] | tuple[ pl.DataFrame, pl.DataFrame, pl.DataFrame, pl.DataFrame ]: @@ -21,16 +27,40 @@ def annotate_trips( return annotate_lookup_table( trips_with_ids, source_table=source_table, - rules=normalized.trip_lookups or normalized.lookups, + rules=( + rules + if rules is not None + else normalized.trip_lookups or normalized.lookups + ), normalized=normalized, inventory=inventory, mode_column=normalized.activitysim.trip_mode_column, skim_store=skim_store, include_fallback_report=include_fallback_report, + collect_reports=collect_reports, table_name="trips", ) +def lookup_trip_output_values( + trips: pl.DataFrame, + normalized: NormalizedConfig, + inventory: pl.DataFrame, + *, + rules: list[NormalizedLookupRule], + skim_store: SkimStore | None = None, +) -> pl.DataFrame: + trips_with_ids = trips.with_row_index("_row_id") + return lookup_output_values( + _trip_lookup_source(trips_with_ids, normalized), + rules=rules, + normalized=normalized, + inventory=inventory, + mode_column=normalized.activitysim.trip_mode_column, + skim_store=skim_store, + ) + + def _trip_lookup_source( trips: pl.DataFrame, normalized: NormalizedConfig, diff --git a/processor/skimjoin/config/validation.py b/processor/skimjoin/config/validation.py index 5444e19..83b13f7 100644 --- a/processor/skimjoin/config/validation.py +++ b/processor/skimjoin/config/validation.py @@ -1,5 +1,6 @@ from __future__ import annotations +from collections import Counter from dataclasses import dataclass from typing import Any @@ -11,6 +12,7 @@ NormalizedConfig, NormalizedLookupRule, ) +from processor.skimjoin.inventory import qualified_matrix_reference class ConfigValidationError(ValueError): @@ -120,18 +122,32 @@ def _inventory_by_name( ] ).to_dicts() inventory_by_name: dict[str, dict[str, object]] = {} - duplicate_names: list[str] = [] + name_counts = Counter(str(row["matrix_name"]) for row in matrix_rows) + qualified_counts = Counter( + qualified_matrix_reference(row["file_path"], str(row["matrix_name"])) + for row in matrix_rows + ) + ambiguous_sources: dict[str, list[str]] = {} for row in matrix_rows: matrix_name = str(row["matrix_name"]) - if matrix_name in inventory_by_name: - duplicate_names.append(matrix_name) - continue - inventory_by_name[matrix_name] = row + qualified_name = qualified_matrix_reference(row["file_path"], matrix_name) + inventory_by_name.setdefault(qualified_name, row) + if name_counts[matrix_name] == 1: + inventory_by_name[matrix_name] = row + else: + ambiguous_sources.setdefault(matrix_name, []).append(qualified_name) + for matrix_name, sources in ambiguous_sources.items(): + inventory_by_name[matrix_name] = { + "__ambiguous_sources": sorted(set(sources)) + } failures = [] - if duplicate_names: + duplicate_qualified_names = sorted( + name for name, count in qualified_counts.items() if count > 1 + ) + if duplicate_qualified_names: failures.append( - "Duplicate matrix names in skim inventory: " - + ", ".join(sorted(set(duplicate_names))) + "Duplicate file-qualified matrix references in skim inventory: " + + ", ".join(duplicate_qualified_names) ) return inventory_by_name, failures @@ -318,17 +334,25 @@ def _validate_target_table( failures.extend(f"{rule.name}: {message}" for message in combo_failures) for combo in combos: matrix_name = str(combo["matrix_name"]) - if matrix_name not in inventory_by_name: + inventory_record = inventory_by_name.get(matrix_name) + if inventory_record is None: failures.append( f"{rule.name}: referenced matrix {matrix_name!r} was not found in skim inventory" ) continue + ambiguous_sources = inventory_record.get("__ambiguous_sources") + if ambiguous_sources: + failures.append( + f"{rule.name}: ambiguous matrix reference {matrix_name!r}; qualify it with one of: " + + ", ".join(str(source) for source in ambiguous_sources) + ) + continue if ( - str(inventory_by_name[matrix_name]["source_kind"]) == "od_matrix" + str(inventory_record["source_kind"]) == "od_matrix" and rule.lookup == "od" ): - shape_rows = int(inventory_by_name[matrix_name]["shape_rows"]) - shape_cols = int(inventory_by_name[matrix_name]["shape_cols"]) + shape_rows = int(inventory_record["shape_rows"]) + shape_cols = int(inventory_record["shape_cols"]) zone_failures, zone_warnings = _validate_od_bounds( rule, combo["rows"], @@ -641,6 +665,9 @@ def _referenced_matrices( combos, _ = _rule_matrix_combinations(rule, subset) for combo in combos: matrix_name = str(combo["matrix_name"]) - if matrix_name in inventory_by_name: + inventory_record = inventory_by_name.get(matrix_name) + if inventory_record is not None and not inventory_record.get( + "__ambiguous_sources" + ): referenced.add(matrix_name) return referenced diff --git a/processor/skimjoin/csv_demand.py b/processor/skimjoin/csv_demand.py new file mode 100644 index 0000000..34e4849 --- /dev/null +++ b/processor/skimjoin/csv_demand.py @@ -0,0 +1,84 @@ +from __future__ import annotations + +import polars as pl + +from processor.skimjoin.annotate.tours import _directional_tour_context +from processor.skimjoin.config.schema import NormalizedConfig, NormalizedLookupRule +from processor.skimjoin.skimstore.base import SkimStore + + +def plan_csv_od_demands( + *, + trips: pl.DataFrame, + tours: pl.DataFrame, + normalized: NormalizedConfig, + inventory: pl.DataFrame, + skim_store: SkimStore, +) -> None: + if skim_store.od_csv_demand_planned(): + return + + rules = [*normalized.trip_lookups, *normalized.tour_lookups] + skim_store.plan_csv_tables( + inventory, + matrix_templates={rule.matrix for rule in rules}, + ) + if not skim_store.has_planned_od_csv_tables(): + return + + trip_od = _demand_frames(trips, normalized.trip_lookups) + tour_od: list[pl.DataFrame] = [] + if normalized.tour_lookups: + tours_with_ids = tours.with_row_index("_row_id") + for outbound in (True, False): + context = _directional_tour_context( + tours_with_ids, + normalized, + outbound=outbound, + ) + tour_od.extend(_demand_frames(context, normalized.tour_lookups)) + + skim_store.set_complete_od_csv_demand( + demand=_combine_demands( + [*trip_od, *tour_od], + { + "__lookup_origin": pl.Float64, + "__lookup_destination": pl.Float64, + }, + ), + ) + + +def _demand_frames( + source: pl.DataFrame, + rules: list[NormalizedLookupRule], +) -> list[pl.DataFrame]: + od_columns = { + (rule.origin, rule.destination) + for rule in rules + if rule.lookup == "od" + and rule.origin in source.columns + and rule.destination in source.columns + } + return [ + source.select( + pl.col(origin).cast(pl.Float64).alias("__lookup_origin"), + pl.col(destination).cast(pl.Float64).alias("__lookup_destination"), + ) + for origin, destination in od_columns + ] + + +def _combine_demands( + frames: list[pl.DataFrame], + schema: dict[str, pl.DataType], +) -> pl.DataFrame: + if not frames: + return pl.DataFrame(schema=schema) + columns = list(schema) + return ( + pl.concat(frames, how="vertical_relaxed") + .filter(pl.all_horizontal(pl.col(column).is_not_null() for column in columns)) + .unique() + .cast(schema) + ) diff --git a/processor/skimjoin/hypothetical_sidecars.py b/processor/skimjoin/hypothetical_sidecars.py index 02489d7..12df2f1 100644 --- a/processor/skimjoin/hypothetical_sidecars.py +++ b/processor/skimjoin/hypothetical_sidecars.py @@ -4,9 +4,10 @@ import polars as pl -from processor.skimjoin.annotate.tours import annotate_tours -from processor.skimjoin.annotate.trips import annotate_trips -from processor.skimjoin.config.schema import NormalizedConfig +from processor.skimjoin.annotate.tours import lookup_tour_output_values +from processor.skimjoin.annotate.trips import lookup_trip_output_values +from processor.skimjoin.config.schema import NormalizedConfig, NormalizedLookupRule +from processor.skimjoin.csv_demand import plan_csv_od_demands from processor.skimjoin.skimstore.base import SkimStore TRIP_HYPOTHETICAL_SIDECAR_SCHEMA = { @@ -38,6 +39,14 @@ def build_hypothetical_sidecars( skim_store: SkimStore | None = None, ) -> tuple[pl.DataFrame, pl.DataFrame]: """Build long-form hypothetical skim sidecars for trips and tours.""" + if skim_store is not None: + plan_csv_od_demands( + trips=trips, + tours=tours, + normalized=normalized, + inventory=inventory, + skim_store=skim_store, + ) trip_sidecar = _build_trip_hypothetical_sidecar( trips=trips, normalized=normalized, @@ -72,23 +81,25 @@ def _build_trip_hypothetical_sidecar( frames: list[pl.DataFrame] = [] for mode in _lookup_modes(normalized.trip_lookups): - outputs = _outputs_for_mode(normalized.trip_lookups, mode) + mode_rules = _rules_for_mode(normalized.trip_lookups, mode) + outputs = _outputs_for_mode(mode_rules, mode) if not outputs: continue hypothetical_input = trips.with_columns( pl.col(mode_column).cast(pl.Utf8).alias("__observed_mode"), pl.lit(mode).alias(mode_column), ) - annotated, _, _ = annotate_trips( + output_values = lookup_trip_output_values( hypothetical_input, normalized, inventory, skim_store=skim_store, - include_fallback_report=False, + rules=mode_rules, ) frames.append( - _melt_trip_outputs( - annotated, + _trip_sidecar_from_output_values( + hypothetical_input, + output_values, outputs=outputs, trip_id_column=trip_id_column, hypothetical_mode=mode, @@ -116,23 +127,25 @@ def _build_tour_hypothetical_sidecar( frames: list[pl.DataFrame] = [] for mode in _lookup_modes(normalized.tour_lookups): - outputs = _outputs_for_mode(normalized.tour_lookups, mode) + mode_rules = _rules_for_mode(normalized.tour_lookups, mode) + outputs = _outputs_for_mode(mode_rules, mode) if not outputs: continue hypothetical_input = tours.with_columns( pl.col(mode_column).cast(pl.Utf8).alias("__observed_mode"), pl.lit(mode).alias(mode_column), ) - annotated, _, _ = annotate_tours( + output_values = lookup_tour_output_values( hypothetical_input, normalized, inventory, skim_store=skim_store, - include_fallback_report=False, + rules=mode_rules, ) frames.append( - _melt_tour_outputs( - annotated, + _tour_sidecar_from_output_values( + hypothetical_input, + output_values, outputs=outputs, tour_id_column=tour_id_column, hypothetical_mode=mode, @@ -149,30 +162,44 @@ def _outputs_for_mode(rules, mode: str) -> list[str]: return sorted({str(rule.output) for rule in rules if str(rule.mode) == str(mode)}) -def _melt_trip_outputs( - annotated: pl.DataFrame, +def _rules_for_mode( + rules: list[NormalizedLookupRule], + mode: str, +) -> list[NormalizedLookupRule]: + return [rule for rule in rules if str(rule.mode) == str(mode)] + + +def _trip_sidecar_from_output_values( + trips: pl.DataFrame, + output_values: pl.DataFrame, *, outputs: list[str], trip_id_column: str, hypothetical_mode: str, ) -> pl.DataFrame: - available_outputs = [column for column in outputs if column in annotated.columns] + available_outputs = _available_outputs(output_values, outputs) if not available_outputs: return pl.DataFrame(schema=TRIP_HYPOTHETICAL_SIDECAR_SCHEMA) return ( - annotated.select( - pl.col(trip_id_column).cast(pl.Int64, strict=False).alias("trip_id"), - pl.col("__observed_mode").cast(pl.Utf8).alias("observed_mode"), - pl.col("finalweight").cast(pl.Float64), - *[pl.col(column).cast(pl.Float64, strict=False).alias(column) for column in available_outputs], + pl.DataFrame({"component": available_outputs}) + .join( + trips.with_row_index("_row_id").select( + "_row_id", + pl.col(trip_id_column).cast(pl.Int64, strict=False).alias("trip_id"), + pl.col("__observed_mode").cast(pl.Utf8).alias("observed_mode"), + pl.col("finalweight").cast(pl.Float64), + ), + how="cross", ) - .melt( - id_vars=["trip_id", "observed_mode", "finalweight"], - value_vars=available_outputs, - variable_name="component", - value_name="value", + .join( + output_values.rename({"output": "component"}), + on=["_row_id", "component"], + how="left", + ) + .with_columns( + pl.lit(hypothetical_mode).alias("hypothetical_mode"), + pl.col("value").cast(pl.Float64, strict=False).fill_nan(None), ) - .with_columns(pl.lit(hypothetical_mode).alias("hypothetical_mode")) .select( "trip_id", "observed_mode", @@ -185,31 +212,36 @@ def _melt_trip_outputs( ) -def _melt_tour_outputs( - annotated: pl.DataFrame, +def _tour_sidecar_from_output_values( + tours: pl.DataFrame, + output_values: pl.DataFrame, *, outputs: list[str], tour_id_column: str, hypothetical_mode: str, ) -> pl.DataFrame: - available_outputs = [column for column in outputs if column in annotated.columns] + available_outputs = _available_outputs(output_values, outputs) if not available_outputs: return pl.DataFrame(schema=TOUR_HYPOTHETICAL_SIDECAR_SCHEMA) return ( - annotated.select( - pl.col(tour_id_column).cast(pl.Int64, strict=False).alias("tour_id"), - pl.col("__observed_mode").cast(pl.Utf8).alias("observed_mode"), - pl.col("finalweight").cast(pl.Float64), - *[pl.col(column).cast(pl.Float64, strict=False).alias(column) for column in available_outputs], + pl.DataFrame({"component": available_outputs}) + .join( + tours.with_row_index("_row_id").select( + "_row_id", + pl.col(tour_id_column).cast(pl.Int64, strict=False).alias("tour_id"), + pl.col("__observed_mode").cast(pl.Utf8).alias("observed_mode"), + pl.col("finalweight").cast(pl.Float64), + ), + how="cross", ) - .melt( - id_vars=["tour_id", "observed_mode", "finalweight"], - value_vars=available_outputs, - variable_name="component", - value_name="value", + .join( + output_values.rename({"output": "component"}), + on=["_row_id", "component"], + how="left", ) .with_columns( pl.lit(hypothetical_mode).alias("hypothetical_mode"), + pl.col("value").cast(pl.Float64, strict=False).fill_nan(None), pl.when(pl.col("component").str.ends_with("_outbound")) .then(pl.lit("outbound")) .when(pl.col("component").str.ends_with("_inbound")) @@ -230,6 +262,16 @@ def _melt_tour_outputs( ) +def _available_outputs( + output_values: pl.DataFrame, + configured_outputs: list[str], +) -> list[str]: + if output_values.is_empty(): + return [] + present = set(output_values.get_column("output").to_list()) + return [output for output in configured_outputs if output in present] + + def _concat_frames( frames: list[pl.DataFrame], schema: dict[str, pl.DataType], diff --git a/processor/skimjoin/inventory.py b/processor/skimjoin/inventory.py index 1c5939d..7e9c2fd 100644 --- a/processor/skimjoin/inventory.py +++ b/processor/skimjoin/inventory.py @@ -9,6 +9,9 @@ import polars as pl +MATRIX_REFERENCE_SEPARATOR = "::" + + @dataclass(frozen=True) class MatrixRecord: file_path: str @@ -24,6 +27,10 @@ class MatrixRecord: destination_column_name: str | None = None +def qualified_matrix_reference(file_path: str | Path, matrix_name: str) -> str: + return f"{Path(file_path).name}{MATRIX_REFERENCE_SEPARATOR}{matrix_name}" + + def expand_paths(paths: Iterable[str | Path]) -> list[Path]: expanded: list[Path] = [] for raw_path in paths: @@ -73,17 +80,24 @@ def inventory_hdf5_file(path: str | Path) -> list[MatrixRecord]: def inventory_csv_file(path: str | Path) -> list[MatrixRecord]: path = Path(path) - table = pl.read_csv(path) - if table.width < 2: + scan = pl.scan_csv(path) + schema = scan.collect_schema() + columns = schema.names() + if len(columns) < 2: raise ValueError(f"CSV skim file must contain one key column and at least one value column: {path}") - - if _looks_like_od_csv(table.columns): - origin_column = table.columns[0] - destination_column = table.columns[1] + height = int( + scan.select(pl.len().cast(pl.Int64).alias("height")) + .collect(engine="streaming") + .item() + ) + + if _looks_like_od_csv(columns): + origin_column = columns[0] + destination_column = columns[1] value_columns = [ column - for column in table.columns[2:] - if table.schema[column].is_numeric() + for column in columns[2:] + if schema[column].is_numeric() ] if not value_columns: raise ValueError(f"CSV O-D skim file must contain at least one numeric value column: {path}") @@ -94,9 +108,9 @@ def inventory_csv_file(path: str | Path) -> list[MatrixRecord]: file_path=str(path), matrix_path=value_column, matrix_name=f"{path.stem}__{value_column}", - shape_rows=int(table.height), + shape_rows=height, shape_cols=1, - dtype=str(table.schema[value_column]), + dtype=str(schema[value_column]), source_kind="od_table", value_column_name=value_column, origin_column_name=origin_column, @@ -105,11 +119,11 @@ def inventory_csv_file(path: str | Path) -> list[MatrixRecord]: ) return rows - key_column = table.columns[0] + key_column = columns[0] value_columns = [ column - for column in table.columns[1:] - if table.schema[column].is_numeric() + for column in columns[1:] + if schema[column].is_numeric() ] if not value_columns: raise ValueError(f"CSV skim file must contain at least one numeric value column: {path}") @@ -121,9 +135,9 @@ def inventory_csv_file(path: str | Path) -> list[MatrixRecord]: file_path=str(path), matrix_path=value_column, matrix_name=f"{path.stem}__{value_column}", - shape_rows=int(table.height), + shape_rows=height, shape_cols=1, - dtype=str(table.schema[value_column]), + dtype=str(schema[value_column]), source_kind="keyed_column", key_column_name=key_column, value_column_name=value_column, diff --git a/processor/skimjoin/runtime_execution.py b/processor/skimjoin/runtime_execution.py index c5110b0..7a245d2 100644 --- a/processor/skimjoin/runtime_execution.py +++ b/processor/skimjoin/runtime_execution.py @@ -10,8 +10,12 @@ from processor.models import RunData from processor.skimjoin.annotate.tours import annotate_tours from processor.skimjoin.annotate.trips import annotate_trips +from processor.skimjoin.csv_demand import plan_csv_od_demands from processor.skimjoin.hypothetical_sidecars import build_hypothetical_sidecars -from processor.skimjoin.inventory import inventory_skim_files +from processor.skimjoin.inventory import ( + inventory_skim_files, + qualified_matrix_reference, +) from processor.skimjoin.runtime_types import _RuntimeSkimjoinResult from processor.skimjoin.skimstore.omx import OmxSkimStore @@ -35,7 +39,8 @@ def _validate_runtime_inventory(inventory: pl.DataFrame) -> None: ) matrix_names = [ - str(value) for value in inventory.get_column("matrix_name").to_list() + qualified_matrix_reference(row["file_path"], row["matrix_name"]) + for row in inventory.select(["file_path", "matrix_name"]).to_dicts() ] duplicates = sorted( matrix_name @@ -44,8 +49,8 @@ def _validate_runtime_inventory(inventory: pl.DataFrame) -> None: ) if duplicates: raise ValueError( - "Integrated skimjoin requires unique matrix names across skim inputs. " - + "Duplicate names: " + "Integrated skimjoin requires unique file-qualified matrix references. " + + "Duplicate references: " + ", ".join(repr(name) for name in duplicates) ) @@ -66,6 +71,13 @@ def _run_integrated_skimjoin( ) -> _RuntimeSkimjoinResult: inventory = _resolved_runtime_inventory(normalized) skim_store = OmxSkimStore() + plan_csv_od_demands( + trips=rd.trips, + tours=rd.tours, + normalized=normalized, + inventory=inventory, + skim_store=skim_store, + ) trip_outputs = annotate_trips_fn( rd.trips, normalized, diff --git a/processor/skimjoin/skimstore/base.py b/processor/skimjoin/skimstore/base.py index b0e4690..f9df965 100644 --- a/processor/skimjoin/skimstore/base.py +++ b/processor/skimjoin/skimstore/base.py @@ -1,6 +1,7 @@ from __future__ import annotations from pathlib import Path +import re import h5py import numpy as np @@ -12,8 +13,64 @@ def __init__(self) -> None: self._cache: dict[tuple[str, str], np.ndarray] = {} self._keyed_cache: dict[tuple[str, str, str], dict[int, float]] = {} self._keyed_table_cache: dict[tuple[str, str, str], pl.DataFrame] = {} + self._keyed_csv_cache: dict[tuple[str, str], pl.DataFrame] = {} + self._keyed_csv_columns: dict[tuple[str, str], set[str]] = {} self._od_table_cache: dict[tuple[str, str, str, str], dict[tuple[int, int], float]] = {} self._od_table_frame_cache: dict[tuple[str, str, str, str], pl.DataFrame] = {} + self._od_csv_cache: dict[tuple[str, str, str], pl.DataFrame] = {} + self._od_csv_columns: dict[tuple[str, str, str], set[str]] = {} + self._od_csv_demands: dict[tuple[str, str, str], pl.DataFrame] = {} + self._od_csv_demand_planned = False + + def plan_csv_tables( + self, + inventory: pl.DataFrame, + *, + matrix_templates: set[str], + ) -> None: + patterns = [_matrix_template_pattern(template) for template in matrix_templates] + if not patterns: + return + + rows = inventory.filter( + pl.col("source_kind").is_in(["keyed_column", "od_table"]) + ).to_dicts() + for row in rows: + file_path = str(row["file_path"]) + matrix_name = str(row["matrix_name"]) + qualified_name = f"{Path(file_path).name}::{matrix_name}" + if not any( + pattern.fullmatch(matrix_name) or pattern.fullmatch(qualified_name) + for pattern in patterns + ): + continue + + value_column = str(row["value_column_name"]) + if row["source_kind"] == "keyed_column": + key = (file_path, str(row["key_column_name"])) + self._keyed_csv_columns.setdefault(key, set()).add(value_column) + else: + key = ( + file_path, + str(row["origin_column_name"]), + str(row["destination_column_name"]), + ) + self._od_csv_columns.setdefault(key, set()).add(value_column) + + def has_planned_od_csv_tables(self) -> bool: + return bool(self._od_csv_columns) + + def set_complete_od_csv_demand( + self, + *, + demand: pl.DataFrame, + ) -> None: + for key in self._od_csv_columns: + self._od_csv_demands[key] = demand + self._od_csv_demand_planned = True + + def od_csv_demand_planned(self) -> bool: + return self._od_csv_demand_planned def get_matrix(self, file_path: str, matrix_path: str) -> np.ndarray: key = (file_path, matrix_path) @@ -109,17 +166,51 @@ def get_keyed_table( if key in self._keyed_table_cache: return self._keyed_table_cache[key] + csv_key = (file_path, key_column_name) + value_columns = self._keyed_csv_columns.setdefault(csv_key, set()) + value_columns.add(value_column_name) + combined = self._keyed_csv_cache.get(csv_key) + if combined is None or value_column_name not in combined.columns: + self._keyed_table_cache = { + cache_key: table + for cache_key, table in self._keyed_table_cache.items() + if cache_key[:2] != csv_key + } + self._keyed_cache = { + cache_key: values + for cache_key, values in self._keyed_cache.items() + if cache_key[:2] != csv_key + } + combined = ( + pl.scan_csv(file_path) + .select( + pl.col(key_column_name).cast(pl.Float64).alias("__lookup_key"), + *( + pl.col(column).cast(pl.Float64) + for column in sorted(value_columns) + ), + ) + .filter(pl.col("__lookup_key").is_not_null()) + .group_by("__lookup_key") + .agg( + *( + pl.col(column) + .drop_nulls() + .last() + .alias(column) + for column in sorted(value_columns) + ) + ) + .collect(engine="streaming") + ) + self._keyed_csv_cache[csv_key] = combined + table = ( - pl.read_csv(file_path) - .with_row_index("__row_id") - .select( - pl.col("__row_id"), - pl.col(key_column_name).cast(pl.Float64).alias("__lookup_key"), - pl.col(value_column_name).cast(pl.Float64).alias("__lookup_value"), + combined.select( + pl.col("__lookup_key"), + pl.col(value_column_name).alias("__lookup_value"), ) - .filter(pl.col("__lookup_key").is_not_null() & pl.col("__lookup_value").is_not_null()) - .group_by("__lookup_key", maintain_order=True) - .agg(pl.col("__lookup_value").sort_by("__row_id").last()) + .filter(pl.col("__lookup_value").is_not_null()) ) self._keyed_table_cache[key] = table return table @@ -219,25 +310,72 @@ def get_od_table_frame( if key in self._od_table_frame_cache: return self._od_table_frame_cache[key] - table = ( - pl.read_csv(file_path) - .with_row_index("__row_id") - .select( - pl.col("__row_id"), - pl.col(origin_column_name).cast(pl.Float64).alias("__lookup_origin"), - pl.col(destination_column_name).cast(pl.Float64).alias("__lookup_destination"), - pl.col(value_column_name).cast(pl.Float64).alias("__lookup_value"), + csv_key = (file_path, origin_column_name, destination_column_name) + value_columns = self._od_csv_columns.setdefault(csv_key, set()) + value_columns.add(value_column_name) + combined = self._od_csv_cache.get(csv_key) + if combined is None or value_column_name not in combined.columns: + self._od_table_frame_cache = { + cache_key: table + for cache_key, table in self._od_table_frame_cache.items() + if cache_key[:3] != csv_key + } + self._od_table_cache = { + cache_key: values + for cache_key, values in self._od_table_cache.items() + if cache_key[:3] != csv_key + } + scan = ( + pl.scan_csv(file_path) + .select( + pl.col(origin_column_name) + .cast(pl.Float64) + .alias("__lookup_origin"), + pl.col(destination_column_name) + .cast(pl.Float64) + .alias("__lookup_destination"), + *( + pl.col(column).cast(pl.Float64) + for column in sorted(value_columns) + ), + ) + .filter( + pl.col("__lookup_origin").is_not_null() + & pl.col("__lookup_destination").is_not_null() + ) ) - .filter( - pl.col("__lookup_origin").is_not_null() - & pl.col("__lookup_destination").is_not_null() - & pl.col("__lookup_value").is_not_null() + demand = self._od_csv_demands.get(csv_key) + if demand is not None: + scan = scan.join( + demand.lazy(), + on=["__lookup_origin", "__lookup_destination"], + how="semi", + ) + combined = ( + scan + .group_by( + ["__lookup_origin", "__lookup_destination"], + ) + .agg( + *( + pl.col(column) + .drop_nulls() + .last() + .alias(column) + for column in sorted(value_columns) + ) + ) + .collect(engine="streaming") ) - .group_by( - ["__lookup_origin", "__lookup_destination"], - maintain_order=True, + self._od_csv_cache[csv_key] = combined + + table = ( + combined.select( + "__lookup_origin", + "__lookup_destination", + pl.col(value_column_name).alias("__lookup_value"), ) - .agg(pl.col("__lookup_value").sort_by("__row_id").last()) + .filter(pl.col("__lookup_value").is_not_null()) ) self._od_table_frame_cache[key] = table return table @@ -352,3 +490,14 @@ def _map_with_zone_map(values: np.ndarray, zone_map: dict[int, int]) -> np.ndarr if valid.any(): result[valid] = mapped[positions[valid]] return result + + +def _matrix_template_pattern(template: str) -> re.Pattern[str]: + parts: list[str] = [] + start = 0 + for match in re.finditer(r"\{[^{}]+\}", template): + parts.append(re.escape(template[start : match.start()])) + parts.append(".+") + start = match.end() + parts.append(re.escape(template[start:])) + return re.compile("".join(parts)) diff --git a/processor/summarize/builder.py b/processor/summarize/builder.py index e4af017..ce6a6ce 100644 --- a/processor/summarize/builder.py +++ b/processor/summarize/builder.py @@ -36,11 +36,18 @@ def summary_builder_identity(summary_id: str) -> dict[str, object]: } -def summary_digest(summary_id: str, config: Config) -> str: +def summary_digest( + summary_id: str, + config: Config, + *, + analysis_unit_identity: dict[str, object] | None = None, +) -> str: payload = { "summary_config_digest": config.summary_config_digest, "summary": summary_builder_identity(summary_id), } + if analysis_unit_identity is not None: + payload["analysis_unit"] = analysis_unit_identity return hashlib.sha256( json.dumps(payload, separators=(",", ":"), ensure_ascii=True).encode("utf-8") ).hexdigest() @@ -49,9 +56,15 @@ def summary_digest(summary_id: str, config: Config) -> str: def summary_digests( config: Config, summary_ids: list[str] | None = None, + *, + analysis_unit_identity: dict[str, object] | None = None, ) -> dict[str, str]: return { - summary_id: summary_digest(summary_id, config) + summary_id: summary_digest( + summary_id, + config, + analysis_unit_identity=analysis_unit_identity, + ) for summary_id in ( summary_ids if summary_ids is not None else DEFAULT_SUMMARY_IDS ) diff --git a/processor/summarize/cache.py b/processor/summarize/cache.py index 51cafa3..ae5cc57 100644 --- a/processor/summarize/cache.py +++ b/processor/summarize/cache.py @@ -4,6 +4,7 @@ from pathlib import Path +from processor.analysis_units import AnalysisUnit from processor.cache_identity import build_run_fingerprint, build_run_keys, slugify from processor.summarize import builder as summary_builder from processor.summarize.cache_storage import ( @@ -25,6 +26,32 @@ SUMMARY_BY_ID, ) from runtime.config import Config +from runtime.config.signatures import segmentation_unit_signature_payload + + +def analysis_unit_key(*, segmentation_type: str, segment_id: str) -> str: + return f"{segmentation_type}::{segment_id}" + + +def _analysis_unit_summary_digests( + config: Config, + *, + segmentation_type: str, + segment_id: str, + summary_ids: list[str], +) -> dict[str, str]: + identity = None + if (segmentation_type, segment_id) != ("full", "full"): + identity = segmentation_unit_signature_payload( + config, + segmentation_type=segmentation_type, + segment_id=segment_id, + ) + return summary_builder.summary_digests( + config, + summary_ids, + analysis_unit_identity=identity, + ) def summary_file_map(summary_ids: list[str]) -> dict[str, str]: @@ -95,13 +122,25 @@ def write_summary_run_bundle( ) -> Path: """Write one run cache directory containing all segment variants.""" requested_ids = list(next(iter(summary_runs[0].summaries_by_mode.values())).keys()) + summary_digests_by_unit = { + analysis_unit_key( + segmentation_type=summary_run.segmentation_type, + segment_id=summary_run.segment_id, + ): _analysis_unit_summary_digests( + config, + segmentation_type=summary_run.segmentation_type, + segment_id=summary_run.segment_id, + summary_ids=requested_ids, + ) + for summary_run in summary_runs + } return _write_summary_run_bundle( summary_runs, config, output_root=output_root, run_fingerprint=run_fingerprint, prepared_manifest_identity=prepared_manifest_identity, - summary_digests=summary_builder.summary_digests(config, requested_ids), + summary_digests_by_unit=summary_digests_by_unit, summary_filename_by_id=SUMMARY_FILENAME_BY_ID, ) @@ -117,7 +156,37 @@ def inspect_summary_run_bundle( expected_prepared_manifest_identity: dict[str, object] | None = None, expected_label: str | None = None, expected_run_key: str | None = None, + expected_analysis_units: list[AnalysisUnit] | None = None, ) -> dict[str, object]: + requested_ids = list(expected_summary_ids or summary_builder.DEFAULT_SUMMARY_IDS) + if expected_analysis_units is not None: + unit_keys = [ + (unit.segmentation_type, unit.segment_id) + for unit in expected_analysis_units + ] + elif config.segmentation.enabled: + unit_keys = [ + ("full", "full"), + *[ + (definition.name, segment.id) + for definition in config.segmentation.definitions + for segment in definition.segments + ], + ] + else: + unit_keys = [("full", "full")] + expected_summary_digests_by_unit = { + analysis_unit_key( + segmentation_type=segmentation_type, + segment_id=segment_id, + ): _analysis_unit_summary_digests( + config, + segmentation_type=segmentation_type, + segment_id=segment_id, + summary_ids=requested_ids, + ) + for segmentation_type, segment_id in unit_keys + } return _inspect_summary_run_bundle( cache_dir, config, @@ -131,6 +200,7 @@ def inspect_summary_run_bundle( expected_summary_digests=summary_builder.summary_digests( config, expected_summary_ids ), + expected_summary_digests_by_unit=expected_summary_digests_by_unit, ) @@ -145,6 +215,7 @@ def load_summary_run_bundle( expected_prepared_manifest_identity: dict[str, object] | None = None, expected_label: str | None = None, expected_run_key: str | None = None, + expected_summary_ids_by_unit: dict[str, list[str]] | None = None, ) -> list[SummaryRun]: """Load one run cache directory and return all segment variants.""" return _load_summary_run_bundle( @@ -157,6 +228,7 @@ def load_summary_run_bundle( expected_prepared_manifest_identity=expected_prepared_manifest_identity, expected_label=expected_label, expected_run_key=expected_run_key, + expected_summary_ids_by_unit=expected_summary_ids_by_unit, summary_spec_by_id=SUMMARY_BY_ID, ) @@ -164,6 +236,7 @@ def load_summary_run_bundle( __all__ = [ "SCHEMA_VERSION", "SummaryRun", + "analysis_unit_key", "build_run_fingerprint", "build_run_keys", "discover_cache_dirs", diff --git a/processor/summarize/cache_storage.py b/processor/summarize/cache_storage.py index 38fe0f3..c27615e 100644 --- a/processor/summarize/cache_storage.py +++ b/processor/summarize/cache_storage.py @@ -4,6 +4,7 @@ from datetime import datetime, timezone from pathlib import Path +import shutil import polars as pl @@ -147,6 +148,10 @@ def _summary_manifest( "summary_digests": summary_digests, "run_fingerprint": run_fingerprint or {}, "prepared_manifest_identity": prepared_manifest_identity, + "identity": { + "upstream_prepared": prepared_manifest_identity, + "summary_config": config.summary_config_digest, + }, } @@ -252,7 +257,7 @@ def write_summary_run_bundle( output_root: str | Path | None = None, run_fingerprint: dict[str, object] | None = None, prepared_manifest_identity: dict[str, object] | None = None, - summary_digests: dict[str, str] | None = None, + summary_digests_by_unit: dict[str, dict[str, str]] | None = None, summary_filename_by_id: dict[str, str], ) -> Path: """Write one run cache directory containing full and segmented summary outputs.""" @@ -279,6 +284,8 @@ def write_summary_run_bundle( segmentation_type_entries: dict[str, dict[str, object]] = {} for summary_run in summary_runs: + unit_key = f"{summary_run.segmentation_type}::{summary_run.segment_id}" + unit_summary_digests = (summary_digests_by_unit or {}).get(unit_key, {}) segment_states: dict[str, dict[str, str]] = {} segment_diagnostics: dict[str, dict[str, str]] = {} segment_digests: dict[str, dict[str, str]] = {} @@ -297,13 +304,13 @@ def write_summary_run_bundle( failed_summaries[mode] = list(mode_payload["failed_summaries"]) summary_diagnostics[mode] = dict(mode_payload["summary_diagnostics"]) manifest_summary_digests[mode] = { - summary_id: (summary_digests or {}).get(summary_id, "") + summary_id: unit_summary_digests.get(summary_id, "") for summary_id in summary_ids } segment_states[mode] = dict(mode_payload["summary_states"]) segment_diagnostics[mode] = dict(mode_payload["summary_diagnostics"]) segment_digests[mode] = { - summary_id: (summary_digests or {}).get(summary_id, "") + summary_id: unit_summary_digests.get(summary_id, "") for summary_id in summary_ids } mode_dir = ( @@ -344,6 +351,27 @@ def write_summary_run_bundle( ) ) + current_segment_keys = { + (run.segmentation_type, run.segment_id) + for run in summary_runs + if not run.is_full_segment + } + for mode in weighting_modes: + segments_root = run_dir / "summary_tables" / mode / "segments" + if not segments_root.exists(): + continue + for segmentation_dir in segments_root.iterdir(): + if not segmentation_dir.is_dir(): + continue + for segment_dir in segmentation_dir.iterdir(): + if segment_dir.is_dir() and ( + segmentation_dir.name, + segment_dir.name, + ) not in current_segment_keys: + shutil.rmtree(segment_dir) + if not any(segmentation_dir.iterdir()): + segmentation_dir.rmdir() + manifest = _summary_manifest( summary_run=full_run, config=config, @@ -625,6 +653,54 @@ def _segment_mode_dirs( return segment_dirs +def _manifest_unit_metadata( + cache_dir: Path, + manifest: dict[str, object], + expected_modes: list[str], +) -> dict[str, dict[str, object]]: + _, _, _, _, full_digests = _manifest_summary_metadata(manifest) + units: dict[str, dict[str, object]] = { + "full::full": { + "mode_dirs": { + mode: ( + cache_dir / "summary_tables" / mode + if (cache_dir / "summary_tables" / mode).exists() + else cache_dir / mode + ) + for mode in expected_modes + }, + "summary_digests": full_digests, + } + } + for raw_group in list(manifest.get("segmentation_types", [])): + group = dict(raw_group) + segmentation_type = str(group.get("segmentation_type", "full")) + for raw_segment in list(group.get("segments", [])): + segment = dict(raw_segment) + segment_id = str(segment.get("segment_id", "full")) + summary_roots = { + str(mode): str(path) + for mode, path in dict(segment.get("summary_roots", {})).items() + } + units[f"{segmentation_type}::{segment_id}"] = { + "mode_dirs": { + mode: cache_dir + / Path(summary_roots.get(mode, f"summary_tables/{mode}")) + for mode in expected_modes + }, + "summary_digests": { + str(mode): { + str(summary_id): str(digest) + for summary_id, digest in dict(mode_digests).items() + } + for mode, mode_digests in dict( + segment.get("summary_digests", {}) + ).items() + }, + } + return units + + def inspect_summary_run_bundle( cache_dir: str | Path, config: Config, @@ -637,6 +713,7 @@ def inspect_summary_run_bundle( expected_label: str | None = None, expected_run_key: str | None = None, expected_summary_digests: dict[str, str] | None = None, + expected_summary_digests_by_unit: dict[str, dict[str, str]] | None = None, ) -> dict[str, object]: cache_dir = Path(cache_dir) manifest = read_manifest(cache_dir, error_cls=SummaryCacheError) @@ -671,6 +748,54 @@ def inspect_summary_run_bundle( manifest_summary_digests, ) = _manifest_summary_metadata(manifest) expected_summary_digests = dict(expected_summary_digests or {}) + if expected_summary_digests_by_unit is not None: + manifest_units = _manifest_unit_metadata(cache_dir, manifest, expected_modes) + reusable_by_unit: dict[str, list[str]] = {} + stale_by_unit: dict[str, list[str]] = {} + for unit_key, unit_expected_digests in expected_summary_digests_by_unit.items(): + reusable_by_unit[unit_key] = [] + stale_by_unit[unit_key] = [] + unit_metadata = manifest_units.get(unit_key) + for summary_id in resolved_summary_ids: + if unit_metadata is None: + stale_by_unit[unit_key].append(summary_id) + continue + mode_dirs = dict(unit_metadata["mode_dirs"]) + unit_manifest_digests = dict(unit_metadata["summary_digests"]) + filename = summary_files.get(summary_id, f"{summary_id}.csv") + is_reusable = all( + dict(unit_manifest_digests.get(mode, {})).get(summary_id) + == unit_expected_digests.get(summary_id) + and (mode_dirs[mode] / filename).exists() + for mode in expected_modes + ) + target = reusable_by_unit if is_reusable else stale_by_unit + target[unit_key].append(summary_id) + + stale_summary_ids = [ + summary_id + for summary_id in resolved_summary_ids + if any( + summary_id in unit_stale_ids + for unit_stale_ids in stale_by_unit.values() + ) + ] + reusable_summary_ids = [ + summary_id + for summary_id in resolved_summary_ids + if summary_id not in stale_summary_ids + ] + return { + "manifest": manifest, + "reusable_summary_ids": reusable_summary_ids, + "stale_summary_ids": stale_summary_ids, + "reusable_summary_ids_by_unit": reusable_by_unit, + "stale_summary_ids_by_unit": stale_by_unit, + "obsolete_unit_keys": sorted( + set(manifest_units) - set(expected_summary_digests_by_unit) + ), + } + stale_summary_ids: list[str] = [] reusable_summary_ids: list[str] = [] segment_dirs = _segment_mode_dirs(cache_dir, manifest, expected_modes) @@ -787,18 +912,26 @@ def load_summary_run_bundle( expected_prepared_manifest_identity: dict[str, object] | None = None, expected_label: str | None = None, expected_run_key: str | None = None, + expected_summary_ids_by_unit: dict[str, list[str]] | None = None, summary_spec_by_id: dict[str, object], ) -> list[SummaryRun]: """Load one run cache directory and return all persisted segment variants.""" cache_dir = Path(cache_dir) manifest = read_manifest(cache_dir, error_cls=SummaryCacheError) if "segmentation_types" not in manifest and "segments" not in manifest: + full_summary_ids = ( + expected_summary_ids_by_unit.get("full::full", []) + if expected_summary_ids_by_unit is not None + else expected_summary_ids + ) + if expected_summary_ids_by_unit is not None and not full_summary_ids: + return [] return [ load_summary_run_cache( cache_dir, config, expected_modes=expected_modes, - expected_summary_ids=expected_summary_ids, + expected_summary_ids=full_summary_ids, expected_summary_config_digest=expected_summary_config_digest, expected_run_fingerprint=expected_run_fingerprint, expected_prepared_manifest_identity=expected_prepared_manifest_identity, @@ -823,12 +956,21 @@ def load_summary_run_bundle( expected_label=expected_label, expected_run_key=expected_run_key, ) + requested_summary_ids = expected_summary_ids + if expected_summary_ids_by_unit is not None: + requested_summary_ids = list( + dict.fromkeys( + summary_id + for unit_summary_ids in expected_summary_ids_by_unit.values() + for summary_id in unit_summary_ids + ) + ) expected_modes, expected_summary_ids = _validated_mode_and_summary_ids( manifest=manifest, config=config, cache_dir=cache_dir, expected_modes=expected_modes, - expected_summary_ids=expected_summary_ids, + expected_summary_ids=requested_summary_ids, ) ( summary_files, @@ -839,16 +981,23 @@ def load_summary_run_bundle( ) = _manifest_summary_metadata(manifest) loaded_runs: list[SummaryRun] = [] + full_expected_summary_ids = ( + expected_summary_ids_by_unit.get("full::full", []) + if expected_summary_ids_by_unit is not None + else expected_summary_ids + ) full_summaries_by_mode: dict[str, dict[str, pl.DataFrame]] = {} full_summary_metadata_by_mode: dict[str, dict[str, dict[str, object]]] = {} for mode in expected_modes: + if not full_expected_summary_ids: + break full_mode_dir = cache_dir / "summary_tables" / mode if not full_mode_dir.exists(): full_mode_dir = cache_dir / mode mode_tables, mode_metadata = _load_mode_tables( mode_dir=full_mode_dir, mode=mode, - expected_summary_ids=expected_summary_ids, + expected_summary_ids=full_expected_summary_ids, summary_files=summary_files, empty_summaries=empty_summaries, manifest_summary_states=manifest_summary_states, @@ -857,8 +1006,9 @@ def load_summary_run_bundle( ) full_summaries_by_mode[mode] = mode_tables full_summary_metadata_by_mode[mode] = mode_metadata - loaded_runs.append( - SummaryRun( + if full_expected_summary_ids: + loaded_runs.append( + SummaryRun( label=str(manifest.get("label", cache_dir.name)), run_key=str(manifest.get("run_key", cache_dir.name)), summaries_by_mode=full_summaries_by_mode, @@ -869,8 +1019,8 @@ def load_summary_run_bundle( is_full_segment=True, source_run_dir=manifest.get("source_run_dir"), manifest=manifest, + ) ) - ) if "segmentation_types" in manifest: segment_groups = [] for raw_group in list(manifest.get("segmentation_types", [])): @@ -884,6 +1034,14 @@ def load_summary_run_bundle( for raw_segment in list(manifest.get("segments", [])) ] for segmentation_type, segment in segment_groups: + unit_key = f"{segmentation_type}::{segment.get('segment_id', 'full')}" + unit_expected_summary_ids = ( + expected_summary_ids_by_unit.get(unit_key, []) + if expected_summary_ids_by_unit is not None + else expected_summary_ids + ) + if not unit_expected_summary_ids: + continue summary_roots = { str(mode): str(path) for mode, path in dict(segment.get("summary_roots", {})).items() @@ -919,7 +1077,7 @@ def load_summary_run_bundle( mode_tables, mode_metadata = _load_mode_tables( mode_dir=cache_dir / mode_root, mode=mode, - expected_summary_ids=expected_summary_ids, + expected_summary_ids=unit_expected_summary_ids, summary_files=summary_files, empty_summaries=empty_summaries, manifest_summary_states=manifest_summary_states, diff --git a/processor/summarize/summaries/daily_travel_activity.py b/processor/summarize/summaries/daily_travel_activity.py index 722be56..1b50ab5 100644 --- a/processor/summarize/summaries/daily_travel_activity.py +++ b/processor/summarize/summaries/daily_travel_activity.py @@ -9,10 +9,178 @@ from processor.summarize.summaries.summary_helpers import ( _all_person_types_rollup, _summary_purpose_column, + joint_participant_weight_expr, ) from runtime.config import Config +def _person_exposure_records(rd: RunData) -> pl.DataFrame: + persons = rd.per.filter( + pl.col("person_id").is_not_null() + & pl.col("person_type").is_not_null() + & pl.col("finalweight").is_not_null() + ) + if "surveyable" in persons.columns: + surveyable = ( + pl.col("surveyable") + .cast(pl.Utf8) + .str.strip_chars() + .str.to_lowercase() + .is_in(["1", "true", "yes", "y"]) + ) + persons = persons.filter(pl.col("surveyable").is_null() | surveyable) + persons = persons.select( + "person_id", + pl.col("person_type").cast(pl.Utf8), + pl.col("finalweight").cast(pl.Float64).alias("person_weight"), + ) + + if not rd.day.is_empty() and "person_id" in rd.day.columns: + day_weight = ( + pl.col("finalweight").cast(pl.Float64).alias("_day_weight") + if "finalweight" in rd.day.columns + else pl.lit(None, dtype=pl.Float64).alias("_day_weight") + ) + return ( + rd.day.filter(pl.col("person_id").is_not_null()) + .select("person_id", day_weight) + .join(persons, on="person_id", how="inner") + .with_columns( + pl.coalesce("_day_weight", "person_weight").alias("person_weight") + ) + .select("person_id", "person_type", "person_weight") + ) + + return persons + + +def _day_person_records(rd: RunData, *day_columns: str) -> pl.DataFrame: + """Return every attributable day row with its day or person weight.""" + person_columns = rd.per.select( + "person_id", + pl.col("person_type").cast(pl.Utf8), + pl.col("finalweight").cast(pl.Float64).alias("_person_weight"), + ) + day_weight = ( + pl.col("finalweight").cast(pl.Float64).alias("_day_weight") + if "finalweight" in rd.day.columns + else pl.lit(None, dtype=pl.Float64).alias("_day_weight") + ) + return ( + rd.day.filter(pl.col("person_id").is_not_null()) + .select("person_id", *day_columns, day_weight) + .join(person_columns, on="person_id", how="inner") + .with_columns(pl.coalesce("_day_weight", "_person_weight").alias("finalweight")) + .select("person_id", "person_type", *day_columns, "finalweight") + ) + + +def _weighted_activity_by_person_type( + rd: RunData, + activity: pl.DataFrame, + *, + purpose_col: str, + participant_col: str, +) -> pl.DataFrame: + base = activity.filter( + pl.col(purpose_col).is_not_null() & pl.col("finalweight").is_not_null() + ) + + def _scalar_rows(rows: pl.DataFrame) -> pl.DataFrame: + return rows.with_columns( + pl.col("person_id").alias("_rate_person_id"), + joint_participant_weight_expr( + rows, + participant_col=participant_col, + output_col="activity_weight", + ), + ).select( + "_rate_person_id", + pl.col(purpose_col).cast(pl.Utf8).alias("activity_purpose"), + "activity_weight", + ) + + weighted_rows = _scalar_rows(base) + participant_required = {"tour_id", "person_id"} + can_attribute_participants = ( + "tour_id" in base.columns + and "tour_category" in base.columns + and participant_required.issubset(rd.joint_participants.columns) + and not rd.joint_participants.is_empty() + ) + if can_attribute_participants: + participants = ( + rd.joint_participants.filter( + pl.col("tour_id").is_not_null() & pl.col("person_id").is_not_null() + ) + .select( + "tour_id", + pl.col("person_id").alias("_rate_person_id"), + ) + .join( + rd.per.filter(pl.col("person_id").is_not_null()).select( + pl.col("person_id").alias("_rate_person_id"), + pl.col("finalweight").cast(pl.Float64).alias("_participant_weight"), + ), + on="_rate_person_id", + how="left", + ) + ) + participant_tour_ids = participants.select("tour_id") + category = ( + pl.col("tour_category").cast(pl.Utf8).str.strip_chars().str.to_lowercase() + ) + joint_record = ( + pl.col("tour_category").is_not_null() & (category == "joint") + ).fill_null(False) + participant_rows = ( + base.filter(joint_record) + .join(participants, on="tour_id", how="inner") + .select( + "_rate_person_id", + pl.col(purpose_col).cast(pl.Utf8).alias("activity_purpose"), + pl.coalesce( + pl.col("_participant_weight"), + pl.col("finalweight").cast(pl.Float64), + ).alias("activity_weight"), + ) + ) + scalar_rows = pl.concat( + [ + base.filter(~joint_record), + base.filter(joint_record).join( + participant_tour_ids, + on="tour_id", + how="anti", + ), + ], + how="vertical", + ) + weighted_rows = pl.concat( + [_scalar_rows(scalar_rows), participant_rows], + how="vertical", + ) + + eligible_people = _person_exposure_records(rd).select( + pl.col("person_id").alias("_rate_person_id") + ).unique() + return ( + weighted_rows.join(eligible_people, on="_rate_person_id", how="semi") + .join( + rd.per.filter( + pl.col("person_id").is_not_null() & pl.col("person_type").is_not_null() + ).select( + pl.col("person_id").alias("_rate_person_id"), + pl.col("person_type").cast(pl.Utf8), + ), + on="_rate_person_id", + how="inner", + ) + .group_by(["person_type", "activity_purpose"]) + .agg(activity_count=pl.col("activity_weight").sum()) + ) + + @summary( id="daily_activity_pattern_by_person_type", schema={ @@ -20,15 +188,50 @@ "daily_activity_pattern": pl.Utf8, "person_count": pl.Float64, }, - required_columns={"per": ("person_type", "cdap_activity", "finalweight")}, + required_columns={"per": ("person_type", "finalweight")}, ) def dap_summary(rd: RunData, config: Config) -> pl.DataFrame: """DAP by person type. Columns: person_type, daily_activity_pattern, person_count""" - if "person_type" not in rd.per.columns or "cdap_activity" not in rd.per.columns: + person_activity_required = {"person_type", "cdap_activity", "finalweight"} + day_required = {"person_id", "cdap_activity"} + person_required = {"person_id", "person_type", "finalweight"} + + sources: list[pl.DataFrame] = [] + person_choice_ids: pl.DataFrame | None = None + if person_activity_required.issubset(rd.per.columns): + person_choices = rd.per.filter(pl.col("cdap_activity").is_not_null()) + if not person_choices.is_empty(): + sources.append( + person_choices.select("person_type", "cdap_activity", "finalweight") + ) + if "person_id" in rd.per.columns: + person_choice_ids = person_choices.filter( + pl.col("person_id").is_not_null() + ).select("person_id") + + can_use_days = ( + not rd.day.is_empty() + and day_required.issubset(rd.day.columns) + and person_required.issubset(rd.per.columns) + ) + if can_use_days: + day_choices = _day_person_records(rd, "cdap_activity").filter( + pl.col("cdap_activity").is_not_null() + ) + if person_choice_ids is not None: + day_choices = day_choices.join( + person_choice_ids, on="person_id", how="anti" + ) + sources.append( + day_choices.select("person_type", "cdap_activity", "finalweight") + ) + + if not sources: return dap_summary.empty() + source = pl.concat(sources, how="vertical_relaxed") df = ( - rd.per.filter(pl.col("cdap_activity").is_not_null()) + source.filter(pl.col("cdap_activity").is_not_null()) .group_by(["person_type", "cdap_activity"]) .agg(person_count=pl.col("finalweight").sum()) .rename({"cdap_activity": "daily_activity_pattern"}) @@ -57,19 +260,103 @@ def dap_summary(rd: RunData, config: Config) -> pl.DataFrame: "mandatory_tour_frequency": pl.Int32, "person_count": pl.Float64, }, - required_columns={"per": ("person_type", "imf_choice", "finalweight")}, + required_columns={"per": ("person_id", "person_type", "finalweight")}, ) def mandatory_tour_freq(rd: RunData, config: Config) -> pl.DataFrame: """Returns DataFrame: person_type, mandatory_tour_frequency, person_count.""" - if "person_type" not in rd.per.columns or "imf_choice" not in rd.per.columns: + person_choice_required = {"person_type", "imf_choice", "finalweight"} + person_required = {"person_id", "person_type", "finalweight"} + day_required = {"person_id", "day_id"} + tour_required = {"person_id", "day_id"} + purpose_col = _summary_purpose_column(rd.tours) + + sources: list[pl.DataFrame] = [] + person_choice_ids: pl.DataFrame | None = None + if person_choice_required.issubset(rd.per.columns): + person_choices = rd.per.filter(pl.col("imf_choice").is_not_null()) + if not person_choices.is_empty(): + sources.append( + person_choices.select( + "person_type", + pl.col("imf_choice").cast(pl.Int32, strict=False), + "finalweight", + ) + ) + if "person_id" in rd.per.columns: + person_choice_ids = person_choices.filter( + pl.col("person_id").is_not_null() + ).select("person_id") + + can_derive_days = ( + person_required.issubset(rd.per.columns) + and day_required.issubset(rd.day.columns) + and tour_required.issubset(rd.tours.columns) + and purpose_col is not None + ) + if can_derive_days: + mandatory_tours = rd.tours.filter( + pl.col("person_id").is_not_null() + & pl.col("day_id").is_not_null() + & pl.col(purpose_col).is_not_null() + ) + if "tour_category" in mandatory_tours.columns: + mandatory_tours = mandatory_tours.filter( + pl.col("tour_category") == "mandatory" + ) + + tour_counts = mandatory_tours.group_by(["person_id", "day_id"]).agg( + (pl.col(purpose_col).cast(pl.Utf8).str.to_lowercase() == "work") + .sum() + .alias("_work_tours"), + (pl.col(purpose_col).cast(pl.Utf8).str.to_lowercase() == "school") + .sum() + .alias("_school_tours"), + ) + + day_choices = ( + _day_person_records(rd, "day_id") + .filter(pl.col("day_id").is_not_null()) + .join(tour_counts, on=["person_id", "day_id"], how="left") + .with_columns( + pl.col("_work_tours").fill_null(0), + pl.col("_school_tours").fill_null(0), + ) + .with_columns( + pl.when((pl.col("_work_tours") == 0) & (pl.col("_school_tours") == 0)) + .then(pl.lit(0)) + .when((pl.col("_work_tours") == 1) & (pl.col("_school_tours") == 0)) + .then(pl.lit(1)) + .when((pl.col("_work_tours") == 2) & (pl.col("_school_tours") == 0)) + .then(pl.lit(2)) + .when((pl.col("_work_tours") == 0) & (pl.col("_school_tours") == 1)) + .then(pl.lit(3)) + .when((pl.col("_work_tours") == 0) & (pl.col("_school_tours") == 2)) + .then(pl.lit(4)) + .when((pl.col("_work_tours") == 1) & (pl.col("_school_tours") == 1)) + .then(pl.lit(5)) + .otherwise(None) + .alias("imf_choice") + ) + ) + if person_choice_ids is not None: + day_choices = day_choices.join( + person_choice_ids, on="person_id", how="anti" + ) + sources.append(day_choices.select("person_type", "imf_choice", "finalweight")) + + if not sources: return mandatory_tour_freq.empty() + source = pl.concat(sources, how="vertical_relaxed") df = ( - rd.per.filter(pl.col("imf_choice") > 0) + source.filter(pl.col("imf_choice") > 0) .group_by(["person_type", "imf_choice"]) .agg(person_count=pl.col("finalweight").sum()) .rename({"imf_choice": "mandatory_tour_frequency"}) - .with_columns(pl.col("person_type").cast(pl.Utf8).alias("person_type")) + .with_columns( + pl.col("person_type").cast(pl.Utf8).alias("person_type"), + pl.col("mandatory_tour_frequency").cast(pl.Int32), + ) .select("person_type", "mandatory_tour_frequency", "person_count") ) @@ -98,30 +385,157 @@ def mandatory_tour_freq(rd: RunData, config: Config) -> pl.DataFrame: }, ) def indiv_nm_summary(rd: RunData, config: Config) -> pl.DataFrame: - """Returns DataFrame: person_type, nonmandatory_tour_frequency, person_count.""" + """Return nonmandatory frequency by person-day when day keys are safe. + + Multi-day data falls back to the legacy person-level calculation when any + joint-participant row cannot be assigned to a day without guessing. + """ per = rd.per if "person_type" not in per.columns: return indiv_nm_summary.empty() if "tour_category" in rd.tours.columns: - inm_counts = ( - rd.tours.filter(pl.col("tour_category") == "non_mandatory") + individual_tours = rd.tours.filter(pl.col("tour_category") == "non_mandatory") + else: + individual_tours = rd.tours.head(0) + + day_required = {"person_id", "day_id"} + has_multiday_people = False + if day_required.issubset(rd.day.columns): + has_multiday_people = not ( + rd.day.filter( + pl.col("person_id").is_not_null() & pl.col("day_id").is_not_null() + ) .group_by("person_id") - .agg(pl.len().alias("inmTours")) + .agg(pl.col("day_id").n_unique().alias("_day_count")) + .filter(pl.col("_day_count") > 1) + .is_empty() + ) + + person_days: pl.DataFrame | None = None + participant_days: pl.DataFrame | None = None + can_use_person_days = ( + has_multiday_people + and day_required.issubset(individual_tours.columns) + and {"person_id", "person_type", "finalweight"}.issubset(per.columns) + ) + if can_use_person_days: + person_days = _day_person_records(rd, "day_id").filter( + pl.col("day_id").is_not_null() + ) + valid_day_keys = person_days.select("person_id", "day_id").unique() + keyed_individual_tours = individual_tours.filter( + pl.col("person_id").is_not_null() + ) + can_use_person_days = ( + keyed_individual_tours.filter(pl.col("day_id").is_null()).is_empty() + and keyed_individual_tours.join( + valid_day_keys, on=["person_id", "day_id"], how="anti" + ).is_empty() + ) + + participant_rows = rd.joint_participants + if can_use_person_days and "person_id" in participant_rows.columns: + participant_rows = participant_rows.filter(pl.col("person_id").is_not_null()) + if participant_rows.is_empty(): + participant_days = participant_rows.select("person_id").with_columns( + pl.lit(None, dtype=pl.Int64).alias("day_id") + ) + elif "day_id" in participant_rows.columns: + participant_days = participant_rows.select("person_id", "day_id") + can_use_person_days = ( + participant_days.filter(pl.col("day_id").is_null()).is_empty() + and participant_days.join( + valid_day_keys, on=["person_id", "day_id"], how="anti" + ).is_empty() + ) + elif "tour_id" in participant_rows.columns and {"tour_id", "day_id"}.issubset( + rd.tours.columns + ): + missing_tour_ids = participant_rows.filter(pl.col("tour_id").is_null()) + keyed_participant_rows = participant_rows.filter( + pl.col("tour_id").is_not_null() + ) + tour_day_rows = ( + rd.tours.filter( + pl.col("tour_id").is_not_null() & pl.col("day_id").is_not_null() + ) + .select("tour_id", "day_id") + .join( + keyed_participant_rows.select("tour_id").unique(), + on="tour_id", + how="semi", + ) + ) + ambiguous_tour_ids = ( + tour_day_rows.group_by("tour_id") + .agg(pl.col("day_id").n_unique().alias("_day_count")) + .filter(pl.col("_day_count") != 1) + ) + tour_day_map = tour_day_rows.unique() + participant_days = keyed_participant_rows.join( + tour_day_map, on="tour_id", how="left" + ).select("person_id", "day_id") + can_use_person_days = ( + missing_tour_ids.is_empty() + and ambiguous_tour_ids.is_empty() + and keyed_participant_rows.height == participant_days.height + and participant_days.filter(pl.col("day_id").is_null()).is_empty() + and participant_days.join( + valid_day_keys, on=["person_id", "day_id"], how="anti" + ).is_empty() + ) + else: + can_use_person_days = participant_rows.is_empty() + elif can_use_person_days: + can_use_person_days = participant_rows.is_empty() + participant_days = pl.DataFrame( + schema={"person_id": pl.Int64, "day_id": pl.Int64} ) - else: - inm_counts = per.select("person_id").with_columns(pl.lit(0).alias("inmTours")) - if "person_id" in rd.joint_participants.columns: - jnm_counts = rd.joint_participants.group_by("person_id").agg( - pl.len().alias("jnumTours") + if can_use_person_days and person_days is not None: + inm_counts = individual_tours.group_by(["person_id", "day_id"]).agg( + pl.len().alias("inmTours") + ) + jnm_counts = ( + participant_days.group_by(["person_id", "day_id"]).agg( + pl.len().alias("jnumTours") + ) + if participant_days is not None + else pl.DataFrame( + schema={ + "person_id": pl.Int64, + "day_id": pl.Int64, + "jnumTours": pl.UInt32, + } + ) ) + source = person_days + count_keys = ["person_id", "day_id"] else: - jnm_counts = per.select("person_id").with_columns(pl.lit(0).alias("jnumTours")) + if "person_id" in individual_tours.columns: + inm_counts = individual_tours.group_by("person_id").agg( + pl.len().alias("inmTours") + ) + else: + inm_counts = per.select("person_id").with_columns( + pl.lit(0).alias("inmTours") + ) + + if "person_id" in rd.joint_participants.columns: + jnm_counts = rd.joint_participants.group_by("person_id").agg( + pl.len().alias("jnumTours") + ) + else: + jnm_counts = per.select("person_id").with_columns( + pl.lit(0).alias("jnumTours") + ) + source = per + count_keys = ["person_id"] per2 = ( - per.join(inm_counts, on="person_id", how="left") - .join(jnm_counts, on="person_id", how="left") + source.join(inm_counts, on=count_keys, how="left") + .join(jnm_counts, on=count_keys, how="left") .with_columns( [ pl.col("inmTours").fill_null(0), @@ -168,12 +582,12 @@ def indiv_nm_summary(rd: RunData, config: Config) -> pl.DataFrame: }, required_columns={ "per": ("person_id", "person_type", "finalweight"), - "tours": ("person_id", "tour_purpose"), + "tours": ("person_id", "tour_purpose", "finalweight"), }, ) def tour_rate_per_person(rd: RunData, config: Config) -> pl.DataFrame: person_required = {"person_id", "person_type", "finalweight"} - tour_required = {"person_id", "tour_purpose"} + tour_required = {"person_id", "tour_purpose", "finalweight"} if not person_required.issubset(set(rd.per.columns)) or not tour_required.issubset( set(rd.tours.columns) @@ -185,40 +599,31 @@ def tour_rate_per_person(rd: RunData, config: Config) -> pl.DataFrame: return tour_rate_per_person.empty() weighted_person_days = ( - rd.per.filter( - pl.col("person_id").is_not_null() - & pl.col("person_type").is_not_null() - & pl.col("finalweight").is_not_null() - ) - .select( - "person_id", - pl.col("person_type").cast(pl.Utf8), - pl.col("finalweight").cast(pl.Float64).alias("person_weight"), - ) + _person_exposure_records(rd) .group_by("person_type") .agg(weighted_person_days=pl.col("person_weight").sum()) ) + tours = rd.tours + if "tour_category" in tours.columns: + category = ( + pl.col("tour_category").cast(pl.Utf8).str.strip_chars().str.to_lowercase() + ) + tours = tours.filter(pl.col("tour_category").is_null() | (category != "atwork")) + weighted_tours = ( - rd.tours.filter( - pl.col("person_id").is_not_null() & pl.col(purpose_col).is_not_null() + _weighted_activity_by_person_type( + rd, + tours, + purpose_col=purpose_col, + participant_col="NUMBER_HH", ) - .join( - rd.per.filter( - pl.col("person_id").is_not_null() - & pl.col("person_type").is_not_null() - & pl.col("finalweight").is_not_null() - ).select( - "person_id", - pl.col("person_type").cast(pl.Utf8), - pl.col("finalweight").cast(pl.Float64).alias("person_weight"), - ), - on="person_id", - how="inner", + .rename( + { + "activity_purpose": "tour_purpose", + "activity_count": "weighted_tours", + } ) - .group_by(["person_type", purpose_col]) - .agg(weighted_tours=pl.col("person_weight").sum()) - .rename({purpose_col: "tour_purpose"}) .with_columns(pl.col("tour_purpose").cast(pl.Utf8)) ) @@ -284,24 +689,25 @@ def trip_rate_per_person(rd: RunData, config: Config) -> pl.DataFrame: return trip_rate_per_person.empty() person_totals = ( - rd.per.filter(pl.col("person_type").is_not_null()) + _person_exposure_records(rd) .group_by("person_type") - .agg(person_count=pl.col("finalweight").sum()) + .agg(person_count=pl.col("person_weight").sum()) .with_columns(pl.col("person_type").cast(pl.Utf8)) ) trip_totals = ( - rd.trips.filter( - pl.col("person_id").is_not_null() & pl.col("trip_purpose").is_not_null() + _weighted_activity_by_person_type( + rd, + rd.trips, + purpose_col="trip_purpose", + participant_col="num_participants", ) - .join( - rd.per.select("person_id", "person_type"), - on="person_id", - how="inner", + .rename( + { + "activity_purpose": "trip_purpose", + "activity_count": "trip_count", + } ) - .filter(pl.col("person_type").is_not_null()) - .group_by(["person_type", "trip_purpose"]) - .agg(trip_count=pl.col("finalweight").sum()) .with_columns( pl.col("person_type").cast(pl.Utf8), pl.col("trip_purpose").cast(pl.Utf8), diff --git a/processor/summarize/summaries/daily_travel_escort_distributions.py b/processor/summarize/summaries/daily_travel_escort_distributions.py index abfb1b1..da112a4 100644 --- a/processor/summarize/summaries/daily_travel_escort_distributions.py +++ b/processor/summarize/summaries/daily_travel_escort_distributions.py @@ -14,7 +14,7 @@ _sorted_distance_bins, ) from processor.summarize.summaries.summary_helpers import ( - _rounded_distance_bin_expr, + _distance_bin_expr, _summary_purpose_column, _trip_direction_expr, ) @@ -44,8 +44,10 @@ def adult_escorted_tour_distance_distribution_by_direction( if escorted.is_empty(): return adult_escorted_tour_distance_distribution_by_direction.empty() - base = escorted.filter(pl.col("SKIMDIST").is_not_null()).with_columns( - _rounded_distance_bin_expr("SKIMDIST") + base = ( + escorted.filter(pl.col("SKIMDIST").is_not_null()) + .with_columns(_distance_bin_expr("SKIMDIST")) + .filter(pl.col("distance_bin").is_not_null()) ) if base.is_empty(): return adult_escorted_tour_distance_distribution_by_direction.empty() @@ -105,9 +107,13 @@ def adult_escorted_trip_distance_distribution_by_direction( ): return adult_escorted_trip_distance_distribution_by_direction.empty() - base = trips.filter(pl.col("od_dist").is_not_null()).with_columns( - direction_expr, - _rounded_distance_bin_expr("od_dist"), + base = ( + trips.filter(pl.col("od_dist").is_not_null()) + .with_columns( + direction_expr, + _distance_bin_expr("od_dist"), + ) + .filter(pl.col("distance_bin").is_not_null()) ) if base.is_empty(): return adult_escorted_trip_distance_distribution_by_direction.empty() diff --git a/processor/summarize/summaries/daily_travel_escort_shared.py b/processor/summarize/summaries/daily_travel_escort_shared.py index ff71630..eb4e0f7 100644 --- a/processor/summarize/summaries/daily_travel_escort_shared.py +++ b/processor/summarize/summaries/daily_travel_escort_shared.py @@ -5,7 +5,11 @@ import polars as pl from processor.models import RunData -from processor.summarize.summaries.summary_helpers import _summary_purpose_column +from processor.summarize.summaries.summary_helpers import ( + _distance_bin_sort_expr, + _ensure_zero_distance_bin, + _summary_purpose_column, +) _CHILD_PERSON_TYPES = {"6", "7", "8"} _STUDENT_SCHOOL_ESCORT_TYPES = ("not_escorted", "pure_escort", "ride_share") @@ -219,14 +223,15 @@ def _sorted_distance_bins( value_col: str, ) -> pl.DataFrame: return ( - df.with_columns( + _ensure_zero_distance_bin( + df, + group_cols=[direction_col], + value_col=value_col, + ).with_columns( pl.col("distance_bin").cast(pl.Utf8), pl.col(direction_col).cast(pl.Utf8), pl.col(value_col).cast(pl.Float64), - pl.when(pl.col("distance_bin") == "40+") - .then(999) - .otherwise(pl.col("distance_bin").cast(pl.Int64, strict=False)) - .alias("_sort_distance"), + _distance_bin_sort_expr().alias("_sort_distance"), ) .select("distance_bin", direction_col, value_col, "_sort_distance") .sort([direction_col, "_sort_distance"]) diff --git a/processor/summarize/summaries/demographics.py b/processor/summarize/summaries/demographics.py index efd7b9b..6a179d0 100644 --- a/processor/summarize/summaries/demographics.py +++ b/processor/summarize/summaries/demographics.py @@ -4,6 +4,9 @@ from runtime.config import Config from processor.models import RunData from processor.summarize.contracts import summary +from processor.summarize.summaries.summary_helpers import ( + attach_person_travel_weight, +) @summary( @@ -77,16 +80,34 @@ def person_type(rd: RunData, config: Config) -> pl.DataFrame: }, ) def population_totals(rd: RunData, config: Config | None = None) -> pl.DataFrame: + tour_count = attach_person_travel_weight( + rd, + rd.tours, + participant_col="NUMBER_HH", + output_col="person_tour_weight", + )["person_tour_weight"].sum() + trip_count = attach_person_travel_weight( + rd, + rd.trips, + participant_col="num_participants", + output_col="person_trip_weight", + )["person_trip_weight"].sum() + stops = rd.trips.filter(pl.col("stops") == 1) + stop_count = attach_person_travel_weight( + rd, + stops, + participant_col="num_participants", + output_col="person_stop_weight", + )["person_stop_weight"].sum() + return pl.DataFrame( [ { "person_count": float(rd.per["finalweight"].sum()), "household_count": float(rd.hh["finalweight"].sum()), - "tour_count": float(rd.tours["finalweight"].sum()), - "trip_count": float(rd.trips["finalweight"].sum()), - "stop_count": float( - rd.trips.filter(pl.col("stops") == 1)["finalweight"].sum() - ), + "tour_count": float(tour_count), + "trip_count": float(trip_count), + "stop_count": float(stop_count), } ] ) diff --git a/processor/summarize/summaries/joint_travel.py b/processor/summarize/summaries/joint_travel.py index b7a47a6..b667581 100644 --- a/processor/summarize/summaries/joint_travel.py +++ b/processor/summarize/summaries/joint_travel.py @@ -5,6 +5,163 @@ from runtime.config import Config from processor.models import RunData from processor.summarize.contracts import summary +from processor.summarize.summaries.summary_helpers import ( + household_tour_weight_expr, + joint_party_size_expr, +) + + +JTF_PURPOSE_ALIASES = ( + ("shopping", "shop", "joint_shopping"), + ("othmaint", "maintenance", "maint", "other_maintenance", "joint_othmaint"), + ("eatout", "eating_out", "eat", "joint_eatout"), + ("social", "visiting", "visit", "joint_social"), + ( + "othdiscr", + "other_discretionary", + "discretionary", + "joint_othdiscr", + ), +) + +JTF_NAMES = ( + "No Joint Tours", + "1 Shopping", + "1 Maintenance", + "1 Eating Out", + "1 Visiting", + "1 Other Discretionary", + "2 Shopping", + "1 Shopping / 1 Maintenance", + "1 Shopping / 1 Eating Out", + "1 Shopping / 1 Visiting", + "1 Shopping / 1 Other Discretionary", + "2 Maintenance", + "1 Maintenance / 1 Eating Out", + "1 Maintenance / 1 Visiting", + "1 Maintenance / 1 Other Discretionary", + "2 Eating Out", + "1 Eating Out / 1 Visiting", + "1 Eating Out / 1 Other Discretionary", + "2 Visiting", + "1 Visiting / 1 Other Discretionary", + "2 Other Discretionary", +) + +# Each tuple is (alternative code, minimum count in each fixed purpose slot). +JTF_ALTERNATIVES = ( + (2, 1, 0, 0, 0, 0), + (3, 0, 1, 0, 0, 0), + (4, 0, 0, 1, 0, 0), + (5, 0, 0, 0, 1, 0), + (6, 0, 0, 0, 0, 1), + (7, 2, 0, 0, 0, 0), + (8, 1, 1, 0, 0, 0), + (9, 1, 0, 1, 0, 0), + (10, 1, 0, 0, 1, 0), + (11, 1, 0, 0, 0, 1), + (12, 0, 2, 0, 0, 0), + (13, 0, 1, 1, 0, 0), + (14, 0, 1, 0, 1, 0), + (15, 0, 1, 0, 0, 1), + (16, 0, 0, 2, 0, 0), + (17, 0, 0, 1, 1, 0), + (18, 0, 0, 1, 0, 1), + (19, 0, 0, 0, 2, 0), + (20, 0, 0, 0, 1, 1), + (21, 0, 0, 0, 0, 2), +) + + +def _has_multiday_household_history(rd: RunData) -> bool: + """Return whether prepared days span multiple diary days per household.""" + required = {"household_id", "day_num"} + if rd.day.is_empty() or not required.issubset(rd.day.columns): + return False + return not ( + rd.day.filter( + pl.col("household_id").is_not_null() & pl.col("day_num").is_not_null() + ) + .group_by("household_id") + .agg(pl.col("day_num").n_unique().alias("_day_count")) + .filter(pl.col("_day_count") > 1) + .is_empty() + ) + + +def _valid_identifier(column: str) -> pl.Expr: + text = pl.col(column).cast(pl.Utf8, strict=False).str.strip_chars() + numeric = pl.col(column).cast(pl.Float64, strict=False) + return ( + text.is_not_null() + & (text != "") + & ( + numeric.is_null() + | (numeric.is_finite() & (numeric > 0) & (numeric != 995)) + ) + ) + + +def _joint_identity_expr(df: pl.DataFrame, *, row_fallback: str | None = None) -> pl.Expr: + candidates: list[pl.Expr] = [] + for prefix, column in (("joint:", "joint_tour_id"), ("tour:", "tour_id")): + if column in df.columns: + candidates.append( + pl.when(_valid_identifier(column)).then( + pl.concat_str(pl.lit(prefix), pl.col(column).cast(pl.Utf8)) + ) + ) + if row_fallback is not None: + candidates.append( + pl.concat_str(pl.lit("row:"), pl.col(row_fallback).cast(pl.Utf8)) + ) + if not candidates: + return pl.lit(None, dtype=pl.Utf8) + return pl.coalesce(candidates) + + +def _household_observations(rd: RunData) -> tuple[pl.DataFrame, list[str]]: + hhsize_col = "HHSIZE" if "HHSIZE" in rd.hh.columns else "hhsize" + hh_columns = ["household_id", "finalweight"] + if hhsize_col in rd.hh.columns: + hh_columns.append(hhsize_col) + households = rd.hh.select(hh_columns) + if hhsize_col in households.columns and hhsize_col != "HHSIZE": + households = households.rename({hhsize_col: "HHSIZE"}) + + if _has_multiday_household_history(rd): + keys = ["household_id", "day_num"] + observations = ( + rd.day.filter( + pl.col("household_id").is_not_null() + & pl.col("day_num").is_not_null() + ) + .select(keys) + .unique() + .join(households, on="household_id", how="inner") + ) + return observations, keys + return households, ["household_id"] + + +def _unique_joint_tours(rd: RunData, observation_keys: list[str]) -> pl.DataFrame: + if not {"household_id", *observation_keys}.issubset(rd.tours.columns): + return pl.DataFrame() + if "tour_category" not in rd.tours.columns: + return rd.tours.head(0) + return ( + rd.tours.filter( + pl.col("tour_category").cast(pl.Utf8).str.strip_chars().str.to_lowercase() + == "joint" + ) + .with_row_index("_joint_row") + .with_columns( + _joint_identity_expr(rd.tours, row_fallback="_joint_row").alias( + "_joint_identity" + ) + ) + .unique([*observation_keys, "_joint_identity"], maintain_order=True) + ) @summary( @@ -18,30 +175,6 @@ ) def joint_tour_freq(rd: RunData, config: Config | None = None) -> pl.DataFrame: """Returns DataFrame: jtf_code, jtf_label, household_count.""" - JTF_NAMES = [ - "No Joint Tours", - "1 Shopping", - "1 Maintenance", - "1 Eating Out", - "1 Visiting", - "1 Other Discretionary", - "2 Shopping", - "2 Maintenance", - "2 Eating Out", - "2 Visiting", - "2 Other Discretionary", - "1 Shopping / 1 Maintenance", - "1 Shopping / 1 Eating Out", - "1 Shopping / 1 Visiting", - "1 Shopping / 1 Other Discretionary", - "1 Maintenance / 1 Eating Out", - "1 Maintenance / 1 Visiting", - "1 Maintenance / 1 Other Discretionary", - "1 Eating Out / 1 Visiting", - "1 Eating Out / 1 Other Discretionary", - "1 Visiting / 1 Other Discretionary", - ] - jtf_lookup = pl.DataFrame( { "jtf_code": list(range(1, 22)), @@ -53,71 +186,34 @@ def joint_tour_freq(rd: RunData, config: Config | None = None) -> pl.DataFrame: }, ) - if "tour_category" not in rd.tours.columns: - return jtf_lookup.with_columns(pl.lit(0.0).alias("household_count")) - - joint_tours = rd.tours.filter(pl.col("tour_category") == "joint") - if "tour_purpose" not in joint_tours.columns: - return jtf_lookup.with_columns(pl.lit(0.0).alias("household_count")) - # Map purpose strings to slot letters (a-e for up to 5 NM purposes) - nm_purposes = joint_tours["tour_purpose"].drop_nulls().unique().sort().to_list() - # Take up to 5 most common (for JTF coding) - purpose_slots = {p: f"j{i}" for i, p in enumerate(nm_purposes[:5])} - slot_cols = [f"j{i}" for i in range(len(purpose_slots))] - - hh_joint = pl.DataFrame({"household_id": rd.hh["household_id"]}) - - for purp, slot in purpose_slots.items(): - counts = ( - joint_tours.filter(pl.col("tour_purpose") == purp) - .group_by("household_id") - .agg(pl.len().cast(pl.Int64).alias(slot)) - ) - hh_joint = hh_joint.join(counts, on="household_id", how="left").with_columns( - pl.col(slot).fill_null(0) - ) - - for slot in slot_cols: - if slot not in hh_joint.columns: + observations, observation_keys = _household_observations(rd) + joint_tours = _unique_joint_tours(rd, observation_keys) + if joint_tours.width == 0 and _has_multiday_household_history(rd): + return joint_tour_freq.empty() + hh_joint = observations.select(observation_keys) + normalized_purpose = ( + pl.col("tour_purpose").cast(pl.Utf8).str.strip_chars().str.to_lowercase() + ) + slot_cols = [f"j{i}" for i in range(len(JTF_PURPOSE_ALIASES))] + for slot, aliases in zip(slot_cols, JTF_PURPOSE_ALIASES, strict=True): + if "tour_purpose" in joint_tours.columns: + counts = ( + joint_tours.filter(normalized_purpose.is_in(aliases)) + .group_by(observation_keys) + .agg(pl.len().cast(pl.Int64).alias(slot)) + ) + hh_joint = hh_joint.join( + counts, on=observation_keys, how="left" + ).with_columns(pl.col(slot).fill_null(0)) + else: hh_joint = hh_joint.with_columns(pl.lit(0).alias(slot)) - all_hh = rd.hh.select(["household_id", "finalweight"]) - hh_joint = all_hh.join(hh_joint, on="household_id", how="left") - - for slot in slot_cols: - if slot in hh_joint.columns: - hh_joint = hh_joint.with_columns(pl.col(slot).fill_null(0)) - - # TODO: JTF coding is simplified; verify category assignment matches formal ActivitySim joint tour frequency definitions. - - # Code JTF (simplified: 1=none, 2-6=single, 7-11=two same, 12-21=two different) + hh_joint = observations.join(hh_joint, on=observation_keys, how="left") + hh_joint = hh_joint.with_columns(pl.col(slot).fill_null(0) for slot in slot_cols) hh_joint = hh_joint.with_columns(pl.lit(1).alias("jtf")) - codes = [ - (2, 1, 0, 0, 0, 0), - (3, 0, 1, 0, 0, 0), - (4, 0, 0, 1, 0, 0), - (5, 0, 0, 0, 1, 0), - (6, 0, 0, 0, 0, 1), - (7, 2, 0, 0, 0, 0), - (8, 0, 2, 0, 0, 0), - (9, 0, 0, 2, 0, 0), - (10, 0, 0, 0, 2, 0), - (11, 0, 0, 0, 0, 2), - (12, 1, 1, 0, 0, 0), - (13, 1, 0, 1, 0, 0), - (14, 1, 0, 0, 1, 0), - (15, 1, 0, 0, 0, 1), - (16, 0, 1, 1, 0, 0), - (17, 0, 1, 0, 1, 0), - (18, 0, 1, 0, 0, 1), - (19, 0, 0, 1, 1, 0), - (20, 0, 0, 1, 0, 1), - (21, 0, 0, 0, 1, 1), - ] - - for code, *vals in codes: + for code, *vals in JTF_ALTERNATIVES: conds = [] for i, v in enumerate(vals): col = f"j{i}" @@ -176,16 +272,18 @@ def joint_tours_hhsize(rd: RunData, config: Config | None = None) -> pl.DataFram ): return joint_tours_hhsize.empty() + observations, observation_keys = _household_observations(rd) + if not set(observation_keys).issubset(rd.tours.columns): + return joint_tours_hhsize.empty() joint_tour_hhs = ( - rd.tours.filter(pl.col("tour_category") == "joint") - .select("household_id") + _unique_joint_tours(rd, observation_keys) + .select(observation_keys) .unique() .with_columns(has_joint_tour=pl.lit(True)) ) return ( - rd.hh.select(["household_id", "HHSIZE", "finalweight"]) - .join(joint_tour_hhs, on="household_id", how="left") + observations.join(joint_tour_hhs, on=observation_keys, how="left") .with_columns(has_joint_tour=pl.col("has_joint_tour").fill_null(False)) .group_by("HHSIZE") .agg( @@ -209,38 +307,50 @@ def joint_tours_hhsize(rd: RunData, config: Config | None = None) -> pl.DataFram required_columns={"tours": ("tour_category", "NUMBER_HH", "finalweight")}, ) def joint_party_size(rd: RunData, config: Config | None = None) -> pl.DataFrame: - """Joint tour party size distribution (capped at 5+). Columns: party_size (1-5), joint_tour_count.""" + """Joint tour party size distribution, with sizes of 5 or more capped at 5.""" if "tour_category" not in rd.tours.columns or "NUMBER_HH" not in rd.tours.columns: return joint_party_size.empty() - joint_tours = rd.tours.filter(pl.col("tour_category") == "joint") + joint_tours = ( + rd.tours.filter( + pl.col("tour_category").cast(pl.Utf8).str.to_lowercase() == "joint" + ) + .with_columns( + joint_party_size_expr(rd.tours).alias("_party_size"), + household_tour_weight_expr( + rd.tours, + output_col="_household_tour_weight", + ), + ) + .filter(pl.col("_party_size").is_not_null()) + ) if joint_tours.is_empty(): return joint_party_size.empty() df = ( - joint_tours.group_by("NUMBER_HH") - .agg(joint_tour_count=pl.col("finalweight").sum()) - .sort("NUMBER_HH") + joint_tours.group_by("_party_size") + .agg(joint_tour_count=pl.col("_household_tour_weight").sum()) + .sort("_party_size") ) - cap5 = df.filter(pl.col("NUMBER_HH") >= 5)["joint_tour_count"].sum() + cap5 = df.filter(pl.col("_party_size") >= 5)["joint_tour_count"].sum() - df = df.filter(pl.col("NUMBER_HH") < 5).with_columns( - pl.col("NUMBER_HH").cast(pl.Int32) + df = df.filter(pl.col("_party_size") < 5).with_columns( + pl.col("_party_size").cast(pl.Int32) ) cap_row = pl.DataFrame( { - "NUMBER_HH": pl.Series([5], dtype=pl.Int32), + "_party_size": pl.Series([5], dtype=pl.Int32), "joint_tour_count": pl.Series([cap5 or 0.0], dtype=pl.Float64), } ) return ( pl.concat([df, cap_row]) - .sort("NUMBER_HH") - .rename({"NUMBER_HH": "party_size"}) + .sort("_party_size") + .rename({"_party_size": "party_size"}) .select( pl.col("party_size").cast(pl.Int32), pl.col("joint_tour_count").cast(pl.Float64), @@ -271,8 +381,14 @@ def joint_composition(rd: RunData, config: Config | None = None) -> pl.DataFrame return joint_composition.empty() return ( - joint_tours.group_by(comp_col) - .agg(joint_tour_count=pl.col("finalweight").sum()) + joint_tours.with_columns( + household_tour_weight_expr( + joint_tours, + output_col="_household_tour_weight", + ) + ) + .group_by(comp_col) + .agg(joint_tour_count=pl.col("_household_tour_weight").sum()) .rename({comp_col: "tour_composition"}) .sort("tour_composition") ) @@ -286,38 +402,39 @@ def joint_composition(rd: RunData, config: Config | None = None) -> pl.DataFrame "joint_tour_count": pl.Float64, }, required_columns={ - "tours": ( - "tour_category", - "composition", - "number_of_participants", - "finalweight", - ) + "tours": ("tour_category", "finalweight") }, ) def joint_composition_by_party_size(rd: RunData, config: Config) -> pl.DataFrame: - required = { - "tour_category", - "composition", - "number_of_participants", - "finalweight", - } - if not required.issubset(set(rd.tours.columns)): + if not {"tour_category", "finalweight"}.issubset(rd.tours.columns): return joint_composition_by_party_size.empty() + joint_tours = rd.tours.filter( + pl.col("tour_category").cast(pl.Utf8).str.to_lowercase() == "joint" + ) + comp_col = ( + "composition" if "composition" in joint_tours.columns else "tour_composition" + ) + if comp_col not in joint_tours.columns: + return joint_composition_by_party_size.empty() return ( - rd.tours.filter( - pl.col("tour_category").cast(pl.Utf8).str.to_lowercase() == "joint" + joint_tours.with_columns( + joint_party_size_expr(joint_tours).alias("_party_size"), + household_tour_weight_expr( + joint_tours, + output_col="_household_tour_weight", + ), ) .filter( - pl.col("composition").is_not_null() - & pl.col("number_of_participants").is_not_null() + pl.col(comp_col).is_not_null() + & pl.col("_party_size").is_not_null() ) - .group_by(["composition", "number_of_participants"]) - .agg(joint_tour_count=pl.col("finalweight").sum()) + .group_by([comp_col, "_party_size"]) + .agg(joint_tour_count=pl.col("_household_tour_weight").sum()) .rename( { - "composition": "tour_composition", - "number_of_participants": "party_size", + comp_col: "tour_composition", + "_party_size": "party_size", } ) .with_columns( @@ -338,24 +455,99 @@ def joint_composition_by_party_size(rd: RunData, config: Config) -> pl.DataFrame "total_person_count": pl.Float64, }, required_columns={ - "per": ("household_id", "num_joint_tours", "finalweight"), - "hh": ("household_id", "hhsize"), + "per": ("household_id", "person_id", "finalweight"), + "hh": ("household_id",), }, ) def joint_participation_person_by_hhsize(rd: RunData, config: Config) -> pl.DataFrame: - person_required = {"household_id", "num_joint_tours", "finalweight"} - hh_required = {"household_id", "hhsize"} - - if not person_required.issubset(set(rd.per.columns)) or not hh_required.issubset( - set(rd.hh.columns) - ): + person_required = {"household_id", "person_id", "finalweight"} + hhsize_col = "HHSIZE" if "HHSIZE" in rd.hh.columns else "hhsize" + if not person_required.issubset(rd.per.columns) or hhsize_col not in rd.hh.columns: return joint_participation_person_by_hhsize.empty() - persons_with_hhsize = rd.per.join( - rd.hh.select("household_id", "hhsize"), - on="household_id", - how="left", - ).filter(pl.col("hhsize").is_not_null()) + if _has_multiday_household_history(rd) and { + "household_id", + "person_id", + "day_num", + }.issubset(rd.day.columns): + person_weights = rd.per.select( + "person_id", + pl.col("finalweight").cast(pl.Float64).alias("_person_weight"), + ).unique("person_id") + day_weight = ( + pl.col("finalweight").cast(pl.Float64).alias("_day_weight") + if "finalweight" in rd.day.columns + else pl.lit(None, dtype=pl.Float64).alias("_day_weight") + ) + persons_with_hhsize = ( + rd.day.select("household_id", "person_id", "day_num", day_weight) + .filter( + pl.col("person_id").is_not_null() & pl.col("day_num").is_not_null() + ) + .unique(["person_id", "day_num"]) + .join(person_weights, on="person_id", how="inner") + .with_columns( + pl.coalesce("_day_weight", "_person_weight").alias("finalweight") + ) + .join( + rd.hh.select( + "household_id", pl.col(hhsize_col).alias("hhsize") + ), + on="household_id", + how="left", + ) + ) + + tour_days = _unique_joint_tours(rd, ["household_id", "day_num"]) + if "_joint_identity" not in tour_days.columns: + return joint_participation_person_by_hhsize.empty() + tour_days = tour_days.select("_joint_identity", "day_num").unique() + if ( + tour_days.group_by("_joint_identity") + .agg(pl.col("day_num").n_unique().alias("_day_count")) + .filter(pl.col("_day_count") != 1) + .height + ): + return joint_participation_person_by_hhsize.empty() + participants = ( + rd.joint_participants.with_columns( + _joint_identity_expr(rd.joint_participants).alias("_joint_identity") + ) + .filter( + pl.col("person_id").is_not_null() + & pl.col("_joint_identity").is_not_null() + ) + .join(tour_days, on="_joint_identity", how="inner") + .select("person_id", "day_num") + .unique() + .with_columns(_has_joint_tour=pl.lit(True)) + ) + persons_with_hhsize = persons_with_hhsize.join( + participants, on=["person_id", "day_num"], how="left" + ).with_columns(pl.col("_has_joint_tour").fill_null(False)) + else: + if "num_joint_tours" in rd.per.columns: + participation = (pl.col("num_joint_tours") > 0).fill_null(False) + elif "person_id" in rd.joint_participants.columns: + participation = pl.col("person_id").is_in( + rd.joint_participants.select("person_id").drop_nulls().to_series() + ) + else: + participation = pl.lit(False) + persons_with_hhsize = ( + rd.per.join( + rd.hh.select( + "household_id", pl.col(hhsize_col).alias("hhsize") + ), + on="household_id", + how="left", + ) + .with_columns( + participation.alias("_has_joint_tour") + ) + ) + + persons_with_hhsize = persons_with_hhsize.filter(pl.col("hhsize").is_not_null()) if persons_with_hhsize.is_empty(): return joint_participation_person_by_hhsize.empty() @@ -365,9 +557,7 @@ def joint_participation_person_by_hhsize(rd: RunData, config: Config) -> pl.Data ) joint_tour_people = ( - persons_with_hhsize.filter( - pl.col("num_joint_tours").is_not_null() & (pl.col("num_joint_tours") > 0) - ) + persons_with_hhsize.filter(pl.col("_has_joint_tour")) .group_by("hhsize") .agg(joint_tour_person_weight=pl.col("finalweight").sum()) ) @@ -405,19 +595,20 @@ def joint_participation_person_by_hhsize(rd: RunData, config: Config) -> pl.Data def jtf_by_hhsize(rd: RunData, config: Config | None = None) -> pl.DataFrame: """Joint tour count category (0/1/2+) by HH size as proportions. Returns DataFrame: jtf, household_size, household_percent.""" - hh = rd.hh - - if "tour_category" not in rd.tours.columns or "HHSIZE" not in hh.columns: + if "tour_category" not in rd.tours.columns or "HHSIZE" not in rd.hh.columns: return jtf_by_hhsize.empty() - joint_tours = rd.tours.filter(pl.col("tour_category") == "joint") + observations, observation_keys = _household_observations(rd) + if not set(observation_keys).issubset(rd.tours.columns): + return jtf_by_hhsize.empty() + joint_tours = _unique_joint_tours(rd, observation_keys) - jt_counts = joint_tours.group_by("household_id").agg( + jt_counts = joint_tours.group_by(observation_keys).agg( pl.len().cast(pl.Int64).alias("jtours") ) hh2 = ( - hh.join(jt_counts, on="household_id", how="left") + observations.join(jt_counts, on=observation_keys, how="left") .with_columns(pl.col("jtours").fill_null(0)) .with_columns( pl.when(pl.col("jtours") == 0) diff --git a/processor/summarize/summaries/long_term_distance.py b/processor/summarize/summaries/long_term_distance.py index f9a1a48..cf9c954 100644 --- a/processor/summarize/summaries/long_term_distance.py +++ b/processor/summarize/summaries/long_term_distance.py @@ -6,17 +6,23 @@ from processor.models import RunData from processor.summarize.contracts import output_schema, summary -from processor.summarize.summaries.long_term_shared import _student_filter_expr +from processor.summarize.summaries.long_term_shared import ( + _internal_non_wfh_workers, + _student_filter_expr, +) from processor.summarize.summaries.summary_helpers import ( _configured_geography_columns, _configured_geography_dimensions, + _distance_bin_expr, + _distance_bin_labels, + _distance_bin_sort_expr, ) from runtime.config import Config @output_schema( schema={ - "distance_bin": pl.Int32, + "distance_bin": pl.Utf8, "geography_type": pl.Utf8, "geography_id": pl.Utf8, "person_count": pl.Float64, @@ -24,10 +30,8 @@ ) def tlfd(rd: RunData, config: Config) -> dict[str, pl.DataFrame]: def _bin_dist(df: pl.DataFrame, dist_col: str) -> pl.DataFrame: - return df.with_columns( - pl.col(dist_col).fill_null(0.0).clip(0, 9999) - ).with_columns( - pl.col(dist_col).cast(pl.Int32).clip(0, 51).alias("distance_bin") + return df.with_columns(_distance_bin_expr(dist_col, cap_value=51)).filter( + pl.col("distance_bin").is_not_null() ) def _make_tlfd(persons: pl.DataFrame, dist_col: str) -> pl.DataFrame: @@ -48,8 +52,11 @@ def _make_tlfd(persons: pl.DataFrame, dist_col: str) -> pl.DataFrame: ), dist_col, ) + if df.is_empty(): + return empty distance_bins = pl.DataFrame( - {"distance_bin": list(range(0, 52))}, schema={"distance_bin": pl.Int32} + {"distance_bin": _distance_bin_labels(51)}, + schema={"distance_bin": pl.Utf8}, ) outputs: list[pl.DataFrame] = [] @@ -112,24 +119,14 @@ def _make_tlfd(persons: pl.DataFrame, dist_col: str) -> pl.DataFrame: return ( pl.concat(outputs, how="vertical") - .sort(["geography_type", "geography_id", "distance_bin"]) + .with_columns(_distance_bin_sort_expr().alias("_sort_distance")) + .sort(["geography_type", "geography_id", "_sort_distance"]) + .drop("_sort_distance") .select("distance_bin", "geography_type", "geography_id", "person_count") ) ptype_col = "person_type" if "person_type" in rd.per.columns else None - workers = ( - rd.per.filter( - (pl.col("workplace_zone_id") > 0) - & ( - pl.col("is_worker") - .cast(pl.Utf8) - .str.to_lowercase() - .is_in(["true", "1"]) - ) - ) - if "is_worker" in rd.per.columns - else rd.per.head(0) - ) + workers = _internal_non_wfh_workers(rd.per) if ptype_col is not None: univ = ( @@ -164,12 +161,22 @@ def _make_tlfd(persons: pl.DataFrame, dist_col: str) -> pl.DataFrame: @summary( id="work_location_distance_distribution_by_geography", schema={ - "distance_bin": pl.Int32, + "distance_bin": pl.Utf8, "geography_type": pl.Utf8, "geography_id": pl.Utf8, "person_count": pl.Float64, }, - required_columns={"per": ("distance_to_work", "finalweight")}, + required_columns={ + "per": ( + "distance_to_work", + "external_workplace_zone_id", + "finalweight", + "is_external_worker", + "is_worker", + "work_from_home", + "workplace_zone_id", + ) + }, ) def work_tlfd(rd: RunData, config: Config) -> pl.DataFrame: return tlfd(rd, config)["work"] @@ -178,7 +185,7 @@ def work_tlfd(rd: RunData, config: Config) -> pl.DataFrame: @summary( id="university_location_distance_distribution_by_geography", schema={ - "distance_bin": pl.Int32, + "distance_bin": pl.Utf8, "geography_type": pl.Utf8, "geography_id": pl.Utf8, "person_count": pl.Float64, @@ -192,7 +199,7 @@ def univ_tlfd(rd: RunData, config: Config) -> pl.DataFrame: @summary( id="school_location_distance_distribution_by_geography", schema={ - "distance_bin": pl.Int32, + "distance_bin": pl.Utf8, "geography_type": pl.Utf8, "geography_id": pl.Utf8, "person_count": pl.Float64, diff --git a/processor/summarize/summaries/long_term_person.py b/processor/summarize/summaries/long_term_person.py index 0f2f2f9..9a9c48b 100644 --- a/processor/summarize/summaries/long_term_person.py +++ b/processor/summarize/summaries/long_term_person.py @@ -33,11 +33,15 @@ def license_holding_status(rd: RunData, config: Config) -> pl.DataFrame: if not required.issubset(set(rd.per.columns)): return license_holding_status.empty() + eligible_driver = ( + pl.col("can_drive").cast(pl.Int64, strict=False).is_in([1, 3]) + if "can_drive" in rd.per.columns + else pl.col("age") >= 16 + ) base = rd.per.filter( pl.col("person_type").is_not_null() & pl.col("has_license").is_not_null() - & pl.col("age").is_not_null() - & (pl.col("age") >= 16) + & eligible_driver ).with_columns( pl.col("person_type").cast(pl.Utf8), pl.when(pl.col("has_license")) diff --git a/processor/summarize/summaries/long_term_shared.py b/processor/summarize/summaries/long_term_shared.py index bffd301..12f3ebe 100644 --- a/processor/summarize/summaries/long_term_shared.py +++ b/processor/summarize/summaries/long_term_shared.py @@ -20,6 +20,47 @@ def _worker_filter_expr() -> pl.Expr: ) +def _internal_non_wfh_workers(persons: pl.DataFrame) -> pl.DataFrame: + """Return workers with an internal, non-home workplace assignment.""" + required = {"is_worker", "workplace_zone_id", "work_from_home"} + if not required.issubset(persons.columns): + return persons.head(0) + + external_expr: pl.Expr | None = None + if "is_external_worker" in persons.columns: + external_expr = ( + pl.col("is_external_worker") + .cast(pl.Utf8) + .str.to_lowercase() + .is_in(["true", "1", "yes", "external"]) + .fill_null(True) + ) + if "external_workplace_zone_id" in persons.columns: + external_location = ( + pl.col("external_workplace_zone_id").cast(pl.Float64, strict=False) > 0 + ).fill_null(False) + external_expr = ( + external_location + if external_expr is None + else external_expr | external_location + ) + if external_expr is None: + return persons.head(0) + + works_from_home = ( + pl.col("work_from_home") + .cast(pl.Utf8) + .str.to_lowercase() + .is_in(["true", "1", "yes", "work_from_home", "home"]) + ) + return persons.filter( + _worker_filter_expr() + & (pl.col("workplace_zone_id").cast(pl.Float64, strict=False) > 0) + & ~works_from_home + & ~external_expr.fill_null(True) + ) + + def _student_filter_expr() -> pl.Expr: return ( pl.col("is_student") diff --git a/processor/summarize/summaries/skimjoin.py b/processor/summarize/summaries/skimjoin.py index eb20983..3c0369e 100644 --- a/processor/summarize/summaries/skimjoin.py +++ b/processor/summarize/summaries/skimjoin.py @@ -43,6 +43,21 @@ def _mode_value(values: list[float], weights: list[float]) -> float | None: return min(value for value, weight in totals.items() if weight == best_weight) +def _weighted_nonzero_mean( + values: list[float], + weights: list[float], +) -> float | None: + nonzero_pairs = [ + (value, weight) + for value, weight in zip(values, weights, strict=False) + if value != 0.0 + ] + nonzero_weight = sum(weight for _, weight in nonzero_pairs) + if nonzero_weight <= 0: + return None + return sum(value * weight for value, weight in nonzero_pairs) / nonzero_weight + + def _weighted_quantile( values: list[float], weights: list[float], @@ -338,6 +353,7 @@ def _weighted_stats_row( "n_total": n_total, "n_valid": 0.0, "mean": None, + "mean_nonzero": None, "std": None, "min": None, "max": None, @@ -364,6 +380,7 @@ def _weighted_stats_row( "n_total": n_total, "n_valid": 0.0, "mean": None, + "mean_nonzero": None, "std": None, "min": None, "max": None, @@ -394,6 +411,7 @@ def _weighted_stats_row( "n_total": n_total, "n_valid": n_valid, "mean": mean, + "mean_nonzero": _weighted_nonzero_mean(values, weights), "std": math.sqrt(variance), "min": min(values), "max": max(values), @@ -476,6 +494,7 @@ def _weighted_stats_row_from_sidecar_group( "n_total": n_total, "n_valid": 0.0, "mean": None, + "mean_nonzero": None, "std": None, "min": None, "max": None, @@ -518,6 +537,7 @@ def _weighted_stats_row_from_sidecar_group( "n_total": n_total, "n_valid": n_valid, "mean": mean, + "mean_nonzero": _weighted_nonzero_mean(values, weights), "std": math.sqrt(variance), "min": min(values), "max": max(values), @@ -580,6 +600,7 @@ def _weighted_ecdf_rows_from_sidecar_group( "n_total": pl.Float64, "n_valid": pl.Float64, "mean": pl.Float64, + "mean_nonzero": pl.Float64, "std": pl.Float64, "min": pl.Float64, "max": pl.Float64, @@ -630,6 +651,7 @@ def trip_skim_component_ecdf(rd: RunData, config: Config) -> pl.DataFrame: "n_total": pl.Float64, "n_valid": pl.Float64, "mean": pl.Float64, + "mean_nonzero": pl.Float64, "std": pl.Float64, "min": pl.Float64, "max": pl.Float64, diff --git a/processor/summarize/summaries/summary_helpers.py b/processor/summarize/summaries/summary_helpers.py index 4ed1d0f..9279819 100644 --- a/processor/summarize/summaries/summary_helpers.py +++ b/processor/summarize/summaries/summary_helpers.py @@ -7,6 +7,7 @@ import numpy as np import polars as pl +from processor.models import RunData from processor.tour_purpose import purpose_column ALL_TOUR_PURPOSES = "all_tour_purposes" @@ -30,6 +31,181 @@ def weighted_group_sum( return df.group_by(group_cols).agg(pl.col(weight_col).sum().alias(output_col)) +def joint_participant_weight_expr( + df: pl.DataFrame, + *, + participant_col: str, + output_col: str, + weight_col: str = "finalweight", + category_col: str = "tour_category", +) -> pl.Expr: + """Return person-level weight, expanding valid joint-party records only.""" + weight = pl.col(weight_col).cast(pl.Float64) + if category_col not in df.columns or participant_col not in df.columns: + return weight.alias(output_col) + + participant_count = pl.col(participant_col).cast(pl.Float64, strict=False) + valid_participant_count = ( + participant_count.is_not_null() + & (participant_count > 0) + & (participant_count < 995) + ) + joint_record = ( + pl.col(category_col).cast(pl.Utf8).str.strip_chars().str.to_lowercase() + == "joint" + ) + multiplier = pl.when(valid_participant_count).then(participant_count).otherwise(1.0) + return ( + pl.when(joint_record) + .then(weight * multiplier) + .otherwise(weight) + .alias(output_col) + ) + + +def attach_person_travel_weight( + rd: RunData, + activity: pl.DataFrame, + *, + participant_col: str, + output_col: str, + weight_col: str = "finalweight", + category_col: str = "tour_category", + tour_id_col: str = "tour_id", +) -> pl.DataFrame: + """Attach a person-tour or person-trip weight to every activity row.""" + weighted = activity.with_columns( + joint_participant_weight_expr( + activity, + participant_col=participant_col, + output_col=output_col, + weight_col=weight_col, + category_col=category_col, + ) + ) + if ( + category_col not in activity.columns + or tour_id_col not in activity.columns + or not {tour_id_col, "person_id"}.issubset(rd.joint_participants.columns) + or not {"person_id", "finalweight"}.issubset(rd.per.columns) + or rd.joint_participants.is_empty() + or rd.per.is_empty() + ): + return weighted + + attributed_col = "_attributed_person_travel_weight" + participant_weights = ( + rd.joint_participants.filter(pl.col(tour_id_col).is_not_null()) + .select(tour_id_col, "person_id") + .join( + rd.per.filter( + pl.col("person_id").is_not_null() + & pl.col("finalweight").is_not_null() + ).select( + "person_id", + pl.col("finalweight").cast(pl.Float64).alias(attributed_col), + ), + on="person_id", + how="inner", + ) + .group_by(tour_id_col) + .agg(pl.col(attributed_col).sum()) + ) + if participant_weights.is_empty(): + return weighted + + joint_record = ( + pl.col(category_col).cast(pl.Utf8).str.strip_chars().str.to_lowercase() + == "joint" + ) + return ( + weighted.join(participant_weights, on=tour_id_col, how="left") + .with_columns( + pl.when(joint_record & pl.col(attributed_col).is_not_null()) + .then(pl.col(attributed_col)) + .otherwise(pl.col(output_col)) + .alias(output_col) + ) + .drop(attributed_col) + ) + + +def joint_party_size_expr(df: pl.DataFrame) -> pl.Expr: + """Resolve a valid household joint-tour party size from prepared columns.""" + candidates: list[pl.Expr] = [] + for column in ("number_of_participants", "NUMBER_HH"): + if column not in df.columns: + continue + party_size = pl.col(column).cast(pl.Float64, strict=False) + candidates.append( + pl.when( + party_size.is_not_null() + & party_size.is_finite() + & (party_size >= 2) + & (party_size < 995) + ).then(party_size) + ) + if not candidates: + return pl.lit(None, dtype=pl.Float64) + return pl.coalesce(candidates) + + +def household_tour_weight_expr( + df: pl.DataFrame, + *, + output_col: str, + weight_col: str = "finalweight", + category_col: str = "tour_category", + joint_tour_id_col: str = "joint_tour_id", +) -> pl.Expr: + """Return a row-preserving household-tour representation weight. + + Some survey tour tables contain one joint-tour row per participant. When a + valid joint-tour ID has exactly the reported number of participant rows, + each row receives 1/k of its source weight. ActivitySim's one-row joint + tours and incomplete or malformed groups retain their full source weight. + """ + weight = pl.col(weight_col).cast(pl.Float64) + if category_col not in df.columns or joint_tour_id_col not in df.columns: + return weight.alias(output_col) + + joint_record = ( + pl.col(category_col).cast(pl.Utf8).str.strip_chars().str.to_lowercase() + == "joint" + ) + joint_tour_id = pl.col(joint_tour_id_col) + joint_tour_id_text = joint_tour_id.cast(pl.Utf8, strict=False).str.strip_chars() + numeric_joint_tour_id = joint_tour_id.cast(pl.Float64, strict=False) + valid_joint_tour_id = ( + joint_tour_id_text.is_not_null() + & (joint_tour_id_text != "") + & ( + numeric_joint_tour_id.is_null() + | ( + numeric_joint_tour_id.is_finite() + & (numeric_joint_tour_id > 0) + & (numeric_joint_tour_id != 995) + ) + ) + ) + + party_size = joint_party_size_expr(df) + represented_rows = ( + pl.when(joint_record).then(1).otherwise(0).sum().over(joint_tour_id_col) + ).cast(pl.Float64) + complete_participant_group = ( + (represented_rows > 1) + & (party_size.min().over(joint_tour_id_col) == represented_rows) + & (party_size.max().over(joint_tour_id_col) == represented_rows) + ) + return ( + pl.when(joint_record & valid_joint_tour_id & complete_participant_group) + .then(weight / represented_rows) + .otherwise(weight) + .alias(output_col) + ) + + def _first_existing_column(columns: list[str], candidates: list[str]) -> str | None: candidate_set = set(columns) for candidate in candidates: @@ -410,17 +586,69 @@ def aggregate_weighted_average_across_geographies( return pl.concat(outputs, how="vertical") -def _rounded_distance_bin_expr(distance_col: str) -> pl.Expr: - """Return the common 0-decimal distance bin label expression with 40+ cap.""" - rounded = pl.col(distance_col).cast(pl.Float64).round(0) +POSITIVE_SUBMILE_DISTANCE_BIN = ">0-<1" + + +def _distance_bin_labels(cap_value: int = 40) -> list[str]: + """Return the common exact-zero, sub-mile, whole-mile, and terminal bins.""" + return [ + "0", + POSITIVE_SUBMILE_DISTANCE_BIN, + *[str(value) for value in range(1, cap_value)], + f"{cap_value}+", + ] + + +def _distance_bin_expr(distance_col: str, *, cap_value: int = 40) -> pl.Expr: + """Bin usable distances without combining exact zero and sub-mile values.""" + distance = pl.col(distance_col).cast(pl.Float64, strict=False) return ( - pl.when(rounded >= 40) - .then(pl.lit("40+")) - .otherwise(rounded.cast(pl.Int64, strict=False).cast(pl.Utf8)) + pl.when(~distance.is_finite() | (distance < 0)) + .then(pl.lit(None, dtype=pl.Utf8)) + .when(distance == 0) + .then(pl.lit("0")) + .when(distance < 1) + .then(pl.lit(POSITIVE_SUBMILE_DISTANCE_BIN)) + .when(distance >= cap_value) + .then(pl.lit(f"{cap_value}+")) + .otherwise(distance.floor().cast(pl.Int64, strict=False).cast(pl.Utf8)) .alias("distance_bin") ) +def _distance_bin_sort_expr(column: str = "distance_bin") -> pl.Expr: + """Sort the common distance-bin labels by their numeric axis position.""" + label = pl.col(column).cast(pl.Utf8) + numeric = label.str.replace(r"\+$", "").cast(pl.Float64, strict=False) + return ( + pl.when(label == POSITIVE_SUBMILE_DISTANCE_BIN) + .then(pl.lit(0.5)) + .when(numeric.is_null()) + .then(pl.lit(float("inf"))) + .otherwise(numeric) + ) + + +def _ensure_zero_distance_bin( + df: pl.DataFrame, + *, + group_cols: list[str], + value_col: str, +) -> pl.DataFrame: + """Add a zero-valued exact-zero bin for every represented group.""" + if df.is_empty(): + return df + zero_rows = df.select(group_cols).unique().with_columns( + pl.lit("0").alias("distance_bin"), + pl.lit(0.0).alias(value_col), + ) + return ( + pl.concat([df, zero_rows.select(df.columns)], how="vertical") + .group_by(["distance_bin", *group_cols]) + .agg(pl.col(value_col).sum()) + ) + + def _trip_direction_expr(df: pl.DataFrame) -> pl.Expr | None: """Infer a standard outbound/inbound direction label from trip flags.""" if "inbound" in df.columns: diff --git a/processor/summarize/summaries/tour.py b/processor/summarize/summaries/tour.py index 138411f..3832d68 100644 --- a/processor/summarize/summaries/tour.py +++ b/processor/summarize/summaries/tour.py @@ -4,7 +4,10 @@ from processor.models import RunData from processor.summarize.contracts import summary -from processor.summarize.summaries.summary_helpers import _summary_purpose_column +from processor.summarize.summaries.summary_helpers import ( + _summary_purpose_column, + household_tour_weight_expr, +) from runtime.config import Config @@ -20,8 +23,14 @@ def tour_category(rd: RunData, config: Config) -> pl.DataFrame: return ( rd.tours.filter(pl.col("tour_category").is_not_null()) + .with_columns( + household_tour_weight_expr( + rd.tours, + output_col="_tour_weight", + ) + ) .group_by("tour_category") - .agg(tour_count=pl.col("finalweight").sum()) + .agg(tour_count=pl.col("_tour_weight").sum()) .with_columns( pl.col("tour_category").cast(pl.Utf8), pl.col("tour_count").cast(pl.Float64), @@ -46,8 +55,14 @@ def tour_purpose(rd: RunData, config: Config) -> pl.DataFrame: return ( rd.tours.filter(pl.col(purpose_col).is_not_null()) + .with_columns( + household_tour_weight_expr( + rd.tours, + output_col="_tour_weight", + ) + ) .group_by(purpose_col) - .agg(tour_count=pl.col("finalweight").sum()) + .agg(tour_count=pl.col("_tour_weight").sum()) .with_columns( pl.col(purpose_col).cast(pl.Utf8).alias("tour_purpose"), pl.col("tour_count").cast(pl.Float64), diff --git a/processor/summarize/summaries/tour_geography.py b/processor/summarize/summaries/tour_geography.py index 1018d1b..7a28723 100644 --- a/processor/summarize/summaries/tour_geography.py +++ b/processor/summarize/summaries/tour_geography.py @@ -6,6 +6,7 @@ from processor.models import RunData from processor.summarize.contracts import summary +from processor.summarize.summaries.long_term_shared import _internal_non_wfh_workers from processor.summarize.summaries.summary_helpers import ( aggregate_counts_across_geographies, _configured_geography_columns, @@ -25,24 +26,12 @@ "average_tour_distance": pl.Float64, "person_count": pl.Float64, }, - required_columns={"per": ("finalweight",)}, + required_columns={"per": ("finalweight", "home_zone_id")}, ) def avg_mand_tour_distance(rd: RunData, config: Config) -> pl.DataFrame: ptype_col = "person_type" if "person_type" in rd.per.columns else None - workers = ( - rd.per.filter( - (pl.col("workplace_zone_id") > 0) - & ( - pl.col("is_worker") - .cast(pl.Utf8) - .str.to_lowercase() - .is_in(["true", "1"]) - ) - ) - if "is_worker" in rd.per.columns - else rd.per.head(0) - ) + workers = _internal_non_wfh_workers(rd.per) if ptype_col is not None: univ_s = ( @@ -56,7 +45,7 @@ def avg_mand_tour_distance(rd: RunData, config: Config) -> pl.DataFrame: ) & (pl.col(ptype_col).cast(pl.Utf8) == "3") ) - if "is_student" in rd.per.columns + if {"is_student", "school_zone_id"}.issubset(rd.per.columns) else rd.per.head(0) ) schl_s = ( @@ -70,7 +59,7 @@ def avg_mand_tour_distance(rd: RunData, config: Config) -> pl.DataFrame: ) & (pl.col(ptype_col).cast(pl.Utf8).cast(pl.Int32, strict=False) >= 6) ) - if "is_student" in rd.per.columns + if {"is_student", "school_zone_id"}.issubset(rd.per.columns) else rd.per.head(0) ) else: @@ -186,7 +175,6 @@ def _avg_by_geo( "person_id", "tour_category", "tour_purpose", - "SKIMDIST", "finalweight", ), }, @@ -197,19 +185,29 @@ def avg_non_mand_tour_distance(rd: RunData, config: Config) -> pl.DataFrame: "person_id", "tour_category", "tour_purpose", - "SKIMDIST", "finalweight", } if not person_required.issubset(set(rd.per.columns)) or not tour_required.issubset( set(rd.tours.columns) ): return avg_non_mand_tour_distance.empty() + distance_col = "SKIMDIST" + if distance_col not in rd.tours.columns: + return avg_non_mand_tour_distance.empty() + category = ( + pl.col("tour_category") + .cast(pl.Utf8) + .str.strip_chars() + .str.to_lowercase() + .str.replace_all("-", "_") + .str.replace_all(" ", "_") + ) tours = rd.tours.filter( - (pl.col("tour_category").cast(pl.Utf8).str.to_lowercase() == "non_mandatory") + (category == "non_mandatory") & pl.col("person_id").is_not_null() & pl.col("tour_purpose").is_not_null() - & pl.col("SKIMDIST").is_not_null() + & pl.col(distance_col).is_not_null() & pl.col("finalweight").is_not_null() ) purpose_col = _summary_purpose_column(rd.tours) @@ -233,7 +231,7 @@ def avg_non_mand_tour_distance(rd: RunData, config: Config) -> pl.DataFrame: .filter(pl.col("home_zone_id").is_not_null()) .select( "tour_purpose", - "SKIMDIST", + distance_col, "finalweight", "home_zone_id", *_configured_geography_columns( @@ -256,7 +254,7 @@ def avg_non_mand_tour_distance(rd: RunData, config: Config) -> pl.DataFrame: base_col="home_zone_id", role_prefix="home", ), - value_col="SKIMDIST", + value_col=distance_col, output_col="average_tour_distance", group_cols=["tour_purpose"], count_col="tour_count", @@ -273,7 +271,7 @@ def avg_non_mand_tour_distance(rd: RunData, config: Config) -> pl.DataFrame: base.group_by("tour_purpose") .agg( average_tour_distance=( - (pl.col("SKIMDIST") * pl.col("finalweight")).sum() + (pl.col(distance_col) * pl.col("finalweight")).sum() / pl.col("finalweight").sum() ).cast(pl.Float64), tour_count=pl.col("finalweight").sum().cast(pl.Float64), diff --git a/processor/summarize/summaries/tour_profiles.py b/processor/summarize/summaries/tour_profiles.py index 9057ed1..4a2161a 100644 --- a/processor/summarize/summaries/tour_profiles.py +++ b/processor/summarize/summaries/tour_profiles.py @@ -9,23 +9,17 @@ from processor.summarize.summaries.summary_helpers import ( ALL_TOUR_PURPOSES, _all_purpose_rollup, - _rounded_distance_bin_expr, + _distance_bin_expr, + _distance_bin_sort_expr, + _ensure_zero_distance_bin, _summary_purpose_column, + attach_person_travel_weight, + household_tour_weight_expr, weighted_group_sum, ) from runtime.config import Config -def _tour_weights_for_summary(tours: pl.DataFrame) -> pl.DataFrame: - """Apply current joint-tour weighting semantics for tour-level summaries.""" - return tours.with_columns( - pl.when(pl.col("tour_category").cast(pl.Utf8).str.to_lowercase() == "joint") - .then(pl.col("finalweight") * pl.col("NUMBER_HH").cast(pl.Float64)) - .otherwise(pl.col("finalweight")) - .alias("_tour_weight"), - ) - - @summary( id="tour_mode_by_tour_purpose_and_auto_sufficiency", schema={ @@ -37,29 +31,43 @@ def _tour_weights_for_summary(tours: pl.DataFrame) -> pl.DataFrame: "tour_count_all_households": pl.Float64, }, required_columns={ - "tours": ("tour_mode", "tour_purpose", "finalweight", "AUTOSUFF") + "tours": ( + "tour_mode", + "tour_purpose", + "tour_category", + "NUMBER_HH", + "finalweight", + "AUTOSUFF", + ) }, ) def tour_mode(rd: RunData, config: Config) -> pl.DataFrame: - required = {"tour_mode", "tour_purpose", "finalweight", "AUTOSUFF"} - if ( - not required.issubset(set(rd.tours.columns)) - or "tour_category" not in rd.tours.columns - ): + required = { + "tour_mode", + "tour_purpose", + "tour_category", + "NUMBER_HH", + "finalweight", + "AUTOSUFF", + } + if not required.issubset(set(rd.tours.columns)): return tour_mode.empty() purpose_col = _summary_purpose_column(rd.tours) if not purpose_col: return tour_mode.empty() - base = ( + base = attach_person_travel_weight( + rd, rd.tours.filter( pl.col(purpose_col).is_not_null() & pl.col("tour_mode").is_not_null() & pl.col("AUTOSUFF").is_not_null() - ) - .with_columns(pl.col(purpose_col).cast(pl.Utf8).alias("tour_purpose")) - .pipe(_tour_weights_for_summary) + ), + participant_col="NUMBER_HH", + output_col="_tour_weight", + ).with_columns( + pl.col(purpose_col).cast(pl.Utf8).alias("tour_purpose") ) if base.is_empty(): return tour_mode.empty() @@ -161,6 +169,10 @@ def stop_freq(rd: RunData, config: Config) -> pl.DataFrame: pl.col("num_ob_stops").clip(0, 3).alias("outbound_stop_count"), pl.col("num_ib_stops").clip(0, 3).alias("inbound_stop_count"), pl.col("num_tot_stops").clip(0, 6).alias("total_stop_count"), + household_tour_weight_expr( + rd.tours, + output_col="_tour_weight", + ), ] ) .group_by( @@ -171,7 +183,7 @@ def stop_freq(rd: RunData, config: Config) -> pl.DataFrame: "total_stop_count", ] ) - .agg(tour_count=pl.col("finalweight").sum()) + .agg(tour_count=pl.col("_tour_weight").sum()) .select( "tour_purpose", "outbound_stop_count", @@ -242,13 +254,13 @@ def at_work_sub_tour_freq(rd: RunData, config: Config) -> pl.DataFrame: "arrival_tour_count": pl.Float64, "duration_tour_count": pl.Float64, }, - required_columns={"tours": ("tour_category", "tour_purpose", "finalweight")}, + required_columns={ + "tours": ("tour_category", "tour_purpose", "finalweight") + }, ) def tour_tod(rd: RunData, config: Config) -> pl.DataFrame: - if ( - "tour_category" not in rd.tours.columns - or "tour_purpose" not in rd.tours.columns - ): + required = {"tour_category", "tour_purpose", "finalweight"} + if not required.issubset(set(rd.tours.columns)): return tour_tod.empty() purpose_col = _summary_purpose_column(rd.tours) if not purpose_col: @@ -256,8 +268,13 @@ def tour_tod(rd: RunData, config: Config) -> pl.DataFrame: base = ( rd.tours.filter(pl.col(purpose_col).is_not_null()) - .with_columns(pl.col(purpose_col).cast(pl.Utf8).alias("tour_purpose")) - .pipe(_tour_weights_for_summary) + .with_columns( + pl.col(purpose_col).cast(pl.Utf8).alias("tour_purpose"), + household_tour_weight_expr( + rd.tours, + output_col="_tour_weight", + ), + ) ) purposes = base["tour_purpose"].drop_nulls().unique().sort().to_list() @@ -267,7 +284,7 @@ def tour_tod(rd: RunData, config: Config) -> pl.DataFrame: max_period = int(rd.tours["start_hour"].max()) except Exception: max_period = 48 - bins = list(range(1, 25 if max_period <= 24 else 49)) + bins = list(range(0, 25 if max_period <= 24 else 49)) def _hist(df: pl.DataFrame, col: str, filt) -> pl.DataFrame: if col not in df.columns: @@ -276,7 +293,7 @@ def _hist(df: pl.DataFrame, col: str, filt) -> pl.DataFrame: df.filter(filt) .select([col, "_tour_weight"]) .with_columns(pl.col(col).cast(pl.Int32).alias("time_bin")) - .filter(pl.col("time_bin").is_between(1, bins[-1])) + .filter(pl.col("time_bin").is_between(0, bins[-1])) ) counts = sub.group_by("time_bin").agg(pl.col("_tour_weight").sum().alias("n")) return ( @@ -347,8 +364,6 @@ def _hist(df: pl.DataFrame, col: str, filt) -> pl.DataFrame: "tours": ( "tour_purpose", "tour_category", - "number_of_participants", - "SKIMDIST", "finalweight", ) }, @@ -357,8 +372,6 @@ def tour_distance(rd: RunData, config: Config) -> pl.DataFrame: required = { "tour_purpose", "tour_category", - "number_of_participants", - "SKIMDIST", "finalweight", } if not required.issubset(set(rd.tours.columns)): @@ -366,36 +379,37 @@ def tour_distance(rd: RunData, config: Config) -> pl.DataFrame: purpose_col = _summary_purpose_column(rd.tours) if not purpose_col: return tour_distance.empty() + distance_col = "SKIMDIST" + if distance_col not in rd.tours.columns: + return tour_distance.empty() base = ( rd.tours.filter( - pl.col(purpose_col).is_not_null() & pl.col("SKIMDIST").is_not_null() + pl.col(purpose_col).is_not_null() + & pl.col(distance_col).is_not_null() + & pl.col("tour_category") + .cast(pl.Utf8) + .str.strip_chars() + .str.to_lowercase() + .str.replace_all("-", "_") + .str.replace_all(" ", "_") + .is_in(["non_mandatory", "joint", "atwork"]) ) .with_columns( pl.col(purpose_col).cast(pl.Utf8).alias("tour_purpose"), - pl.col("SKIMDIST") - .cast(pl.Float64) - .round(0) - .alias("distance_miles_rounded"), - ) - .with_columns( - pl.when(pl.col("tour_category").cast(pl.Utf8).str.to_lowercase() == "joint") - .then( - pl.col("finalweight") - * pl.coalesce( - [pl.col("number_of_participants").cast(pl.Float64), pl.lit(1.0)] - ) - ) - .otherwise(pl.col("finalweight")) - .alias("adjusted_weight"), - _rounded_distance_bin_expr("distance_miles_rounded"), + household_tour_weight_expr( + rd.tours, + output_col="_tour_weight", + ), ) + .with_columns(_distance_bin_expr(distance_col)) + .filter(pl.col("distance_bin").is_not_null()) ) by_purpose = weighted_group_sum( base, ["distance_bin", "tour_purpose"], - weight_col="adjusted_weight", + weight_col="_tour_weight", output_col="tour_count", ) all_purposes = _all_purpose_rollup( @@ -404,15 +418,16 @@ def tour_distance(rd: RunData, config: Config) -> pl.DataFrame: value_col="tour_count", ) return ( - pl.concat([by_purpose, all_purposes], how="vertical") + _ensure_zero_distance_bin( + pl.concat([by_purpose, all_purposes], how="vertical"), + group_cols=["tour_purpose"], + value_col="tour_count", + ) .with_columns( pl.col("distance_bin").cast(pl.Utf8), pl.col("tour_purpose").cast(pl.Utf8), pl.col("tour_count").cast(pl.Float64), - pl.when(pl.col("distance_bin") == "40+") - .then(999) - .otherwise(pl.col("distance_bin").cast(pl.Int64, strict=False)) - .alias("_sort_distance"), + _distance_bin_sort_expr().alias("_sort_distance"), ) .select("distance_bin", "tour_purpose", "tour_count", "_sort_distance") .sort(["_sort_distance", "tour_purpose"]) diff --git a/processor/summarize/summaries/trip.py b/processor/summarize/summaries/trip.py index af34f0a..6866766 100644 --- a/processor/summarize/summaries/trip.py +++ b/processor/summarize/summaries/trip.py @@ -10,6 +10,7 @@ _all_purpose_rollup as _all_tour_purpose_rollup, _configured_geography_dimensions, _summary_purpose_column as _trip_purpose_column, + attach_person_travel_weight, weighted_group_sum, ) from runtime.config import Config @@ -33,10 +34,15 @@ def trip_purpose(rd: RunData, config: Config) -> pl.DataFrame: return trip_purpose.empty() base = ( - rd.trips.filter(pl.col("trip_purpose").is_not_null()) + attach_person_travel_weight( + rd, + rd.trips.filter(pl.col("trip_purpose").is_not_null()), + participant_col="num_participants", + output_col="person_trip_weight", + ) .filter(pl.col(purpose_col).is_not_null()) .group_by([purpose_col, "trip_purpose"]) - .agg(trip_count=pl.col("finalweight").sum()) + .agg(trip_count=pl.col("person_trip_weight").sum()) .rename({purpose_col: "tour_purpose"}) .with_columns( pl.col("tour_purpose").cast(pl.Utf8), @@ -78,12 +84,17 @@ def stop_purpose_by_tour_purpose(rd: RunData, config: Config) -> pl.DataFrame: return stop_purpose_by_tour_purpose.empty() return ( - rd.trips.filter(pl.col("stops") == 1) + attach_person_travel_weight( + rd, + rd.trips.filter(pl.col("stops") == 1), + participant_col="num_participants", + output_col="person_trip_weight", + ) .filter( pl.col(purpose_col).is_not_null() & pl.col("trip_purpose").is_not_null() ) .group_by([purpose_col, "trip_purpose"]) - .agg(stop_count=pl.col("finalweight").sum()) + .agg(stop_count=pl.col("person_trip_weight").sum()) .rename( { purpose_col: "tour_purpose", @@ -119,19 +130,23 @@ def trip_mode(rd: RunData, config: Config) -> pl.DataFrame: if not purpose_col: return trip_mode.empty() - base = ( + base = attach_person_travel_weight( + rd, rd.trips.filter( pl.col(purpose_col).is_not_null() & pl.col("tour_mode").is_not_null() & pl.col("trip_mode").is_not_null() - ) - .pipe( - weighted_group_sum, - [purpose_col, "tour_mode", "trip_mode"], - weight_col="finalweight", - output_col="trip_count", - ) - .rename({purpose_col: "tour_purpose"}) + ), + participant_col="num_participants", + output_col="person_trip_weight", + ).pipe( + weighted_group_sum, + [purpose_col, "tour_mode", "trip_mode"], + weight_col="person_trip_weight", + output_col="trip_count", + ) + base = ( + base.rename({purpose_col: "tour_purpose"}) .select( pl.col("tour_purpose").cast(pl.Utf8), pl.col("tour_mode").cast(pl.Utf8), diff --git a/processor/summarize/summaries/trip_distributions.py b/processor/summarize/summaries/trip_distributions.py index 916453d..aff9e2f 100644 --- a/processor/summarize/summaries/trip_distributions.py +++ b/processor/summarize/summaries/trip_distributions.py @@ -10,7 +10,12 @@ ALL_TOUR_PURPOSES, _all_purpose_rollup as _all_tour_purpose_rollup, _dense_zero_fill, + _distance_bin_expr, + _distance_bin_labels, + _distance_bin_sort_expr, + _ensure_zero_distance_bin, _summary_purpose_column as _trip_purpose_column, + attach_person_travel_weight, weighted_group_sum, ) from runtime.config import Config @@ -36,7 +41,12 @@ def trip_stop_tod(rd: RunData, config: Config) -> pl.DataFrame: if not purpose_col: return trip_stop_tod.empty() - all_trips = rd.trips.with_columns( + all_trips = attach_person_travel_weight( + rd, + rd.trips, + participant_col="num_participants", + output_col="person_trip_weight", + ).with_columns( pl.col(purpose_col).cast(pl.Utf8).alias("tour_purpose") ) if "tour_purpose" not in all_trips.columns: @@ -74,13 +84,13 @@ def trip_stop_tod(rd: RunData, config: Config) -> pl.DataFrame: stop_counts = weighted_group_sum( stop_sub, dep_col, - weight_col="finalweight", + weight_col="person_trip_weight", output_col="departure_stop_count", ) trip_counts = weighted_group_sum( trip_sub, dep_col, - weight_col="finalweight", + weight_col="person_trip_weight", output_col="departure_trip_count", ) @@ -149,30 +159,20 @@ def trip_distance(rd: RunData, config: Config) -> pl.DataFrame: if not purpose_col: return trip_distance.empty() - base = ( + base = attach_person_travel_weight( + rd, rd.trips.filter( pl.col(purpose_col).is_not_null() & pl.col("od_dist").is_not_null() - ) - .with_columns( - pl.col(purpose_col).cast(pl.Utf8).alias("tour_purpose"), - ( - pl.col("finalweight") - * pl.coalesce( - [pl.col("num_participants").cast(pl.Float64), pl.lit(1.0)] - ) - ).alias("adjusted_weight"), - pl.col("od_dist").cast(pl.Float64).round(0).alias("distance_miles_rounded"), - ) - .with_columns( - pl.when(pl.col("distance_miles_rounded") >= 40) - .then(pl.lit("40+")) - .otherwise( - pl.col("distance_miles_rounded") - .cast(pl.Int64, strict=False) - .cast(pl.Utf8) - ) - .alias("distance_bin") - ) + ), + participant_col="num_participants", + output_col="adjusted_weight", + ).with_columns( + pl.col(purpose_col).cast(pl.Utf8).alias("tour_purpose") + ) + base = ( + base + .with_columns(_distance_bin_expr("od_dist")) + .filter(pl.col("distance_bin").is_not_null()) ) by_purpose = weighted_group_sum( @@ -188,15 +188,16 @@ def trip_distance(rd: RunData, config: Config) -> pl.DataFrame: ) return ( - pl.concat([by_purpose, all_purposes], how="vertical") + _ensure_zero_distance_bin( + pl.concat([by_purpose, all_purposes], how="vertical"), + group_cols=["tour_purpose"], + value_col="trip_count", + ) .with_columns( pl.col("distance_bin").cast(pl.Utf8), pl.col("tour_purpose").cast(pl.Utf8), pl.col("trip_count").cast(pl.Float64), - pl.when(pl.col("distance_bin") == "40+") - .then(999) - .otherwise(pl.col("distance_bin").cast(pl.Int64, strict=False)) - .alias("_sort_distance"), + _distance_bin_sort_expr().alias("_sort_distance"), ) .select("distance_bin", "tour_purpose", "trip_count", "_sort_distance") .sort(["_sort_distance", "tour_purpose"]) @@ -207,7 +208,7 @@ def trip_distance(rd: RunData, config: Config) -> pl.DataFrame: @summary( id="stop_out_of_direction_distance_by_tour_purpose", schema={ - "distance_bin": pl.Int32, + "distance_bin": pl.Utf8, "tour_purpose": pl.Utf8, "stop_count": pl.Float64, }, @@ -216,7 +217,7 @@ def trip_distance(rd: RunData, config: Config) -> pl.DataFrame: }, ) def stop_ood_distance(rd: RunData, config: Config) -> pl.DataFrame: - """Out-of-direction distance for stops, in 41 bins (0-40 miles).""" + """Out-of-direction distance for stops, with exact-zero and 40+ bins.""" if "stops" not in rd.trips.columns: return stop_ood_distance.empty() @@ -229,27 +230,32 @@ def stop_ood_distance(rd: RunData, config: Config) -> pl.DataFrame: return stop_ood_distance.empty() stops2 = ( - stops.filter(pl.col("out_dir_dist").is_not_null()) + attach_person_travel_weight( + rd, + stops.filter(pl.col("out_dir_dist").is_not_null()), + participant_col="num_participants", + output_col="person_trip_weight", + ) .with_columns( pl.col(purpose_col).cast(pl.Utf8).alias("tour_purpose"), - pl.col("out_dir_dist").clip(0, 999).alias("ood"), + _distance_bin_expr("out_dir_dist"), ) - .with_columns(pl.col("ood").cast(pl.Int32).clip(0, 40).alias("distance_bin")) + .filter(pl.col("distance_bin").is_not_null()) ) if "tour_purpose" not in stops2.columns or stops2.is_empty(): return stop_ood_distance.empty() bins_df = pl.DataFrame( - {"distance_bin": list(range(0, 41))}, schema={"distance_bin": pl.Int32} + {"distance_bin": _distance_bin_labels()}, schema={"distance_bin": pl.Utf8} ) by_purpose = weighted_group_sum( stops2.filter(pl.col("tour_purpose").is_not_null()), ["tour_purpose", "distance_bin"], - weight_col="finalweight", + weight_col="person_trip_weight", output_col="stop_count", ).select( - pl.col("distance_bin").cast(pl.Int32), + pl.col("distance_bin").cast(pl.Utf8), pl.col("tour_purpose").cast(pl.Utf8), pl.col("stop_count").cast(pl.Float64), ) @@ -267,7 +273,7 @@ def stop_ood_distance(rd: RunData, config: Config) -> pl.DataFrame: join_cols=["distance_bin", "tour_purpose"], value_col="stop_count", ).select( - pl.col("distance_bin").cast(pl.Int32), + pl.col("distance_bin").cast(pl.Utf8), pl.col("tour_purpose").cast(pl.Utf8), pl.col("stop_count").cast(pl.Float64), ) @@ -276,12 +282,12 @@ def stop_ood_distance(rd: RunData, config: Config) -> pl.DataFrame: weighted_group_sum( stops2, "distance_bin", - weight_col="finalweight", + weight_col="person_trip_weight", output_col="stop_count", ) .with_columns(pl.lit(ALL_TOUR_PURPOSES).alias("tour_purpose")) .select( - pl.col("distance_bin").cast(pl.Int32), + pl.col("distance_bin").cast(pl.Utf8), pl.col("tour_purpose").cast(pl.Utf8), pl.col("stop_count").cast(pl.Float64), ) @@ -294,8 +300,9 @@ def stop_ood_distance(rd: RunData, config: Config) -> pl.DataFrame: return ( pl.concat([dense_by_purpose, dense_total], how="vertical") + .with_columns(_distance_bin_sort_expr().alias("_sort_distance")) + .sort(["tour_purpose", "_sort_distance"]) .select("distance_bin", "tour_purpose", "stop_count") - .sort(["tour_purpose", "distance_bin"]) ) diff --git a/processor/summarize/summaries/validation.py b/processor/summarize/summaries/validation.py index a035ac6..da5431c 100644 --- a/processor/summarize/summaries/validation.py +++ b/processor/summarize/summaries/validation.py @@ -17,6 +17,29 @@ ALL_AUTO_MODES = "All Auto" DAILY_TIME_PERIOD = "Daily" NON_MOTORIZED_MODES = {"WALK", "BIKE", "EBIKE"} +AUTO_OCCUPANCY_ONE_MODES = { + "AUTO SOV", + "DRIVEALONE", + "DRIVE_ALONE", + "PNR_TRANSIT", + "SOV", + "TAXI", + "TNC_SINGLE", +} +AUTO_OCCUPANCY_TWO_MODES = { + "AUTO 2 PERSON", + "HOV2", + "KNR_TRANSIT", + "SHARED2", + "SHARED_2", +} +AUTO_OCCUPANCY_THREE_PLUS_MODES = { + "AUTO 3+ PERSON", + "HOV3", + "SHARED3", + "SHARED_3", +} +INVALID_PARTICIPANT_CODE = 995.0 # TODO: Update with actual fields from Visum outputs/traffic count inputs @@ -121,6 +144,7 @@ def traffic_count_comparisons(rd: RunData, config: Config) -> pl.DataFrame: "screenline_id": pl.Utf8, "direction": pl.Utf8, "count_period": pl.Utf8, + "facility_type": pl.Utf8, "observed_volume": pl.Float64, "modeled_volume": pl.Float64, }, @@ -130,6 +154,7 @@ def screenline_flow_comparisons(rd: RunData, config: Config) -> pl.DataFrame: "screenline_id": pl.Utf8, "direction": pl.Utf8, "count_period": pl.Utf8, + "facility_type": pl.Utf8, "observed_volume": pl.Float64, "modeled_volume": pl.Float64, } @@ -146,39 +171,48 @@ def screenline_flow_comparisons(rd: RunData, config: Config) -> pl.DataFrame: ) or not required.issubset(set(rd.visum_screenline_flows.columns)): return pl.DataFrame(schema=result_schema) - observed = ( - rd.observed_screenline_flows.filter( - pl.col("screenline_id").is_not_null() - & pl.col("direction").is_not_null() - & pl.col("count_period").is_not_null() - & pl.col("volume").is_not_null() + def normalize(source: pl.DataFrame, value_column: str) -> pl.DataFrame: + facility_output = f"_{value_column}_facility_type" + facility_column = next( + ( + column + for column in ("facility_type", "FACTYPE") + if column in source.columns + ), + None, ) - .group_by(["screenline_id", "direction", "count_period"]) - .agg(observed_volume=pl.col("volume").sum()) - .with_columns( - pl.col("screenline_id").cast(pl.Utf8), - pl.col("direction").cast(pl.Utf8), - pl.col("count_period").cast(pl.Utf8), - pl.col("observed_volume").cast(pl.Float64), + return ( + source.with_columns( + ( + pl.col(facility_column).cast(pl.Utf8) + if facility_column is not None + else pl.lit(None, dtype=pl.Utf8) + ).alias("facility_type") + ) + .filter( + pl.col("screenline_id").is_not_null() + & pl.col("direction").is_not_null() + & pl.col("count_period").is_not_null() + & pl.col("volume").is_not_null() + ) + .group_by(["screenline_id", "direction", "count_period"]) + .agg( + pl.col("volume").sum().cast(pl.Float64).alias(value_column), + pl.col("facility_type") + .drop_nulls() + .first() + .alias(facility_output), + ) + .with_columns( + pl.col("screenline_id").cast(pl.Utf8), + pl.col("direction").cast(pl.Utf8), + pl.col("count_period").cast(pl.Utf8), + pl.col(facility_output).cast(pl.Utf8), + ) ) - ) - modeled = ( - rd.visum_screenline_flows.filter( - pl.col("screenline_id").is_not_null() - & pl.col("direction").is_not_null() - & pl.col("count_period").is_not_null() - & pl.col("volume").is_not_null() - ) - .group_by(["screenline_id", "direction", "count_period"]) - .agg(modeled_volume=pl.col("volume").sum()) - .with_columns( - pl.col("screenline_id").cast(pl.Utf8), - pl.col("direction").cast(pl.Utf8), - pl.col("count_period").cast(pl.Utf8), - pl.col("modeled_volume").cast(pl.Float64), - ) - ) + observed = normalize(rd.observed_screenline_flows, "observed_volume") + modeled = normalize(rd.visum_screenline_flows, "modeled_volume") return ( observed.join( @@ -186,14 +220,24 @@ def screenline_flow_comparisons(rd: RunData, config: Config) -> pl.DataFrame: on=["screenline_id", "direction", "count_period"], how="inner", ) + .with_columns( + pl.coalesce( + [ + pl.col("_modeled_volume_facility_type"), + pl.col("_observed_volume_facility_type"), + pl.lit("All"), + ] + ).alias("facility_type") + ) .select( "screenline_id", "direction", "count_period", + "facility_type", "observed_volume", "modeled_volume", ) - .sort(["screenline_id", "direction", "count_period"]) + .sort(["screenline_id", "direction", "count_period", "facility_type"]) ) @@ -393,105 +437,154 @@ def _first_existing(columns: list[str], candidates: list[str]) -> str | None: schema={ "auto_vmt": pl.Float64, }, - required_columns={"trips": ("trip_mode", "od_dist", "finalweight")}, + required_columns={"trips": ("trip_mode", "finalweight")}, ) def auto_vmt_totals(rd: RunData, config: Config) -> pl.DataFrame: """Returns single-row DataFrame with column: auto_vmt.""" - trips_df = rd.trips + distance_selection = _auto_vmt_base(rd, config) + auto_vmt = ( + distance_selection[0]["auto_vmt"].sum() + if distance_selection is not None + else 0.0 + ) + + return pl.DataFrame( + [{"auto_vmt": float(auto_vmt) if auto_vmt else 0.0}], + schema={"auto_vmt": pl.Float64}, + ) - if ( - "trip_mode" not in trips_df.columns - or "od_dist" not in trips_df.columns - or "finalweight" not in trips_df.columns - ): - return pl.DataFrame([{"auto_vmt": 0.0}], schema={"auto_vmt": pl.Float64}) +def _auto_mode_filter( + config: Config | None, + *, + include_drive_transit: bool, +) -> pl.Expr: auto_modes: list[str] | None = None if config is not None and config.mode_groups and "Auto" in config.mode_groups: auto_modes = config.mode_groups["Auto"] + mode = pl.col("trip_mode").cast(pl.Utf8).str.to_uppercase() if auto_modes is not None: - auto_filter = pl.col("trip_mode").cast(pl.Utf8).is_in(auto_modes) + configured_modes = [str(value).upper() for value in auto_modes] + eligible = mode.is_in(configured_modes) else: - auto_filter = ( - pl.col("trip_mode") - .cast(pl.Utf8) - .str.to_uppercase() - .str.contains("DRIVE|SHARED|SOV|HOV|AUTO") - ) + eligible = mode.str.contains("DRIVE|SHARED|SOV|HOV|AUTO|TAXI|TNC") - auto_trips = trips_df.filter(auto_filter) + if include_drive_transit: + eligible = eligible | mode.str.contains("PNR|KNR") + else: + eligible = eligible & ~mode.str.contains("PNR|KNR") + return eligible - occupancy_expr = ( - pl.col("num_participants").fill_null(1).clip(lower_bound=1) - if "num_participants" in auto_trips.columns - else pl.lit(1) - ) - auto_vmt = auto_trips.with_columns( - (pl.col("od_dist") * pl.col("finalweight") / occupancy_expr).alias("auto_vmt_w") - )["auto_vmt_w"].sum() +def _auto_occupancy_expr(columns: list[str]) -> pl.Expr: + mode = pl.col("trip_mode").cast(pl.Utf8).str.to_uppercase() + if "num_participants" in columns: + participants = pl.col("num_participants").cast(pl.Float64, strict=False) + fallback_occupancy = ( + pl.when( + participants.is_not_null() + & participants.is_finite() + & (participants > 0) + & (participants < INVALID_PARTICIPANT_CODE) + ) + .then(participants) + .otherwise(1.0) + ) + else: + fallback_occupancy = pl.lit(1.0) - return pl.DataFrame( - [{"auto_vmt": float(auto_vmt) if auto_vmt else 0.0}], - schema={"auto_vmt": pl.Float64}, + joint_record = ( + pl.col("tour_category") + .cast(pl.Utf8) + .str.to_lowercase() + .eq("joint") + .fill_null(False) + if "tour_category" in columns + else pl.lit(False) + ) + person_level_joint = ( + joint_record + & pl.col("joint_trip_id").is_not_null() + & (pl.len().over(["tour_category", "joint_trip_id"]) > 1) + if "joint_trip_id" in columns and "tour_category" in columns + else pl.lit(False) + ) + household_joint = joint_record & ~person_level_joint + chauffeur_escort_event = ( + pl.col("escort_event_role").is_not_null() + if "escort_event_role" in columns + else pl.lit(False) ) - -def _auto_mode_filter(config: Config | None) -> pl.Expr: - auto_modes: list[str] | None = None - if config is not None and config.mode_groups and "Auto" in config.mode_groups: - auto_modes = config.mode_groups["Auto"] - - if auto_modes is not None: - return pl.col("trip_mode").cast(pl.Utf8).is_in(auto_modes) return ( - pl.col("trip_mode") - .cast(pl.Utf8) - .str.to_uppercase() - .str.contains("DRIVE|SHARED|SOV|HOV|AUTO|TAXI|TNC") + pl.when(household_joint | chauffeur_escort_event) + .then(1.0) + .when(mode.is_in(sorted(AUTO_OCCUPANCY_ONE_MODES))) + .then(1.0) + .when(mode.is_in(sorted(AUTO_OCCUPANCY_TWO_MODES))) + .then(2.0) + .when(mode.is_in(sorted(AUTO_OCCUPANCY_THREE_PLUS_MODES))) + .then(3.33) + .otherwise(fallback_occupancy) ) -def _distance_base( +def _auto_vmt_base( rd: RunData, config: Config | None, ) -> tuple[pl.DataFrame, str] | None: trips = rd.trips - if "finalweight" not in trips.columns: + if "finalweight" not in trips.columns or "trip_mode" not in trips.columns: return None if ( "skim_auto_distance" in trips.columns - and trips["skim_auto_distance"].is_not_null().any() + and trips.filter( + _auto_mode_filter(config, include_drive_transit=True) + & pl.col("skim_auto_distance").is_not_null() + ).height + > 0 ): LOGGER.info( - "[vmt_by_segment] Run %r using distance_source=skim_auto_distance", + "[auto_vmt] Run %r using distance_source=skim_auto_distance", rd.label, ) - return ( - trips.filter(pl.col("skim_auto_distance").is_not_null()).with_columns( - pl.col("skim_auto_distance").cast(pl.Float64).alias("_vmt_distance") - ), - "skim_auto_distance", + base = trips.filter( + _auto_mode_filter(config, include_drive_transit=True) + & pl.col("skim_auto_distance").is_not_null() + ).with_columns( + pl.col("skim_auto_distance").cast(pl.Float64).alias("_vmt_distance") ) - - if "od_dist" not in trips.columns or "trip_mode" not in trips.columns: + distance_source = "skim_auto_distance" + elif "od_dist" in trips.columns: + LOGGER.info( + "[auto_vmt] Run %r using distance_source=od_dist", + rd.label, + ) + base = trips.filter( + _auto_mode_filter(config, include_drive_transit=False) + & pl.col("od_dist").is_not_null() + ).with_columns(pl.col("od_dist").cast(pl.Float64).alias("_vmt_distance")) + distance_source = "od_dist" + else: LOGGER.warning( - "[vmt_by_segment] Run %r has no usable auto distance source.", + "[auto_vmt] Run %r has no usable auto distance source.", rd.label, ) return None - LOGGER.info( - "[vmt_by_segment] Run %r using distance_source=od_dist", - rd.label, - ) return ( - trips.filter( - _auto_mode_filter(config) & pl.col("od_dist").is_not_null() - ).with_columns(pl.col("od_dist").cast(pl.Float64).alias("_vmt_distance")), - "od_dist", + base.with_columns(_auto_occupancy_expr(base.columns).alias("_occupancy")) + .with_columns( + ( + pl.col("_vmt_distance") + * pl.col("finalweight").cast(pl.Float64) + / pl.col("_occupancy") + ).alias("auto_vmt") + ) + .filter(pl.col("auto_vmt").is_not_null()), + distance_source, ) @@ -615,8 +708,7 @@ def _with_derived_daily_vmt_rows( groups_with_period_rows, on=group_cols, how="anti" ) derived_daily_rows = ( - df.join(groups_with_period_rows, on=group_cols, how="inner") - .group_by(group_cols) + non_daily.group_by(group_cols) .agg( pl.col(value_col).sum().alias(value_col), pl.col("trip_count").sum().alias("trip_count"), @@ -645,13 +737,13 @@ def _with_derived_daily_vmt_rows( "distance_source": pl.Utf8, "time_period_source": pl.Utf8, }, - required_columns={"trips": ("finalweight",)}, + required_columns={"trips": ("trip_mode", "finalweight")}, ) def auto_vmt_by_home_geography_income_hhsize_time_period( rd: RunData, config: Config, ) -> pl.DataFrame: - distance_selection = _distance_base(rd, config) + distance_selection = _auto_vmt_base(rd, config) if distance_selection is None: return auto_vmt_by_home_geography_income_hhsize_time_period.empty() @@ -698,12 +790,6 @@ def auto_vmt_by_home_geography_income_hhsize_time_period( if "hhsize" in base.columns else pl.lit(ALL_HOUSEHOLD_SIZES) ) - occupancy_expr = ( - pl.col("num_participants").fill_null(1).clip(lower_bound=1) - if "num_participants" in base.columns - else pl.lit(1) - ) - base = ( base.with_columns(*income_exprs) .with_columns( @@ -719,13 +805,7 @@ def auto_vmt_by_home_geography_income_hhsize_time_period( if "trip_mode" in base.columns else pl.lit(ALL_AUTO_MODES) ).alias("mode"), - ( - pl.col("_vmt_distance") - * pl.col("finalweight").cast(pl.Float64) - / occupancy_expr - ).alias("auto_vmt"), ) - .filter(pl.col("auto_vmt").is_not_null()) ) geography_dimensions: list[tuple[str, str | None]] = [ diff --git a/processor/summarize/summaries/validation_scaffolds.py b/processor/summarize/summaries/validation_scaffolds.py index 2015cc1..757399e 100644 --- a/processor/summarize/summaries/validation_scaffolds.py +++ b/processor/summarize/summaries/validation_scaffolds.py @@ -101,7 +101,7 @@ def count_location_fit_validation_summary(rd: RunData, config: Config) -> pl.Dat @summary( - id="county_flows_validation_summary", + id="district_commuting_flows_validation_summary", build_by_default=False, schema={ "": pl.Utf8, @@ -112,12 +112,14 @@ def count_location_fit_validation_summary(rd: RunData, config: Config) -> pl.Dat "Total": pl.Float64, }, ) -def county_flows_validation_summary(rd: RunData, config: Config) -> pl.DataFrame: - return county_flows_validation_summary.empty() +def district_commuting_flows_validation_summary( + rd: RunData, config: Config +) -> pl.DataFrame: + return district_commuting_flows_validation_summary.empty() @summary( - id="county_flows_joja_validation_summary", + id="county_commuting_flows_validation_summary", build_by_default=False, schema={ "": pl.Utf8, @@ -127,8 +129,10 @@ def county_flows_validation_summary(rd: RunData, config: Config) -> pl.DataFrame "Total": pl.Float64, }, ) -def county_flows_joja_validation_summary(rd: RunData, config: Config) -> pl.DataFrame: - return county_flows_joja_validation_summary.empty() +def county_commuting_flows_validation_summary( + rd: RunData, config: Config +) -> pl.DataFrame: + return county_commuting_flows_validation_summary.empty() @summary( diff --git a/processor/summarize/validation_derived.py b/processor/summarize/validation_derived.py index e83f6e3..4f60b98 100644 --- a/processor/summarize/validation_derived.py +++ b/processor/summarize/validation_derived.py @@ -149,7 +149,7 @@ def _fit_group( intercept=float(intercept), r_squared=float(r_squared), equation_label=_equation_label(slope, intercept), - r_squared_label=f"R^2 = {r_squared:.2f}", + r_squared_label=f"R² = {r_squared:.2f}", ) return base diff --git a/reference/processor-output-table-reference.md b/reference/processor-output-table-reference.md new file mode 100644 index 0000000..374ccc0 --- /dev/null +++ b/reference/processor-output-table-reference.md @@ -0,0 +1,479 @@ +# Processor Output Table Reference + +This reference documents the tabular products written or accepted by the +ActivitySim Visualizer Output Processor and explains how their fields support +travel analysis and information development. + +Prepared output tables preserve the source ActivitySim fields documented above +them on that page. Their field dictionaries therefore focus on stable canonical +and processor-derived additions. Because regional models and configurations can +add source columns, category mappings, named geographies, and skim components, +the prepared schemas are extensible rather than fixed. Summary and sidecar +schemas are exact. + +Prepared caches can use Parquet or CSV. Summary caches use CSV and may contain +weighted, unweighted, and segmented variants of the same table contract. Empty, +unavailable, and failed tables can be represented by internal cache sentinels; +the cache manifest records the state and should be consulted before analysis. + + +_Generated from the ActivitySim Visualizer processor contracts and analytical metadata._ + +## 1. Prepared Output Tables + +Prepared caches contain eight canonical tables in the configured Parquet or CSV format. Each file is named `.`. Prepare preserves the source columns described earlier in the SimOR Model Outputs documentation; the field tables below list the stable canonical and processor-derived additions. Configuration-specific category mappings and skim outputs can add further columns. + +### `households` Prepared Table + +_Processor runtime name: `hh`._ + +**Information and analytical use.** One row per household. This core prepared table preserves the ActivitySim household record and adds stable weights, household-size and vehicle-count aliases, auto-sufficiency inputs, and home-geography fields. Use it for household controls, demographic profiles, vehicle availability, and residential geography analysis. The cache file is always present, but it contains an empty sentinel when the source table is unavailable or empty. + +Retained source fields keep their source meaning and generally keep their source type. + +| Field | Type | Description | +|---|---|---| +| `HHSIZE` | `Int32` | Household person count clipped to 1 through 5, where 5 represents households of five or more people. | +| `HHVEH` | `Int32` | Household vehicle count clipped to 0 through 4 for standard summary categories. | +| `av_ownership` | `Boolean-like` | Whether the household owns an autonomous vehicle. | +| `finalweight` | `Float64` | Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback. | +| `household_id` | `Int64` | Canonical household identifier used to join people, tours, trips, days, and vehicles. | +| `home_zone_id` | `source-compatible identifier` | Household home-zone identifier, normally a MAZ when MAZ processing is enabled. | +| `WORKERS` | `Int32` | Number of workers copied from the configured household worker-count field. | +| `LICENSEDDRIVERS` | `Int32` | Licensed household members derived from person records when license data are available. | +| `ADULTS` | `Int32` | Number of adults copied from the configured household adult-count field. | +| `home_taz` | `source-compatible identifier` | TAZ corresponding to home_zone_id; copied directly in TAZ models or looked up from land use in MAZ models. | +| `HGEO` | `String` | Legacy configured home-geography label used by compatible summaries. | +| `home_geo__` | `String` | Home-geography identifier for each configured named geography aggregation. | + +### `persons` Prepared Table + +_Processor runtime name: `per`._ + +**Information and analytical use.** One row per person. This core table preserves ActivitySim person attributes and choices and supplies canonical identifiers, worker/student indicators, home/work/school geography, mandatory-distance measures, and analysis weights. Use it for population segmentation, long-term choice analysis, accessibility, and person-level rates. The cache file is always present, with an empty sentinel when the source is unavailable or empty. + +Retained source fields keep their source meaning and generally keep their source type. + +| Field | Type | Description | +|---|---|---| +| `age` | `integer` | Person age in years. | +| `bike_comfort` | `category` | Modeled bicycle-comfort alternative. | +| `distance_to_school` | `Float64` | Prepared home-to-school distance in the configured distance units. | +| `distance_to_work` | `Float64` | Prepared home-to-work distance in the configured distance units. | +| `external_workplace_zone_id` | `identifier` | Assigned external workplace zone identifier. | +| `finalweight` | `Float64` | Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | +| `free_parking_at_work` | `Boolean-like or category` | Free-workplace-parking eligibility or modeled alternative. | +| `has_license` | `Boolean-like` | Whether the person holds a driver's license. | +| `home_zone_id` | `identifier` | Prepared home-zone identifier. | +| `household_id` | `Int64` | Canonical household identifier for the person. | +| `is_external_worker` | `Boolean-like` | Whether the person's assigned workplace is external to the modeled area. | +| `is_student` | `Boolean-like` | Whether the person is classified as a student. | +| `is_worker` | `String` | Whether the person is classified as a worker. | +| `person_id` | `Int64` | Canonical person identifier. | +| `person_type` | `source-compatible category` | ActivitySim person-type code. | +| `school_zone_id` | `identifier` | Assigned school-zone identifier. | +| `student_type` | `String` | Prepared school or enrollment market segment. | +| `telecommute_frequency` | `category` | Modeled telecommute-frequency alternative. | +| `transit_pass_ownership` | `Boolean-like` | Whether the person owns a transit pass. | +| `transit_pass_subsidy` | `category` | Modeled transit-pass-subsidy alternative. | +| `work_from_home` | `String` | Whether the worker has a home workplace. | +| `workplace_zone_id` | `identifier` | Assigned internal workplace-zone identifier. | +| `is_university` | `Boolean-like source value` | Prepared university-student indicator selected from the configured source alias. | +| `adult` | `String` | Prepared adult-status value used when deriving household auto-sufficiency inputs. | +| `mandatory_tour_frequency` | `String` | Mandatory-tour-frequency choice used directly by summaries when available. | +| `imf_choice` | `integer` | Legacy numeric mandatory-tour-frequency code derived from named ActivitySim alternatives. | +| `num_joint_tours` | `Int32` | Number of unique joint tours linked to the person through the joint-participant table. | +| `home_taz` | `source-compatible identifier` | TAZ corresponding to the person's home zone. | +| `work_taz` | `source-compatible identifier` | TAZ corresponding to the person's internal workplace zone. | +| `school_taz` | `source-compatible identifier` | TAZ corresponding to the person's school zone. | +| `HGEO` | `String` | Legacy configured home-geography label. | +| `WGEO` | `String` | Legacy configured workplace-geography label. | +| `home_geo__` | `String` | Home-geography identifier for each configured named geography aggregation. | +| `work_geo__` | `String` | Workplace-geography identifier for each configured named geography aggregation. | +| `school_geo__` | `String` | School-geography identifier for each configured named geography aggregation. | + +### `day` Prepared Table + +_Processor runtime name: `day`._ + +**Information and analytical use.** Optional person-day or household-day records retained from multi-day model or survey output. Use this table for daily activity-pattern and frequency analysis where a person can contribute more than one observed day. An empty sentinel is written when no day table is available. + +Retained source fields keep their source meaning and generally keep their source type. + +| Field | Type | Description | +|---|---|---| +| `day_id` | `Int64` | Canonical day-record identifier when supplied. | +| `household_id` | `Int64` | Household identifier for the observed day when supplied. | +| `person_id` | `Int64` | Person identifier for the observed day when supplied. | +| `person_type` | `String` | Person type retained from the source or joined from the prepared person table. | +| `travel_date` | `String` | Source travel-date label when available. | +| `day_num` | `Int32` | Sequential day number within the person's or household's observation period. | +| `travel_dow` | `Int32` | Source day-of-week code when available. | +| `daily_activity_pattern` | `String` | Daily activity-pattern alternative, such as mandatory, nonmandatory, or home. | +| `day_weight` | `source-compatible number` | Configured source day weight retained for provenance when available. | +| `finalweight` | `Float64` | Day expansion weight from day, person, or household weights, with a 1.0 fallback. | + +### `tours` Prepared Table + +_Processor runtime name: `tours`._ + +**Information and analytical use.** One row per source tour representation. This core table preserves ActivitySim tour choices and adds standardized purpose, household context, stop counts, joint-party size, distance, time-period, and geography fields. Use it for tour frequency, mode, destination, scheduling, stop-pattern, joint-travel, and vehicle-allocation analysis. Some ActivitySim outputs repeat joint tours by participant; summary descriptions state when those rows are fractionally adjusted. The cache file is always present. + +Retained source fields keep their source meaning and generally keep their source type. + +| Field | Type | Description | +|---|---|---| +| `AUTOSUFF` | `Int32` | Prepared household auto-sufficiency category for the tour. | +| `NUMBER_HH` | `Int32` | Joint-party size from participant rows, number_of_participants, an existing NUMBER_HH value, or a fallback of one. | +| `SKIMDIST` | `Float64` | Prepared tour distance in the configured skim distance units. | +| `atwork_subtour_frequency` | `category` | Modeled at-work subtour-frequency alternative. | +| `destination` | `identifier` | Tour destination-zone identifier. | +| `finalweight` | `Float64` | Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | +| `household_id` | `Int64` | Canonical household identifier associated with the tour. | +| `is_external_tour` | `Boolean-like` | Whether the tour destination is external to the modeled area. | +| `num_escortees` | `Int64` | Number of students escorted on the tour. | +| `num_ib_stops` | `Int32` | Number of inbound intermediate stops parsed from stop_frequency. | +| `num_ob_stops` | `Int32` | Number of outbound intermediate stops parsed from stop_frequency. | +| `num_tot_stops` | `Int32` | Total intermediate stops on the tour. | +| `person_id` | `Int64` | Canonical person identifier associated with an individual tour. | +| `school_esc_inbound` | `String` | Inbound school-escort alternative or status. | +| `school_esc_outbound` | `String` | Outbound school-escort alternative or status. | +| `tour_category` | `String` | Tour category such as mandatory, nonmandatory, joint, or at-work. | +| `tour_id` | `Int64` | Canonical tour identifier. | +| `tour_mode` | `String` | Main mode selected for the tour. | +| `tour_purpose` | `String` | Primary purpose of the tour. | +| `vehicle_occup_1` | `vehicle-type code` | Vehicle type allocated to single-occupant auto tours. | +| `vehicle_occup_2` | `vehicle-type code` | Vehicle type allocated to two-occupant auto tours. | +| `vehicle_occup_3.5` | `vehicle-type code` | Vehicle type allocated to auto tours with three or more occupants. | +| `person_type` | `source-compatible category` | Person type copied from a configured tour-table alias when present. | +| `summary_tour_purpose` | `String` | Purpose normalized to the configured summary-purpose categories. | +| `origin` | `source-compatible identifier` | Tour origin zone; home-based and at-work origins are repaired from household or parent-tour context when possible. | +| `start_hour` | `Int32` | Tour start period, using the source period system or the survey adapter's 48-period conversion. | +| `end_hour` | `Int32` | Tour end period in the same period system as start_hour. | +| `start_period` | `String` | Configured named network period corresponding to the tour start period. | +| `end_period` | `String` | Configured named network period corresponding to the tour end period. | +| `first_inbound_trip_depart` | `Int32` | Departure period of the first inbound trip on the tour when it can be derived. | +| `first_inbound_trip_period` | `String` | Named network period of the first inbound trip. | +| `tourdur` | `Int32` | Inclusive tour duration, end_hour minus start_hour plus one, clipped to 1 through 48 periods. | +| `stop_frequency` | `source-compatible category` | ActivitySim stop-frequency choice retained from the configured source field. | +| `composition` | `String` | Joint-party composition retained from the source or derived as adults, children, or mixed from participant attributes. | +| `income_segment` | `Int64` | Household income segment copied to the tour when available. | +| `o_maz` | `Int64` | Tour-origin MAZ copied from origin when MAZ processing is enabled. | +| `d_maz` | `Int64` | Tour-destination MAZ copied from destination when MAZ processing is enabled. | +| `OTAZ` | `Int32` | Origin TAZ copied or looked up from the origin MAZ. | +| `DTAZ` | `Int32` | Destination TAZ copied or looked up from the destination MAZ. | +| `pnr_zone_id` | `Int64` | Park-and-ride lot or zone selected for a drive-transit tour when supplied. | +| `pnr_taz` | `Int32` | TAZ corresponding to pnr_zone_id. | +| `origin_geo__` | `String` | Origin-geography identifier for each configured named geography aggregation. | +| `destination_geo__` | `String` | Destination-geography identifier for each configured named geography aggregation. | +| `tour_distance` | `Float64` | Supplied tour distance with missing values filled from SKIMDIST when both fields are available. | +| `vot_bin` | `String` | Configured value-of-time category used to select segmented skims. | +| `` | `normally Float64` | Skimjoin result whose name, units, direction, and mode applicability are defined by the normalized skimjoin configuration. | + +### `trips` Prepared Table + +_Processor runtime name: `trips`._ + +**Information and analytical use.** One row per source trip representation. This core table preserves ActivitySim trip choices and adds parent-tour context, canonical endpoints, distances, periods, stop indicators, escort-event positions, geography, and weights. Use it for trip and stop rates, mode and purpose analysis, parking demand, time-of-day profiles, VMT, and travel-impedance analysis. Joint records may represent several travelers; summary descriptions state how participant expansion is applied. The cache file is always present. + +Retained source fields keep their source meaning and generally keep their source type. + +| Field | Type | Description | +|---|---|---| +| `escort_event_role` | `String` | Adult escort-event role assigned to the trip. | +| `escort_stops_after_event` | `Int32` | Stops occurring after the associated escort event. | +| `escort_stops_before_event` | `Int32` | Stops occurring before the associated escort event. | +| `finalweight` | `Float64` | Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | +| `num_participants` | `Int32` | Number of travelers represented by a joint trip record. | +| `od_dist` | `Float64` | Origin-to-destination trip distance from the configured distance skim or supported survey distance, in source units. | +| `out_dir_dist` | `Float64` | Additional distance caused by visiting the stop rather than traveling directly to the half-tour endpoint. | +| `parking_zone` | `Int64` | Zone where the auto trip parks. | +| `person_id` | `Int64` | Canonical person identifier associated with the trip. | +| `stops` | `Int32` | Intermediate-stop indicator, 1 before the last trip on a half-tour and 0 for the final leg. | +| `tour_id` | `Int64` | Canonical identifier of the parent tour. | +| `tour_mode` | `String` | Main mode of the parent tour. | +| `tour_purpose` | `String` | Primary purpose of the parent tour. | +| `trip_mode` | `String` | Mode used for the trip leg. | +| `trip_purpose` | `String` | Destination purpose of the trip leg. | +| `household_id` | `Int64` | Canonical household identifier associated with the trip. | +| `trip_id` | `Int64` | Canonical trip identifier. | +| `summary_tour_purpose` | `String` | Parent-tour purpose normalized to configured summary-purpose categories. | +| `tour_category` | `String` | Category of the parent tour, filled from the prepared tour table when necessary. | +| `origin` | `source-compatible identifier` | Trip origin zone; tour-boundary endpoints are repaired from the prepared tour origin when possible. | +| `destination` | `source-compatible identifier` | Trip destination zone; tour-boundary endpoints are repaired from the prepared tour origin when possible. | +| `outbound` | `Boolean-like source value` | Direction flag indicating the outbound half of the tour. | +| `inbound` | `Int32` | Derived inverse direction indicator, 1 for inbound records and 0 otherwise. | +| `trip_num` | `Int32` | Sequential trip number within a tour direction. | +| `max_trip_num` | `integer` | Largest trip number on the same tour half, used to identify intermediate stops. | +| `depart_hour` | `Int32` | Trip departure period; defaults to one only when no departure field is available. | +| `trip_period` | `String` | Configured named network period corresponding to the trip departure period. | +| `income_segment` | `Int64` | Household income segment copied to the trip when available. | +| `AUTOSUFF` | `Int32` | Parent-household auto-sufficiency category under the configured comparison basis. | +| `o_maz` | `Int64` | Trip-origin MAZ copied from origin when MAZ processing is enabled. | +| `d_maz` | `Int64` | Trip-destination MAZ copied from destination when MAZ processing is enabled. | +| `OTAZ` | `Int32` | Origin TAZ copied or looked up from the origin MAZ. | +| `DTAZ` | `Int32` | Destination TAZ copied or looked up from the destination MAZ. | +| `pnr_zone_id` | `Int64` | Park-and-ride zone inherited from the parent tour when supplied. | +| `pnr_taz` | `Int32` | TAZ corresponding to pnr_zone_id. | +| `origin_parking_zone` | `Int64` | Parking zone associated with the trip origin when supplied. | +| `origin_geo__` | `String` | Origin-geography identifier for each configured named geography aggregation. | +| `destination_geo__` | `String` | Destination-geography identifier for each configured named geography aggregation. | +| `prepared_non_motorized_distance` | `Float64` | Optional walk/bicycle/e-bike distance from the configured non-motorized skim source. | +| `escort_event_trip_num` | `Int32` | Trip number at which the matched escort event occurs. | +| `escort_event_match_status` | `String` | Whether escort-event matching was matched, ambiguous, or unmatched. | +| `vot_bin` | `String` | Configured value-of-time category used to select segmented skims. | +| `` | `normally Float64` | Skimjoin result whose name, units, lookup direction, and mode applicability are defined by the normalized skimjoin configuration. | + +### `vehicles` Prepared Table + +_Processor runtime name: `vehicles`._ + +**Information and analytical use.** Optional one-row-per-household-vehicle table. It preserves ActivitySim vehicle records and decodes the standard compound vehicle type into body, age, and fuel attributes. Use it for fleet composition, vehicle technology, energy, emissions, and vehicle-allocation analysis. An empty sentinel is written when no vehicle table is available. + +Retained source fields keep their source meaning and generally keep their source type. + +| Field | Type | Description | +|---|---|---| +| `body_type` | `String` | Vehicle body-style category parsed from vehicle_type. | +| `finalweight` | `Float64` | Vehicle expansion weight inherited from the owning household or set to 1.0 when unavailable. | +| `fuel_type` | `String` | Vehicle fuel or powertrain category parsed from vehicle_type. | +| `vehicle_age` | `Int64` | Vehicle age in years parsed from vehicle_type. | +| `household_id` | `Int64` | Canonical identifier of the household that owns the vehicle. | +| `vehicle_id` | `Int64` | Canonical vehicle identifier. | +| `vehicle_num` | `Int32` | Sequential vehicle number within the household. | +| `vehicle_type` | `String` | Source vehicle-type choice, commonly encoded as body type, age, and fuel type separated by underscores. | + +### `joint_tour_participants` Prepared Table + +_Processor runtime name: `joint_participants`._ + +**Information and analytical use.** Optional participation records linking people to fully joint tours. The usual grain is one person-tour participation, although additional source attributes are retained. Use it to determine joint-tour party size, composition, and person participation without treating a household tour as independent person tours. An empty sentinel is written when the source table is unavailable. + +Retained source fields keep their source meaning and generally keep their source type. + +| Field | Type | Description | +|---|---|---| +| `person_id` | `Int64` | Canonical identifier of a person participating in a joint tour. | +| `tour_id` | `Int64` | Canonical identifier of the joint tour. | + +### `land_use` Prepared Table + +_Processor runtime name: `land_use`._ + +**Information and analytical use.** Optional zone and land-use attribute table, normally one row per source MAZ or TAZ before any enrollment overlays are added. It supports zone translation, named geography joins, employment and enrollment controls, parking capacity, and destination-based analysis. When configured student markets are expanded, additional zone/student-type rows carry enrollment counts. An empty sentinel is written when land use is unavailable. + +Retained source fields keep their source meaning and generally keep their source type. + +| Field | Type | Description | +|---|---|---| +| `MAZ` | `Int64` | Canonical micro-analysis-zone identifier. | +| `employment_count` | `Float64` | Employment opportunities associated with the zone. | +| `enrollment_count` | `Float64` | Enrollment opportunities associated with the zone and student type. | +| `student_type` | `String` | Student or enrollment market segment. | +| `TAZ` | `Int64` | Canonical transportation-analysis-zone identifier. | +| `EMPLOYMENT` | `Float64` | Canonical total employment copied from the configured land-use employment field. | +| `land_use_geo__` | `String` | Geography identifier for each configured named land-use aggregation. | + + +## 2. Summary Output Tables + +The processor calculates **87** summary contracts: **85** in the standard workflow and **2** optional skim ECDF tables. The same schemas apply to weighted, unweighted, and segmented cache variants. + +Every summary table is saved as a CSV file named `.csv`. + +### Population and Demographics + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `household_size_distribution` | Household totals by modeled household size. Use it to profile household composition, calculate size shares, and compare population-synthesis results across runs. | `household_size` (`Int64`): number of people in the household.
    `household_count` (`Float64`): weighted households in that size category. | +| `person_type_distribution` | Person totals by ActivitySim person type, with a display label. Use it to compare demographic market segments and as a denominator for person-type travel rates. | `person_type` (`String`): stable person-type code.
    `person_type_label` (`String`): configured readable label for the code.
    `person_count` (`Float64`): weighted people of that type. | +| `population_totals` | One run-level control-total row for people, households, person-tours, person-trips, and person-stops. Joint tour and trip records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback; sentinel, missing, or nonpositive fallback counts resolve to one. Use it for reasonableness checks and top-level comparisons; the measures use their own table weights and are not additive to one another. | `person_count` (`Float64`): weighted persons.
    `household_count` (`Float64`): weighted households.
    `tour_count` (`Float64`): weighted person-tours; joint records are participant-expanded.
    `trip_count` (`Float64`): weighted person-trips; joint records are participant-expanded.
    `stop_count` (`Float64`): weighted person-trip records flagged as intermediate stops; joint records are participant-expanded. | + +### Person Attributes and Long-Term Choices + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `license_holding_status_distribution` | Licensed and unlicensed people age 16 or older by person type, including an all-person-types rollup. Use it to assess access to driving and explain auto-mode availability. | `person_type` (`String`): person-type code or `all_person_types` rollup.
    `license_holding_status` (`String`): `has_license` or `no_license`.
    `person_type_label` (`String`): configured readable person-type label.
    `person_count` (`Float64`): weighted people in the group. | +| `bicycle_comfort_level_distribution` | Bicycle comfort categories by person type, including an all-person-types rollup. Use it to understand the population assumed willing to use different bicycle facilities. | `person_type` (`String`): person-type code or rollup.
    `bicycle_comfort_level` (`String`): prepared bicycle-comfort category.
    `person_type_label` (`String`): configured readable person-type label.
    `person_count` (`Float64`): weighted people in the group. | +| `transit_pass_ownership_by_person_type` | Transit-pass ownership status by person type, including an all-person-types rollup. Use it to evaluate transit market eligibility and pass-ownership model results. | `person_type` (`String`): person-type code or rollup.
    `transit_pass_ownership_status` (`String`): `has_transit_pass` or `no_transit_pass`.
    `person_type_label` (`String`): configured readable person-type label.
    `person_count` (`Float64`): weighted people in the group. | +| `transit_subsidy_by_person_type` | Transit-pass subsidy alternatives for workers, by person type and with an all-person-types rollup. Use it to examine employer or institutional transit-benefit assumptions. | `person_type` (`String`): person-type code or rollup.
    `transit_subsidy_status` (`String`): prepared subsidy alternative code.
    `transit_subsidy_label` (`String`): configured readable subsidy label.
    `person_type_label` (`String`): configured readable person-type label.
    `person_count` (`Float64`): weighted eligible workers in the group. | +| `telecommute_frequency_distribution` | Non-work-from-home workers by telecommute-frequency category and home geography, plus an all-geographies rollup. Use it to analyze recurring telecommuting among workers who still have an external workplace. | `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `telecommute_frequency` (`String`): prepared telecommute-frequency alternative.
    `person_count` (`Float64`): weighted workers in the group. | + +### Household Vehicles and Vehicle Characteristics + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `autonomous_vehicle_ownership_totals` | One run-level total of households modeled as owning an autonomous vehicle. Use it to report scenario penetration and compare AV-ownership assumptions. | `household_with_autonomous_vehicle_count` (`Float64`): weighted households with `av_ownership` true. | +| `auto_ownership_distribution` | Household totals jointly classified by household size and vehicle count; household sizes of five or more are grouped as `5+`. Use it to assess motorization and auto sufficiency. | `household_size` (`String`): household-size category, with `5+` as the terminal group.
    `household_vehicle_count` (`Int64`): vehicles available to the household.
    `household_count` (`Float64`): weighted households in the joint category. | +| `vehicle_age_distribution` | Household vehicles by age, with ages 20 and older grouped as `20+`. Use it for fleet turnover, emissions, and technology analyses. | `age` (`String`): vehicle age in years or `20+`.
    `vehicle_count` (`Float64`): weighted vehicles in the age category. | +| `vehicle_fuel_type_distribution` | Household vehicles by prepared fuel or powertrain type. Use it for fleet composition, energy, and emissions analysis. | `fuel_type` (`String`): prepared vehicle fuel/powertrain category.
    `vehicle_count` (`Float64`): weighted vehicles of that type. | +| `vehicle_body_type_distribution` | Household vehicles by prepared body type. Use it to characterize the light-duty fleet and support occupancy or emissions comparisons. | `body_type` (`String`): prepared vehicle body-style category.
    `vehicle_count` (`Float64`): weighted vehicles of that type. | + +### Long-Term Geography, Location, and Shadow Pricing + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `work_from_home_rate_by_geography` | All workers and work-from-home workers by home geography, plus a regional rollup. Divide the WFH count by the worker count to calculate the WFH rate and map its spatial pattern. | `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `worker_count` (`Float64`): weighted workers living in the geography.
    `work_from_home_worker_count` (`Float64`): weighted workers flagged as working from home. | +| `internal_external_worker_by_geography` | Internal and external workers by home geography, plus a regional rollup. Use it to understand external-worker incidence and its residential distribution. | `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `internal_worker_count` (`Float64`): weighted workers with an internal workplace.
    `external_worker_count` (`Float64`): weighted workers classified as external. | +| `external_worker_workplace_locations` | External workers by their external workplace geography, plus a regional rollup. Use it to analyze external commute orientation; the all-worker denominator is repeated to support share calculations. | `geography_type` (`String`): external-workplace geography system or rollup.
    `geography_id` (`String`): external-workplace geography identifier or rollup value.
    `external_worker_count` (`Float64`): weighted external workers assigned to that destination.
    `all_worker_count` (`Float64`): weighted workers in the full run, repeated on every row. | +| `workplace_location_employment_comparison` | Land-use employment and modeled worker workplace choices aligned by workplace geography. Use it to compare attraction targets with assigned workers and diagnose location-choice balance. | `geography_type` (`String`): workplace-geography system.
    `geography_id` (`String`): workplace-geography identifier.
    `employment_count` (`Float64`): employment opportunities from land use.
    `worker_count` (`Float64`): weighted workers assigned to the geography. | +| `commuting_flows` | Worker flows from home geography to workplace geography at matching configured geography levels, plus a regional total. Use it as an origin-destination matrix for commute sheds, self-containment, and interjurisdictional flows. | `origin_geography_type` (`String`): home-geography system.
    `origin_geography_id` (`String`): home-geography identifier.
    `destination_geography_type` (`String`): workplace-geography system.
    `destination_geography_id` (`String`): workplace-geography identifier.
    `commuter_count` (`Float64`): weighted workers in the OD pair. | +| `school_location_enrollment_comparison` | Land-use enrollment and modeled student school locations aligned by geography and student type. Use it to compare school-location targets with assigned students. | `geography_type` (`String`): school-geography system.
    `geography_id` (`String`): school-geography identifier.
    `student_type` (`String`): prepared school/enrollment market segment.
    `enrollment_count` (`Float64`): target enrollment from land use.
    `student_count` (`Float64`): weighted students assigned to the geography and type. | +| `workplace_shadow_pricing_residuals` | Zone-level workplace target-versus-modeled residuals. Use positive residuals to find over-assigned workplace geographies and negative residuals to find under-assigned ones. | `geography_type` (`String`): workplace-geography system.
    `geography_id` (`String`): workplace-geography identifier.
    `target_count` (`Float64`): land-use employment target.
    `modeled_count` (`Float64`): weighted assigned workers.
    `residual_count` (`Float64`): `modeled_count - target_count`.
    `absolute_residual_count` (`Float64`): absolute residual magnitude.
    `percent_error` (`Float64`): residual divided by target, times 100; null when target is zero. | +| `school_shadow_pricing_residuals` | Zone-level school target-versus-modeled residuals by student type. Use it to diagnose school-location shadow-pricing convergence and segment-specific imbalance. | `geography_type` (`String`): school-geography system.
    `geography_id` (`String`): school-geography identifier.
    `student_type` (`String`): school/enrollment market segment.
    `target_count` (`Float64`): land-use enrollment target.
    `modeled_count` (`Float64`): weighted assigned students.
    `residual_count` (`Float64`): modeled minus target.
    `absolute_residual_count` (`Float64`): absolute residual magnitude.
    `percent_error` (`Float64`): residual divided by target, times 100; null for zero targets. | +| `workplace_shadow_pricing_residual_histogram` | Distribution of workplace residuals by geography system. Use it to assess convergence across all zones without inspecting each zone separately; zero residuals receive their own zero-width bin. | `geography_type` (`String`): workplace-geography system.
    `bin_start` (`Float64`): inclusive lower residual bound.
    `bin_end` (`Float64`): upper residual bound; both bounds are zero for the exact-zero bin.
    `geography_count` (`Float64`): number of geography records in the bin. | +| `school_shadow_pricing_residual_histogram` | Distribution of school residuals by geography system and student type. Use it to compare convergence across student markets. | `geography_type` (`String`): school-geography system.
    `student_type` (`String`): school/enrollment market segment.
    `bin_start` (`Float64`): lower residual bound.
    `bin_end` (`Float64`): upper residual bound, or zero for the exact-zero bin.
    `geography_count` (`Float64`): number of geography/student-type records in the bin. | +| `park_and_ride_location_residuals` | Modeled park-and-ride tour use compared with lot capacity by lot geography. Use it to identify over-capacity or underused PNR locations. | `geography_type` (`String`): PNR-lot geography system.
    `geography_id` (`String`): PNR-lot geography identifier.
    `pnr_tour_count` (`Float64`): weighted PNR tours assigned to the location.
    `pnr_lot_capacity` (`Float64`): supplied lot capacity target.
    `residual_count` (`Float64`): tours minus capacity.
    `absolute_residual_count` (`Float64`): absolute residual magnitude.
    `percent_error` (`Float64`): residual divided by capacity, times 100; null for zero capacity. | +| `park_and_ride_location_residual_histogram` | Distribution of PNR use-minus-capacity residuals by geography system. Use it for systemwide capacity-fit assessment. | `geography_type` (`String`): PNR-lot geography system.
    `bin_start` (`Float64`): lower residual bound.
    `bin_end` (`Float64`): upper residual bound, or zero for the exact-zero bin.
    `geography_count` (`Float64`): number of PNR geography records in the bin. | +| `free_parking_eligibility_by_workplace_geography` | Workers with and without free workplace parking by workplace geography. Use it to analyze parking-cost exposure and its effect on commute mode choice. | `geography_type` (`String`): workplace-geography system.
    `geography_id` (`String`): workplace-geography identifier.
    `workers_without_free_parking_count` (`Float64`): weighted workers not eligible for free parking.
    `workers_with_free_parking_count` (`Float64`): weighted workers eligible for free parking. | + +### Long-Term Location Distance + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `work_location_distance_distribution_by_geography` | Internal, non-work-from-home workers with valid workplace locations by home geography and home-to-work distance. Use it for commute-length distributions and spatial comparisons. | `distance_bin` (`String`): exact-zero, positive-sub-mile, whole-mile lower-bound, or `51+` category.
    `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `person_count` (`Float64`): weighted workers in the bin. | +| `university_location_distance_distribution_by_geography` | University students, identified by person type 3, by home geography and home-to-school distance. Use it to examine university travel markets and campus catchments. | `distance_bin` (`String`): exact-zero, positive-sub-mile, whole-mile lower-bound, or `51+` category.
    `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `person_count` (`Float64`): weighted university students in the bin. | +| `school_location_distance_distribution_by_geography` | School students, identified by person types 6 and higher, by home geography and home-to-school distance. Use it for K--12 travel-distance and school-catchment analysis. | `distance_bin` (`String`): exact-zero, positive-sub-mile, whole-mile lower-bound, or `51+` category.
    `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `person_count` (`Float64`): weighted school students in the bin. | + +### Daily Activity Patterns and Travel Rates + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `daily_activity_pattern_by_person_type` | Daily activity pattern alternatives by person type, using each person's non-null person-table CDAP value and falling back only for remaining persons to all observed day rows with day weights when available, including an all-person-types rollup. No day rows are deduplicated. Use it to compare mandatory, nonmandatory, and home-stay behavior. | `person_type` (`String`): person-type code or rollup.
    `daily_activity_pattern` (`String`): prepared CDAP/activity-pattern category.
    `person_count` (`Float64`): weighted people or observed person-days in the pattern. | +| `mandatory_tour_frequency_by_person_type` | Positive mandatory-tour-frequency choice by person type, using each person's non-null prepared choice and falling back only for remaining persons to choices derived from work and school tours on every person-day, with day weights when available, plus an all-person-types rollup. Use it to analyze how many mandatory tours travelers make. The table excludes records with a choice of zero. | `person_type` (`String`): person-type code or rollup.
    `mandatory_tour_frequency` (`Int32`): prepared or tour-derived positive mandatory-tour frequency alternative.
    `person_count` (`Float64`): weighted people or person-days choosing that frequency. | +| `nonmandatory_tour_frequency_by_person_type` | Count of individual nonmandatory tours plus joint-tour participation, grouped as 0, 1, 2, or 3+, by person type and for all types. Genuinely multi-day inputs are classified per person-day only when every tour and participant has a safe day key; otherwise the one-day/person fallback is retained rather than dropping records or guessing keys. Use it to compare discretionary travel propensity. | `person_type` (`String`): person-type code or rollup.
    `nonmandatory_tour_frequency` (`String`): combined nonmandatory-tour category `0`, `1`, `2`, or `3+`.
    `person_count` (`Float64`): weighted people or person-days in the category. | +| `tour_rates_by_person_type_and_tour_purpose` | Person-tours per person-day by person type and tour purpose under the selected weighting mode, plus all-person-types rates. Every eligible source row contributes and at-work subtours are excluded. Joint tours are attributed using each known participant's person weight or expanded by a valid party size when participant records are unavailable. Use it to compare tour-generation rates while controlling for population composition. | `person_type` (`String`): person-type code or rollup.
    `tour_purpose` (`String`): prepared tour-purpose category.
    `tour_rate` (`Float64`): sum of eligible person-tour weights, divided by summed person or person-day weights for that person type. | +| `trip_rates_by_person_type_and_trip_purpose` | Person-trips per person-day by person type and trip purpose under the selected weighting mode, plus all-person-types rates. Every eligible source row contributes. Joint trips are attributed using each known participant's person weight or expanded by a valid party size when participant records are unavailable. Use it to compare trip-generation rates across demographic markets. | `person_type` (`String`): person-type code or rollup.
    `trip_purpose` (`String`): destination purpose of the trip.
    `trip_rate` (`Float64`): sum of eligible person-trip weights, divided by summed person or person-day weights for that person type. | + +### School Escorting + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `escorted_tour_totals` | One run-level total of adult-side tours with an outbound or inbound school-escort condition. Use it as the top-level escorted-tour control total. | `tour_count` (`Float64`): weighted distinct eligible tours with at least one escorted direction. | +| `school_escorted_tours_by_escort_type_and_direction` | Adult-side escorted tours by escort arrangement and direction, with an `all_directions` incidence rollup. Use it to compare ride-share and pure-escort patterns. | `escort_type` (`String`): prepared escort arrangement label.
    `direction` (`String`): `outbound`, `inbound`, or `all_directions`.
    `tour_count` (`Float64`): weighted escorted-tour incidences. | +| `adult_escorted_tour_purposes_by_direction` | Purposes of the adult tours that do school escorting, by direction and with an all-directions incidence rollup. Use it to see how escorting connects with work or other adult activities. | `tour_purpose` (`String`): adult tour's primary purpose.
    `direction` (`String`): escorted half or `all_directions`.
    `tour_count` (`Float64`): weighted escorted-tour incidences. | +| `adult_escorted_tours_by_person_type_and_direction` | Adult-side escorted tours by the adult traveler's person type and escorted direction. Use it to identify who performs school escorting. | `person_type` (`String`): adult traveler person-type code.
    `direction` (`String`): `outbound`, `inbound`, or `both`.
    `tour_count` (`Float64`): weighted tours meeting that directional condition. | +| `student_school_escort_status_by_direction` | Student school tours classified by normalized escort type for each direction and for tours escorted both ways. Use it to measure the student-side escort experience. | `direction` (`String`): `outbound`, `inbound`, or `both`.
    `escort_type` (`String`): normalized escort arrangement, including unescorted alternatives where present.
    `tour_count` (`Float64`): weighted student school tours in the group. | +| `student_households_by_student_count` | Households by the number of school-age/student household members recognized by the escort logic. Use it as a denominator for household escort participation. | `student_count` (`Int64`): students in the household.
    `household_count` (`Float64`): weighted households with that count. | +| `households_with_school_escorting_by_student_count_and_direction` | Unique households with at least one escorted student school tour, by number of students and directional condition. Use it to calculate escort-participation rates by household composition. | `student_count` (`Int64`): students in the household.
    `direction` (`String`): `outbound`, `inbound`, or `both`.
    `household_count` (`Float64`): weighted unique households meeting the condition. | +| `schoolkids_per_escorted_tour_by_student_count_and_direction` | Average number of escorted children on adult-side escorted tours by household student count and direction. Use it to analyze escorting efficiency and child grouping. | `student_count` (`Int64`): students in the adult traveler's household.
    `direction` (`String`): `outbound`, `inbound`, or `both`.
    `avg_schoolkids_per_tour` (`Float64`): weighted mean number of escortees per eligible tour.
    `tour_count` (`Float64`): weighted eligible tours used as the mean denominator. | +| `adult_escorted_tour_distance_distribution_by_direction` | Adult-side escorted tours by lower-bound tour-distance bin and directional escort condition. Use it to compare the length of outbound-only, inbound-only, and both-way escort tours. | `distance_bin` (`String`): exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`.
    `direction` (`String`): `outbound`, `inbound`, or `both`.
    `tour_count` (`Float64`): weighted eligible tours in the bin. | +| `adult_escorted_trip_distance_distribution_by_direction` | Trips on adult tours marked as escorted, by outbound or inbound half and lower-bound trip-distance bin. Use it to examine the trip-leg distance for escorting. | `distance_bin` (`String`): exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`.
    `direction` (`String`): `outbound`, `inbound`, or `both` condition.
    `trip_count` (`Float64`): weighted eligible trips in the bin. | +| `adult_escort_event_stop_distribution` | Intermediate stops before and after school drop-off or pickup on adult tours marked as escorted. Use it to analyze trip chains around escort events. | `segment` (`String`): one of `outbound_before_dropoff`, `outbound_after_dropoff`, `inbound_before_pickup`, or `inbound_after_pickup`.
    `stop_count` (`Int32`): prepared count of stops in that segment.
    `tour_count` (`Float64`): weighted escort-event records with that stop count. | +| `adult_escort_trip_stop_frequency` | Adult-side escorted tours jointly classified by purpose and outbound, inbound, and total stop counts. Use it to compare stop-making complexity on escort tours. | `tour_purpose` (`String`): adult tour purpose.
    `outbound_stop_count` (`Int32`): outbound stops capped at 3.
    `inbound_stop_count` (`Int32`): inbound stops capped at 3.
    `total_stop_count` (`Int32`): total stops capped at 6.
    `tour_count` (`Float64`): weighted escorted tours in the combination. | + +### Joint Travel + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `jtf_distribution` | Household observations across the legacy 21 joint-tour-frequency labels, selected by minimum counts in five fixed purpose slots covering shopping, maintenance, eating out, visiting/social, and other discretionary. Purposes outside those slots do not affect the label, and counts above a label's threshold are not rejected, so the labels are not strict total-tour counts. When the day table shows a genuinely multi-day diary and tours contain `day_num`, observations use `(household_id, day_num)`; null or unmatched tour days do not contribute. Without the tour day column, the result is empty rather than pooled into a weekly choice. On a one-day input, the declaration does not require tours, so an unavailable tour table can instead classify every household as `No Joint Tours`; verify tour availability before using that result. Use it to validate compatible joint-tour generation inputs. | `jtf_code` (`Int32`): integer alternative 1--21.
    `jtf_label` (`String`): readable minimum-frequency/purpose-combination label.
    `household_count` (`Float64`): sum of `hh.finalweight` across households or household-day observations assigned to the alternative. | +| `joint_tours_by_household_size` | All household observations and observations making at least one joint tour by household size. A genuinely multi-day diary uses household-days when tours contain `day_num`; null or unmatched tour days do not contribute, and a missing tour day column produces an empty result rather than pooling days. Valid shared joint-tour or tour IDs collapse repeated records; records without either valid ID remain distinct. Use the two counts to calculate joint-tour participation rates. | `household_size` (`Int32`): number of household members.
    `household_count` (`Float64`): sum of `hh.finalweight` across households or household-day observations of that size.
    `joint_tour_hh_count` (`Float64`): sum of `hh.finalweight` for those observations with a joint tour. | +| `joint_tour_party_size_distribution` | Household joint tours by valid participant count, with parties of five or more stored in bin 5. Verified survey participant-row groups retain every row at a fractional representation weight that sums to one household tour; ordinary model joint-tour rows retain full weight. Sentinel values of 995 or greater, missing values, and nonpositive values are excluded. Use it to assess joint-tour occupancy. | `party_size` (`Int32`): household participants; value 5 represents `5+`.
    `joint_tour_count` (`Float64`): weighted joint tours in the party-size bin. | +| `joint_tour_composition_distribution` | Household joint tours by prepared party-composition category. Verified participant-row representations retain every row fractionally so each complete group contributes one household tour. Use it to compare adult-only, child-inclusive, and other modeled compositions. | `tour_composition` (`String`): prepared joint-party composition.
    `joint_tour_count` (`Float64`): weighted joint tours in the category. | +| `joint_tour_composition_by_party_size` | Household joint tours jointly classified by party composition and exact valid participant count. Verified participant-row representations retain every row fractionally so each complete group contributes one household tour. Sentinel values of 995 or greater, missing values, and nonpositive values are excluded. Use it to study how household makeup and group size interact. | `tour_composition` (`String`): prepared party-composition category.
    `party_size` (`Int64`): number of tour participants.
    `joint_tour_count` (`Float64`): weighted joint tours in the combination. | +| `person_jtp_by_household_size` | All person observations and observations participating in one or more joint tours by household size. When a multi-day input has person-day records and each joint-tour identity maps to one day, the observations are person-days and participation comes from linked joint participants. A missing or ambiguous tour-day mapping produces an empty result; if the day table lacks household, person, or day fields, the person-level fallback remains. The multi-day path's participant-table limitations are documented in chapter 25. Use the two counts to calculate participation rates. | `household_size` (`Int64`): size of the person's household.
    `joint_tour_person_count` (`Float64`): sum of the applicable person or day weight for people or person-days participating in a joint tour.
    `total_person_count` (`Float64`): sum of the applicable person or day weight for people or person-days in households of that size. | +| `household_jtp_by_household_size_and_jtf` | For household observations of size two or more, weighted percentage distribution across 0, 1, and 2+ joint-tour identities within each household size. A genuinely multi-day diary uses household-days when tours contain `day_num`; null or unmatched tour days do not contribute, and a missing tour day column produces an empty result rather than pooling days. Valid shared joint-tour or tour IDs collapse repeated records; records without either valid ID remain distinct. Use it to compare joint-tour propensity independent of household-size totals across compatible runs. | `jtf` (`String`): joint-tour count category `0`, `1`, or `2+`.
    `household_size` (`String`): household size as a category.
    `household_percent` (`Float64`): percent of summed `hh.finalweight` for household observations of that size in the JTF category. | + +### Basic Tour Distributions + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `tour_category_distribution` | Household-tour-unit counts by ActivitySim category, such as mandatory, nonmandatory, at-work, or joint. Verified participant-row representations of one joint tour are retained fractionally rather than collapsed. Use it for high-level tour-system composition. | `tour_category` (`String`): prepared tour category.
    `tour_count` (`Float64`): weighted tours in the category. | +| `tour_purpose_distribution` | Household-tour-unit counts by configured summary purpose. Verified participant-row representations of one joint tour are retained fractionally rather than collapsed. Use it to compare the volume and share of work, school, escort, shopping, and other travel. | `tour_purpose` (`String`): canonical summary tour purpose.
    `tour_count` (`Float64`): weighted tours for that purpose. | + +### Vehicles Allocated to Tours + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `allocated_vehicle_age_by_occupancy` | Allocated vehicle ages by occupancy condition. Use it to analyze how fleet age is associated with single- and shared-occupant travel. | `age` (`String`): decoded vehicle age, with `20+` terminal.
    `occupancy` (`String`): allocation condition `1`, `2`, or `3+`.
    `vehicle_count` (`Float64`): weighted tour allocation incidences. | +| `allocated_vehicle_fuel_type_by_occupancy` | Allocated vehicle fuel/powertrain type by occupancy condition. Use it for energy or emissions segmentation of auto travel. | `fuel_type` (`String`): decoded fuel/powertrain category.
    `occupancy` (`String`): allocation condition `1`, `2`, or `3+`.
    `vehicle_count` (`Float64`): weighted tour allocation incidences. | +| `allocated_vehicle_body_type_by_occupancy` | Allocated vehicle body type by occupancy condition. Use it to relate party size to the modeled vehicle used. | `body_type` (`String`): decoded vehicle body-style category.
    `occupancy` (`String`): allocation condition `1`, `2`, or `3+`.
    `vehicle_count` (`Float64`): weighted tour allocation incidences. | + +### Tour Mode, Stops, Time, and Distance + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `tour_mode_by_tour_purpose_and_auto_sufficiency` | Person-tour mode counts by purpose and household auto sufficiency, with all-purpose rows. Joint records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback; sentinel, missing, or nonpositive fallback sizes resolve to one. Use it to compare mode choice across vehicle-availability markets. | `tour_mode` (`String`): prepared tour mode.
    `tour_purpose` (`String`): tour purpose or `all_tour_purposes`.
    `tour_count_zero_auto` (`Float64`): weighted person-tours from zero-auto households.
    `tour_count_auto_deficient` (`Float64`): weighted person-tours from households with fewer autos than workers.
    `tour_count_auto_sufficient` (`Float64`): weighted person-tours from auto-sufficient households.
    `tour_count_all_households` (`Float64`): sum of the three auto-sufficiency counts. | +| `tour_stop_frequency_by_tour_purpose` | Household-tour-unit counts jointly classified by purpose and outbound, inbound, and total intermediate-stop counts. Verified participant-row joint-tour groups retain every row at fractional weight summing to one tour. Use it to measure tour complexity and stop-generation patterns. | `tour_purpose` (`String`): canonical tour purpose.
    `outbound_stop_count` (`Int32`): outbound stops capped at 3.
    `inbound_stop_count` (`Int32`): inbound stops capped at 3.
    `total_stop_count` (`Int32`): total stops capped at 6.
    `tour_count` (`Float64`): weighted tours in the combination. | +| `atwork_subtour_frequency_distribution` | Mandatory work tours by their at-work-subtour-frequency alternative. Use it to validate subtour generation from the workplace. | `atwork_subtour_frequency_category` (`String`): prepared at-work subtour-frequency category.
    `atwork_subtour_count` (`Float64`): weighted parent work tours choosing the category. | +| `tour_time_of_day_by_tour_purpose` | Dense household-tour-unit departure, arrival, and duration profiles by tour purpose plus all-purpose totals. Every eligible row is retained; verified participant-row representations of one joint tour contribute fractionally so their combined weight equals one tour. Use it to compare scheduling and duration distributions. | `time_bin` (`Int32`): ActivitySim period index.
    `tour_purpose` (`String`): tour purpose or `all_tour_purposes`.
    `departure_tour_count` (`Float64`): weighted tours starting in the bin.
    `arrival_tour_count` (`Float64`): weighted tours ending in the bin.
    `duration_tour_count` (`Float64`): weighted tours whose prepared duration falls in the bin. | +| `tour_distance_by_tour_purpose` | Nonmandatory, joint, and at-work household-tour-unit records by lower-bound `SKIMDIST` bin and purpose, plus all-purpose totals; mandatory tours are excluded to match the legacy distribution. The metric is fixed to prepared origin-to-primary-destination `SKIMDIST` and does not switch to `tour_distance` or generic `distance_miles`. Every row is retained, with verified participant-row joint-tour groups contributing fractional weight that sums to one tour. | `distance_bin` (`String`): exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`.
    `tour_purpose` (`String`): purpose or `all_tour_purposes`.
    `tour_count` (`Float64`): sum of eligible household-tour representation weight in the bin. | + +### Tour Geography + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `average_mandatory_tour_distance_by_purpose_and_geography` | Weighted average home-to-work or home-to-school distance for internal non-WFH workers, university students, and school students, by home geography and regionwide. Use it to compare mandatory destination accessibility. | `mandatory_tour_purpose` (`String`): `work`, `university`, or `school`.
    `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup.
    `average_tour_distance` (`Float64`): mean person-level mandatory distance weighted by `per.finalweight`.
    `person_count` (`Float64`): sum of `per.finalweight` for people contributing to the mean. | +| `average_nonmandatory_tour_distance_by_purpose_and_geography` | Weighted average prepared origin-to-primary-destination `SKIMDIST` for individual nonmandatory tours by purpose and traveler home geography, plus regional rows. The metric does not switch to `tour_distance` or generic `distance_miles`. Use it to compare discretionary travel reach. | `nonmandatory_tour_purpose` (`String`): nonmandatory purpose.
    `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup.
    `average_tour_distance` (`Float64`): mean prepared `SKIMDIST` weighted by `tours.finalweight`.
    `tour_count` (`Float64`): sum of `tours.finalweight` for tours contributing to the mean. | +| `internal_external_nonmandatory_tour_frequency_by_home_geography` | Internal and external nonmandatory tours by traveler home geography, plus regional totals. Use it to calculate external-tour shares and locate households producing external discretionary travel. | `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup.
    `internal_nonmandatory_tour_count` (`Float64`): weighted internal nonmandatory tours.
    `external_nonmandatory_tour_count` (`Float64`): weighted external nonmandatory tours. | +| `external_nonmandatory_tour_locations` | External nonmandatory tours by destination geography, plus a regional total. Use it to analyze external destination orientation and gateway demand. | `geography_type` (`String`): destination-geography system or rollup.
    `geography_id` (`String`): destination-geography identifier or rollup.
    `external_nonmandatory_tour_count` (`Float64`): weighted external nonmandatory tours ending there. | + +### Trip Purpose, Mode, and Parking + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `trip_purpose_distribution` | Person-trip destination purpose cross-classified by parent tour purpose, with all-tour-purpose rows. Joint records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to analyze activity chains within different kinds of tours. | `tour_purpose` (`String`): parent tour purpose or `all_tour_purposes`.
    `trip_purpose` (`String`): destination purpose of the trip leg.
    `trip_count` (`Float64`): weighted person-trips in the combination. | +| `stop_destination_purpose_by_tour_purpose` | Person-stop destination purposes by parent tour purpose. Joint records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to understand what activities are chained into tours. | `stop_destination_purpose` (`String`): destination purpose of trip records flagged as stops.
    `tour_purpose` (`String`): parent tour purpose.
    `stop_count` (`Float64`): weighted person-stops in the combination. | +| `trip_mode_by_tour_purpose_and_tour_mode` | Person-trip mode counts by parent tour purpose and main tour mode, including all-purpose, all-tour-mode, and grand rollups. Joint records use the sum of all matching participant person weights when available; valid party-size expansion is the fallback, and sentinel, missing, or nonpositive party sizes fall back to one. Use it to examine access/egress and mode combinations within tours. | `tour_purpose` (`String`): parent purpose or `all_tour_purposes`.
    `tour_mode` (`String`): main tour mode or `all_tour_modes`.
    `trip_mode` (`String`): mode of the individual trip leg.
    `trip_count` (`Float64`): weighted person-trips in the combination. | +| `parking_locations` | Auto-trip parking events by configured parking geography. Use it to map modeled parking demand and compare locations across runs. | `geography_type` (`String`): parking-geography system.
    `geography_id` (`String`): valid positive parking-zone identifier at that geography.
    `trip_count` (`Float64`): weighted trips parking there. | + +### Trip Time and Distance + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `trip_departure_time_by_purpose` | Dense person-trip and person-stop departure-period profiles by parent tour purpose and for all purposes. Joint records use all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to compare trip timing with stop timing. | `tour_purpose` (`String`): parent purpose or `all_tour_purposes`.
    `time_bin` (`Int32`): prepared trip departure period index.
    `departure_trip_count` (`Float64`): weighted person-trips departing in the bin.
    `departure_stop_count` (`Float64`): weighted departing person-stops. | +| `trip_distance_by_purpose` | Person-trips by lower-bound OD-distance bin and parent tour purpose, plus all-purpose totals. Joint records use the sum of all matching participant person weights when available; valid party-size expansion is the fallback, non-joint trips contribute once, and sentinel, missing, or nonpositive fallback sizes resolve to one. Use it for purpose-specific trip length distributions. | `distance_bin` (`String`): exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`.
    `tour_purpose` (`String`): parent purpose or `all_tour_purposes`.
    `trip_count` (`Float64`): sum of participant-adjusted trip weight in the bin. | +| `stop_out_of_direction_distance_by_tour_purpose` | Person-stops by lower-bound out-of-direction-distance bin, with a dense distribution for each purpose and all purposes. Joint records use all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to quantify detour burden from stop-making. | `distance_bin` (`String`): exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`.
    `tour_purpose` (`String`): parent purpose or `all_tour_purposes`.
    `stop_count` (`Float64`): weighted person-stops in the bin. | + +### Skim Diagnostics + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `skimjoin_trip_component_stats` | Weighted descriptive statistics for every numeric `skim_` component on trips, by trip mode and skim scenario. Use it to QA joined time, distance, and cost values and identify missing or zero-heavy components. | `skim_scenario` (`String`): chosen or hypothetical evaluation scenario.
    `trip_mode` (`String`): trip mode or `All Modes`.
    `component` (`String`): numeric skim column name.
    `n_total` (`Float64`): total trip weight eligible for the component/mode group.
    `n_valid` (`Float64`): trip weight with non-null component values, including zeros.
    `mean` (`Float64`): weighted mean including zeros.
    `mean_nonzero` (`Float64`): weighted mean excluding zeros.
    `std` (`Float64`): weighted population standard deviation.
    `min` (`Float64`): minimum observed value.
    `max` (`Float64`): maximum observed value.
    `median` (`Float64`): weighted 50th percentile.
    `mode` (`Float64`): value with greatest total weight, using the smaller value on ties.
    `zero_share` (`Float64`): valid weight at exactly zero divided by `n_valid`.
    `missing_share` (`Float64`): missing weight divided by `n_total`. | +| `skimjoin_trip_component_ecdf` | Optional 0th-through-100th weighted percentile curves for trip skim components by mode and scenario. Use it for distribution comparison when the compact stats table is insufficient. | `skim_scenario` (`String`): chosen or hypothetical scenario.
    `trip_mode` (`String`): trip mode or `All Modes`.
    `component` (`String`): numeric skim column name.
    `percentile` (`Float64`): cumulative probability from 0.00 through 1.00 in 0.01 steps.
    `value` (`Float64`): weighted quantile at that probability.
    `n_valid` (`Float64`): total valid trip weight behind the curve. | +| `skimjoin_tour_component_stats` | Weighted descriptive statistics for numeric tour `skim_` components by tour mode and scenario. Use it to QA tour-level round-trip or composite skims. | `skim_scenario` (`String`): chosen or hypothetical evaluation scenario.
    `tour_mode` (`String`): tour mode or `All Modes`.
    `component` (`String`): numeric skim column name.
    `n_total` (`Float64`): total tour weight in scope.
    `n_valid` (`Float64`): tour weight with a non-null value, including zeros.
    `mean` (`Float64`): weighted mean including zeros.
    `mean_nonzero` (`Float64`): weighted mean excluding zeros.
    `std` (`Float64`): weighted population standard deviation.
    `min` (`Float64`): minimum value.
    `max` (`Float64`): maximum value.
    `median` (`Float64`): weighted median.
    `mode` (`Float64`): highest-weight value, smaller on ties.
    `zero_share` (`Float64`): valid weight at zero divided by `n_valid`.
    `missing_share` (`Float64`): missing weight divided by `n_total`. | +| `skimjoin_tour_component_ecdf` | Optional weighted percentile curves for tour skim components by mode and scenario. Use it to compare complete tour-level distributions across runs. | `skim_scenario` (`String`): chosen or hypothetical scenario.
    `tour_mode` (`String`): tour mode or `All Modes`.
    `component` (`String`): numeric skim column name.
    `percentile` (`Float64`): probability from 0.00 through 1.00.
    `value` (`Float64`): weighted quantile at that probability.
    `n_valid` (`Float64`): valid tour weight behind the curve. | + +### Processor-Built Validation Summaries + +| Summary Table | Information and Analytical Use | Fields | +|---|---|---| +| `traffic_count_comparisons` | Observed and modeled traffic counts that agree at count-location, direction, and period level. Use it for count scatterplots, percent differences, RMSE, and facility calibration. The table keeps only keys in both sources. | `count_location_id` (`String`): traffic-count station/location identifier.
    `direction` (`String`): observed/modeled direction label.
    `count_period` (`String`): count time-period label.
    `observed_volume` (`Float64`): summed observed count for the key.
    `modeled_volume` (`Float64`): summed assigned volume for the matching key. | +| `screenline_flow_comparisons` | Observed and modeled screenline flows matched by screenline, direction, and period, with a representative facility type. Use it for corridor-level flow validation and regression analysis. | `screenline_id` (`String`): screenline/cutline identifier.
    `direction` (`String`): flow direction.
    `count_period` (`String`): comparison period.
    `facility_type` (`String`): supplied facility class, or `All` if absent.
    `observed_volume` (`Float64`): summed observed flow.
    `modeled_volume` (`Float64`): summed modeled flow for the matching key. | +| `transit_boardings_by_operator_and_technology` | Assigned transit boardings summed by operator and transit technology. Use it to compare ridership scale across agencies and modes. | `operator` (`String`): supplied transit operator identifier or name.
    `technology` (`String`): supplied transit mode/technology category.
    `boardings` (`Float64`): total assigned unlinked passenger boardings. | +| `transit_transfer_rate` | Assigned boardings divided by linked transit trips by operator, technology, and access mode. The value is boardings per linked trip, so values above one indicate transfers; subtract one if a transfers-per-trip measure is needed. | `operator` (`String`): transit operator.
    `technology` (`String`): transit technology/mode.
    `access_mode` (`String`): mode used to access transit.
    `transfer_rate` (`Float64`): assigned boardings divided by linked trips; null for a zero linked-trip denominator. | +| `auto_vmt_totals` | One run-level personal-auto VMT total using the same eligible records, distance source, and legacy-compatible occupancy calculation as the detailed personal-auto VMT summary. Every eligible row is retained. Household-level joint rows use one vehicle, while repeated person-level joint rows use their fixed auto-mode occupancy. A prepared auto skim is preferred; otherwise eligible non-drive-transit trips use `od_dist`. Use it for overview controls and scenario comparison. | `auto_vmt` (`Float64`): total weighted personal-auto vehicle miles traveled. | +| `auto_vmt_by_home_geography_income_hhsize_time_period` | Personal-auto VMT and trip counts by home geography, income, household size, time period, and auto mode. When component periods exist, Daily rows are derived only from those non-Daily rows and replace any stale source Daily row; Daily-only groups are retained. Use it for equity, temporal, modal, and spatial VMT analysis. | `geography_type` (`String`): home-geography system or `all_geographies`.
    `geography_id` (`String`): home-geography identifier or rollup.
    `income_segment` (`String`): prepared household income segment or fallback rollup.
    `household_size` (`String`): prepared household size or fallback rollup.
    `time_period` (`String`): configured period or `Daily`.
    `mode` (`String`): trip mode or `All Auto` fallback.
    `auto_vmt` (`Float64`): sum of distance times `trips.finalweight`, divided by legacy-compatible vehicle occupancy.
    `trip_count` (`Float64`): sum of `trips.finalweight` for eligible auto trips.
    `distance_source` (`String`): provenance of the distance used, such as a skim or OD-distance field.
    `time_period_source` (`String`): provenance of the time-period assignment. | +| `non_motorized_vmt_by_home_geography_income_hhsize_time_period` | Walk, bicycle, and e-bike weighted miles and trip counts by home geography, income, household size, period, and mode, including derived daily rows. Use it for active-travel exposure and equity analysis. | `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup.
    `income_segment` (`String`): household income segment or fallback rollup.
    `household_size` (`String`): household size or fallback rollup.
    `time_period` (`String`): configured period or `Daily`.
    `mode` (`String`): `WALK`, `BIKE`, or `EBIKE` as available.
    `non_motorized_vmt` (`Float64`): distance times `trips.finalweight`; despite the VMT name, this is weighted traveler mileage.
    `trip_count` (`Float64`): sum of `trips.finalweight` for eligible trips.
    `distance_source` (`String`): prepared or skim distance source used for the mode.
    `time_period_source` (`String`): provenance of the period assignment. | +| `commercial_vmt_totals` | Commercial-vehicle VMT by vehicle type, split between internal and external travel. Use it for freight VMT totals and internal/external shares. | `commercial_vehicle_type` (`String`): supplied commercial vehicle/truck class.
    `external_vmt` (`Float64`): VMT from records classified as external.
    `internal_vmt` (`Float64`): VMT from records classified as internal. | +| `bicycle_vmt_by_facility_type` | Bicycle VMT by facility type, read directly when supplied or calculated as assigned bicycle trips times link distance. Use it to evaluate bicycle use by facility class. | `facility_type` (`String`): supplied bicycle/network facility category.
    `bicycle_vmt` (`Float64`): summed bicycle vehicle/traveler miles on that facility type. | + +## 3. Hypothetical All-Modes Skim Tables + +These optional long-form tables evaluate skim components for all configured modes, not only the observed mode. Each file uses the prepared-cache format and is named `.`. Operational skimjoin QA report CSVs are outside the scope of this reference. + +| Hypothetical Skim Table | Information and Analytical Use | Fields | +|---|---|---| +| `trip_hypothetical_skims` | Optional long-form trip skim values evaluated under configured modes other than each trip's observed mode. One row represents a trip, hypothetical mode, and skim component. Use it to compare travel impedance across modal alternatives without duplicating the full prepared trip table. It is written only when hypothetical skim tables are enabled and populated. | `trip_id` (`Int64`): Canonical identifier of the evaluated trip.
    `observed_mode` (`String`): Mode selected in the source trip record.
    `hypothetical_mode` (`String`): Alternative mode used for the skim lookup.
    `component` (`String`): Configured skim output name, which determines the value's units and interpretation.
    `value` (`Float64`): Skim value for the trip and hypothetical mode; null when the lookup is unresolved.
    `finalweight` (`Float64`): Expansion weight inherited from the prepared trip record. | +| `tour_hypothetical_skims` | Optional long-form tour skim values evaluated under configured modes other than each tour's observed mode. One row represents a tour, hypothetical mode, direction, and skim component. Use it to compare outbound, inbound, and combined tour impedance across modal alternatives. It is written only when hypothetical skim tables are enabled and populated. | `tour_id` (`Int64`): Canonical identifier of the evaluated tour.
    `observed_mode` (`String`): Main mode selected in the source tour record.
    `hypothetical_mode` (`String`): Alternative mode used for the skim lookup.
    `direction` (`String`): Outbound or inbound for directional components; null for nondirectional components.
    `component` (`String`): Configured skim output name, which determines the value's units and interpretation.
    `value` (`Float64`): Skim value for the tour, direction, and hypothetical mode; null when unresolved.
    `finalweight` (`Float64`): Expansion weight inherited from the prepared tour record. | + +## 4. Externally Supplied Table Schemas + +**These tables are not created by the processor.** They are registered schemas for validation files supplied through `summary_table_map`. A mapped file can be CSV or Parquet and its configured filename does not need to match the schema name. + +| Externally Supplied Table Schema | Information and Analytical Use | Fields | +|---|---|---| +| `link_validation_summary` | Modeled network link volumes with link endpoints and facility class. Use it to aggregate modeled flow by facility or inspect high-volume links. | `id` (`Int64`): link identifier.
    `From_Node` (`Int64`): upstream node identifier.
    `To_Node` (`Int64`): downstream node identifier.
    `FACTYPE` (`Int64`): facility-type code.
    `am_vol` (`Float64`): AM-period modeled link volume.
    `md_vol` (`Float64`): midday modeled link volume.
    `pm_vol` (`Float64`): PM-period modeled link volume.
    `day_vol` (`Float64`): daily modeled link volume. | +| `count_location_counts_validation_summary` | Observed traffic-count volumes by count location and facility class. Use it as the observed side of location-level modeled-versus-observed comparisons. | `id` (`Int64`): count-location identifier.
    `FACTYPE` (`Int64`): facility-type code.
    `am_vol` (`Float64`): observed AM volume.
    `md_vol` (`Float64`): observed midday volume.
    `pm_vol` (`Float64`): observed PM volume.
    `day_vol` (`Float64`): observed daily volume. | +| `count_location_volumes_validation_summary` | Modeled volumes at the traffic-count locations. Use it as the modeled side of location-level count comparisons. | `id` (`Int64`): count-location identifier matching the observed table.
    `FACTYPE` (`Int64`): facility-type code.
    `am_vol` (`Float64`): modeled AM volume.
    `md_vol` (`Float64`): modeled midday volume.
    `pm_vol` (`Float64`): modeled PM volume.
    `day_vol` (`Float64`): modeled daily volume. | +| `count_location_scatter_validation_summary` | Long-form observed/modeled point pairs already prepared for count scatterplots. Use it when the source workflow supplies paired values directly. | `id` (`Int64`): count-location identifier.
    `facility_type` (`String`): facility class code or label.
    `period` (`String`): comparison period.
    `observed_volume` (`Float64`): observed traffic volume.
    `modeled_volume` (`Float64`): modeled traffic volume. | +| `count_location_fit_validation_summary` | Precomputed linear-fit diagnostics for observed-versus-modeled counts by facility type and period. Use it to draw regression lines and report calibration fit. | `facility_type` (`String`): facility class used for the fit.
    `period` (`String`): time period used for the fit.
    `slope` (`Float64`): fitted slope for modeled volume as a function of observed volume.
    `intercept` (`Float64`): fitted modeled-volume intercept.
    `r_squared` (`Float64`): coefficient of determination.
    `n_locations` (`Int64`): paired count locations in the fit.
    `observed_min` (`Float64`): minimum observed volume in the fitting data.
    `observed_max` (`Float64`): maximum observed volume.
    `equation_label` (`String`): preformatted regression-equation text.
    `r_squared_label` (`String`): preformatted R-squared text. | +| `district_commuting_flows_validation_summary` | Supplied district-to-district commute-flow matrix for Albany, Corvallis, Lebanon, and Philomath. Use it as a local validation/control matrix. | `""` (`String`): origin district or row label.
    `Albany` (`Float64`): commuters to Albany.
    `Corvallis` (`Float64`): commuters to Corvallis.
    `Lebanon` (`Float64`): commuters to Lebanon.
    `Philomath` (`Float64`): commuters to Philomath.
    `Total` (`Float64`): row total across destinations. | +| `county_commuting_flows_validation_summary` | Supplied county-to-county commute-flow matrix for Benton, Linn, and Marion counties. Use it as a regional commute-flow validation/control matrix. | `""` (`String`): origin county or row label.
    `Benton` (`Float64`): commuters to Benton County.
    `Linn` (`Float64`): commuters to Linn County.
    `Marion` (`Float64`): commuters to Marion County.
    `Total` (`Float64`): row total across destinations. | +| `commercial_vehicle_validation_summary` | Supplied commercial-vehicle trip totals by time of day and vehicle class. Use it to compare commercial demand composition and daily profiles. | `tod` (`String`): time-of-day row label.
    `car` (`Float64`): commercial-car/light-vehicle trips.
    `mu` (`Float64`): multi-unit truck trips.
    `su` (`Float64`): single-unit truck trips.
    `Total` (`Float64`): total commercial trips across classes. | +| `commercial_vehicle_vmt_validation_summary` | Supplied commercial-vehicle VMT by time of day and vehicle class. Use it to compare freight mileage composition and temporal patterns. | `tod` (`String`): time-of-day row label.
    `car` (`Float64`): commercial-car/light-vehicle VMT.
    `mu` (`Float64`): multi-unit truck VMT.
    `su` (`Float64`): single-unit truck VMT.
    `Total` (`Float64`): total commercial VMT across classes. | +| `external_trip_validation_summary` | Supplied external trip totals by time of day and purpose/class. Use it to analyze gateway demand by travel market. | `tod` (`String`): time-of-day row label.
    `hbcoll` (`Float64`): home-based college trips.
    `hbo` (`Float64`): home-based other trips.
    `hbr` (`Float64`): home-based recreation trips.
    `hbs` (`Float64`): home-based shopping trips.
    `hbsch` (`Float64`): home-based school trips.
    `hbw` (`Float64`): home-based work trips.
    `nhbnw` (`Float64`): non-home-based non-work trips.
    `nhbw` (`Float64`): non-home-based work trips.
    `truck` (`Float64`): truck trips.
    `Total` (`Float64`): total external trips across purposes/classes. | +| `external_vmt_validation_summary` | Supplied external VMT by time of day and purpose/class. Use it to identify which external markets contribute mileage. | `tod` (`String`): time-of-day row label.
    `hbcoll` (`Float64`): home-based college VMT.
    `hbo` (`Float64`): home-based other VMT.
    `hbr` (`Float64`): home-based recreation VMT.
    `hbs` (`Float64`): home-based shopping VMT.
    `hbsch` (`Float64`): home-based school VMT.
    `hbw` (`Float64`): home-based work VMT.
    `nhbnw` (`Float64`): non-home-based non-work VMT.
    `nhbw` (`Float64`): non-home-based work VMT.
    `truck` (`Float64`): truck VMT.
    `Total` (`Float64`): total external VMT across purposes/classes. | +| `auto_vmt_validation_summary` | Supplied auto and truck VMT by time of day and occupancy class. Use it as an independent control for modeled VMT. | `TOD` (`String`): time-of-day row label.
    `SOV` (`Float64`): single-occupant-vehicle VMT.
    `HOV2` (`Float64`): two-occupant shared-ride VMT.
    `HOV3` (`Float64`): three-or-more-occupant shared-ride VMT.
    `Truck` (`Float64`): truck VMT.
    `Total` (`Float64`): total VMT across listed classes. | +| `work_from_home_validation_summary` | Supplied worker and work-from-home controls by district. Use it to compare modeled WFH counts or rates with external targets. | `District` (`String`): district name or identifier.
    `Workers` (`Float64`): total workers in the district.
    `WFH` (`Float64`): workers who work from home. | + diff --git a/run.py b/run.py index cac8221..1f5f74c 100644 --- a/run.py +++ b/run.py @@ -10,6 +10,7 @@ from pathlib import Path from runtime.logging import configure_logging, get_logger, shutdown_logging +from runtime.run_lock import RunLockError, acquire_run_lock, release_run_lock from dashboard.page_registry import ( export_data_requirements, live_data_requirements, @@ -119,6 +120,16 @@ def parse_args() -> argparse.Namespace: action="store_true", help="Do not open the dashboard in a browser automatically", ) + parser.add_argument( + "--explain-cache", + action="store_true", + help="Print cache decisions for the configured pipeline and exit without running it.", + ) + parser.add_argument( + "--log-path", + metavar="PATH", + help="Write the runtime log to PATH instead of the default shared log file.", + ) return parser.parse_args() @@ -216,7 +227,7 @@ def resolve_effective_dashboard_mode( def resolve_effective_plan(args: argparse.Namespace, config) -> WorkflowPlan: - """Resolve logical steps, runtime steps, dashboard mode, and overwrite policy.""" + """Resolve logical steps, runtime steps, dashboard mode, and refresh policy.""" logical_steps = resolve_requested_steps(args, config) dashboard_mode = resolve_effective_dashboard_mode( args, @@ -227,16 +238,28 @@ def resolve_effective_plan(args: argparse.Namespace, config) -> WorkflowPlan: logical_steps = [ step for step in logical_steps if step != "dashboard" ] - overwrite = bool(config.pipeline.overwrite) - if args.refresh_caches or args.refresh_prepared_cache or args.refresh_summary_cache: - overwrite = True + refresh_steps = set(config.pipeline.refresh) + if args.refresh_caches: + refresh_steps.update( + step + for step in logical_steps + if step in {"prepare", "skimjoin", "summarize"} + ) + if args.refresh_prepared_cache: + refresh_steps.add("prepare") + if args.refresh_summary_cache: + refresh_steps.add("summarize") runtime_steps = tuple(collapse_runtime_steps(logical_steps)) return WorkflowPlan( logical_steps=tuple(logical_steps), runtime_steps=runtime_steps, dashboard_mode=dashboard_mode, - overwrite=overwrite, + refresh_steps=tuple( + step + for step in ("prepare", "skimjoin", "summarize") + if step in refresh_steps + ), ) @@ -245,13 +268,23 @@ def _remove_run_cache_dirs( root: Path, run_keys: list[str], cache_label: str, + preserve_names: set[str] | None = None, ) -> None: """Remove per-run cache directories before a forced rebuild.""" for run_key in run_keys: cache_dir = root / run_key if cache_dir.exists(): LOGGER.info("Refreshing %s cache for run key %r", cache_label, run_key) - shutil.rmtree(cache_dir) + if not preserve_names: + shutil.rmtree(cache_dir) + continue + for child in cache_dir.iterdir(): + if child.name in preserve_names: + continue + if child.is_dir(): + shutil.rmtree(child) + else: + child.unlink() def _refresh_requested_caches( @@ -281,6 +314,7 @@ def _refresh_requested_caches( root=cache_root, run_keys=run_keys, cache_label="summary", + preserve_names={"prepared_tables", "base_prepared_tables"}, ) return refresh_prepared, refresh_summary @@ -292,11 +326,11 @@ def resolve_cache_preferences( refreshed_summary: bool, ) -> tuple[bool, bool]: """Resolve cache reuse preferences after config defaults and CLI refresh overrides.""" - prefer_prepared_cache = not ( - "prepare" in plan.runtime_steps and plan.overwrite + prefer_prepared_cache = not any( + plan.refreshes(step) for step in ("prepare", "skimjoin") ) - prefer_summary_cache = not ( - "summarize" in plan.runtime_steps and plan.overwrite + prefer_summary_cache = not any( + plan.refreshes(step) for step in ("prepare", "skimjoin", "summarize") ) if refreshed_prepared: prefer_prepared_cache = False @@ -336,6 +370,149 @@ def resolve_dashboard_execution_mode(dashboard_mode: str) -> str: return normalized_mode +def explain_cache_plan( + *, + config, + plan: WorkflowPlan, + prepared_root: Path, + cache_root: Path, + run_entries: list[dict], +) -> None: + """Print cache decisions without loading tables or writing artifacts.""" + from processor.prepare.cache import PreparedCacheError, inspect_prepared_run_cache + from processor.summarize import builder as summary_builder + from processor.summarize import cache as summary_cache + from processor.summarize.cache_types import SummaryCacheError + from runtime.workflows import prepare as prepare_workflow + from runtime.workflows import summarize as summarize_workflow + + effective = runtime_workflows.effective_processor_config(config, plan=plan) + + def decision(action: str, reason: str | None = None) -> str: + return action if not reason else f"{action} — {reason}" + + for entry, run_key in runtime_workflows.run_entries_with_keys(run_entries): + prepare_metadata = prepare_workflow._run_cache_metadata( + entry=entry, + run_key=run_key, + config=effective, + ) + label = str(prepare_metadata["label"]) + print(f"Pipeline plan — {label}") + + prepare_action = "DISABLED" + prepare_reason = None + if "prepare" in plan.runtime_steps or "summarize" in plan.runtime_steps: + if plan.refreshes("prepare"): + prepare_action = "REBUILD" + prepare_reason = "explicitly refreshed" + else: + base_cache = ( + prepare_workflow.base_prepared_cache_dir(prepared_root, run_key) + if plan.includes("skimjoin") + else prepare_workflow.prepared_cache_dir(prepared_root, run_key) + ) + base_digest = ( + effective.base_prepare_config_digest + if plan.includes("skimjoin") + else effective.prepare_config_digest + ) + base_fingerprint = dict( + prepare_metadata[ + "base_run_fingerprint" + if plan.includes("skimjoin") + else "run_fingerprint" + ] + ) + try: + inspect_prepared_run_cache( + base_cache, + expected_prepare_config_digest=base_digest, + expected_run_fingerprint=base_fingerprint, + expected_label=label, + expected_run_key=run_key, + ) + prepare_action = "REUSE" + except PreparedCacheError as exc: + prepare_action = "REBUILD" + prepare_reason = str(exc) + print(f" prepare {decision(prepare_action, prepare_reason)}") + + if plan.includes("skimjoin"): + if plan.refreshes("skimjoin"): + skimjoin_action = decision("REBUILD", "explicitly refreshed") + elif prepare_action == "REBUILD": + skimjoin_action = decision("REBUILD", "upstream prepare will change") + else: + try: + inspect_prepared_run_cache( + prepare_workflow.prepared_cache_dir(prepared_root, run_key), + expected_prepare_config_digest=effective.prepare_config_digest, + expected_run_fingerprint=dict(prepare_metadata["run_fingerprint"]), + expected_label=label, + expected_run_key=run_key, + ) + skimjoin_action = "REUSE" + except PreparedCacheError as exc: + skimjoin_action = decision("REBUILD", str(exc)) + print(f" skimjoin {skimjoin_action}") + else: + print(" skimjoin DISABLED") + + if "summarize" in plan.runtime_steps: + if any(plan.refreshes(step) for step in ("prepare", "skimjoin", "summarize")): + summary_action = decision("REBUILD", "explicit or upstream refresh") + elif prepare_action == "REBUILD": + summary_action = decision("REBUILD", "upstream prepare will change") + else: + summary_metadata = summarize_workflow._run_cache_metadata( + entry=entry, + run_key=run_key, + config=effective, + ) + try: + inspection = summary_cache.inspect_summary_run_bundle( + cache_root / run_key, + effective, + expected_modes=effective.weighting_modes, + expected_summary_ids=list(summary_builder.DEFAULT_SUMMARY_IDS), + expected_run_fingerprint=dict(summary_metadata["run_fingerprint"]), + expected_prepared_manifest_identity=summary_metadata[ + "prepared_manifest_identity" + ], + expected_label=label, + expected_run_key=run_key, + ) + stale_count = sum( + len(summary_ids) + for summary_ids in dict( + inspection["stale_summary_ids_by_unit"] + ).values() + ) + obsolete_count = len(inspection["obsolete_unit_keys"]) + summary_action = ( + decision( + "REBUILD", + ( + f"{stale_count} analysis-unit summary tables are stale; " + f"{obsolete_count} analysis units are obsolete" + ), + ) + if stale_count or obsolete_count + else "REUSE" + ) + except SummaryCacheError as exc: + summary_action = decision("REBUILD", str(exc)) + print(f" summarize {summary_action}") + else: + print(" summarize DISABLED") + + print( + " dashboard " + + ("RUN" if "dashboard" in plan.runtime_steps else "DISABLED") + ) + + def main() -> None: t0 = time.perf_counter() args = parse_args() @@ -347,22 +524,41 @@ def main() -> None: sys.exit(1) config = runtime_workflows.load_runtime_config(args.config) - log_path = configure_logging(config, level=_resolve_terminal_log_level(config)) - LOGGER.info("Starting ActivitySim Visualizer") - LOGGER.info("Loading config: %s", args.config) - LOGGER.info("Logging to %s", log_path) + run_lock = None + if not args.explain_cache: + try: + run_lock = acquire_run_lock(config.summary_root) + except RunLockError as exc: + print(f"Error: {exc}", file=sys.stderr) + sys.exit(1) + try: + log_path = configure_logging( + config, + level=_resolve_terminal_log_level(config), + log_path=args.log_path, + ) + except Exception: + release_run_lock(run_lock) + raise + LOGGER.info("Starting ActivitySim Visualizer") + LOGGER.info("Loading config: %s", args.config) + LOGGER.info("Logging to %s", log_path) try: plan = resolve_effective_plan(args, config) steps = list(plan.runtime_steps) - LOGGER.info("Requested workflow steps: %s", ", ".join(steps) if steps else "(none)") + LOGGER.info( + "Requested workflow steps: %s", + ", ".join(plan.logical_steps) if plan.logical_steps else "(none)", + ) LOGGER.info("Effective dashboard mode: %s", plan.dashboard_mode) cache_root = runtime_workflows.summary_cache_root( - config, create="summarize" in steps + config, create="summarize" in steps and not args.explain_cache ) prepared_root = runtime_workflows.prepared_cache_root( config, - create="prepare" in steps or "summarize" in steps, + create=("prepare" in steps or "summarize" in steps) + and not args.explain_cache, ) run_entries = runtime_workflows.resolve_run_entries( @@ -371,6 +567,15 @@ def main() -> None: config=config, require_runs="prepare" in steps or "summarize" in steps, ) + if args.explain_cache: + explain_cache_plan( + config=config, + plan=plan, + prepared_root=prepared_root, + cache_root=cache_root, + run_entries=run_entries, + ) + return refreshed_prepared, refreshed_summary = _refresh_requested_caches( args=args, prepared_root=prepared_root, @@ -443,6 +648,7 @@ def main() -> None: explicit_cache_dirs=args.from_csvs, run_entries=run_entries, required_summary_ids=dashboard_requirements.required_summary_ids, + optional_summary_ids=dashboard_requirements.optional_summary_ids, ) required_run_keys = [summary_run.run_key for summary_run in summary_runs] @@ -507,6 +713,8 @@ def main() -> None: print(f"Error: {exc}", file=sys.stderr) shutdown_logging() sys.exit(1) + finally: + release_run_lock(run_lock) elapsed = (time.perf_counter() - t0) / 60 LOGGER.info("Dashboard created in %.2f minutes.", elapsed) diff --git a/runtime/config/__init__.py b/runtime/config/__init__.py index acbdcd9..cca9ff7 100644 --- a/runtime/config/__init__.py +++ b/runtime/config/__init__.py @@ -11,6 +11,7 @@ ExportHTMLSettings, ExportSelectorRequest, PipelineSettings, + PrepareCategoryMappingsSettings, PrepareNonMotorizedDistanceSkimSettings, PreparedColumnSegmentationSource, SegmentationDefinition, @@ -28,6 +29,7 @@ "ExportHTMLSettings", "ExportSelectorRequest", "PipelineSettings", + "PrepareCategoryMappingsSettings", "PrepareNonMotorizedDistanceSkimSettings", "PreparedColumnSegmentationSource", "SegmentationDefinition", diff --git a/runtime/config/constants.py b/runtime/config/constants.py index 489c8e9..a0447a1 100644 --- a/runtime/config/constants.py +++ b/runtime/config/constants.py @@ -13,7 +13,11 @@ "land_use": "final_land_use", } -PREPARED_TABLE_MAP_KEYS: tuple[str, ...] = tuple(FILE_MAPPING_DEFAULTS) +PREPARED_TABLE_MAP_KEYS: tuple[str, ...] = ( + *FILE_MAPPING_DEFAULTS, + "trip_hypothetical_skims", + "tour_hypothetical_skims", +) OPTIONAL_PREPARED_TABLE_IDS: set[str] = { "day", "vehicles", diff --git a/runtime/config/loader.py b/runtime/config/loader.py index 2d8db3a..44b2f7b 100644 --- a/runtime/config/loader.py +++ b/runtime/config/loader.py @@ -3,6 +3,7 @@ from __future__ import annotations import hashlib +import mimetypes from pathlib import Path from typing import TypeVar @@ -48,6 +49,28 @@ ConfigT = TypeVar("ConfigT", bound=Config) + +def _normalize_dashboard_logo(raw_value, *, config_dir: Path) -> str | None: + if raw_value is None: + return None + if not isinstance(raw_value, str) or not raw_value.strip(): + raise ValueError("dashboard.logo must be a non-empty image path when provided.") + + logo_path = Path(raw_value.strip()) + if not logo_path.is_absolute(): + logo_path = config_dir / logo_path + logo_path = logo_path.resolve() + if not logo_path.is_file(): + raise ValueError(f"dashboard.logo file does not exist: {logo_path}") + + media_type, _ = mimetypes.guess_type(logo_path.name) + if media_type is None or not media_type.startswith("image/"): + raise ValueError( + f"dashboard.logo must reference a recognized image file: {logo_path}" + ) + return str(logo_path) + + def load_config_from_yaml(path: str | Path, *, cls: type[ConfigT] = Config) -> ConfigT: config_path = Path(path).resolve() config_bytes = config_path.read_bytes() @@ -182,6 +205,10 @@ def load_config_from_yaml(path: str | Path, *, cls: type[ConfigT] = Config) -> C ) dashboard_title = dashboard_cfg.get("title", "ActivitySim Visualizer") + dashboard_logo = _normalize_dashboard_logo( + dashboard_cfg.get("logo"), + config_dir=config_path.parent, + ) log_level = str(raw.get("log_level", "INFO")).strip().upper() if log_level not in {"DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"}: raise ValueError( @@ -265,11 +292,13 @@ def load_config_from_yaml(path: str | Path, *, cls: type[ConfigT] = Config) -> C config = cls( config_path=str(config_path), config_digest=hashlib.sha256(config_bytes).hexdigest(), + base_prepare_config_digest="", prepare_config_digest="", summary_config_digest="", presentation_config_digest="", name=raw.get("name", ""), dashboard_title=str(dashboard_title), + dashboard_logo=dashboard_logo, log_level=log_level, pipeline=pipeline, dashboard_pages=dashboard_pages, @@ -317,6 +346,9 @@ def load_config_from_yaml(path: str | Path, *, cls: type[ConfigT] = Config) -> C ) if not config.pnr_tour_modes: raise ValueError("summarize.pnr_tour_modes must resolve to at least one mode.") + config.base_prepare_config_digest = digest_payload( + config.base_prepare_signature_payload() + ) config.prepare_config_digest = digest_payload(config.prepare_signature_payload()) config.summary_config_digest = digest_payload(config.summary_signature_payload()) config.presentation_config_digest = digest_payload( diff --git a/runtime/config/models.py b/runtime/config/models.py index cc79350..1382f85 100644 --- a/runtime/config/models.py +++ b/runtime/config/models.py @@ -132,11 +132,14 @@ class PipelineSettings: steps: tuple[str, ...] = ("summarize", "dashboard") dashboard_mode: Literal["none", "live", "export", "host"] = "live" - overwrite: bool = False + refresh: tuple[str, ...] = () def has_step(self, step: str) -> bool: return step in self.steps + def refreshes(self, step: str) -> bool: + return step in self.refresh + @dataclass(frozen=True) class SkimjoinSettings: @@ -156,6 +159,7 @@ class SkimjoinSettings: class RunSkimjoinOverrides: """Optional per-run skimjoin overrides resolved from the main config.""" + enabled: bool | None = None config_path: str | None = None skim_files: tuple[str, ...] = () network_los_file: str | None = None @@ -176,6 +180,34 @@ def mapping_for_run(self, run_label: str) -> dict[str, str] | None: return self.mappings.get(normalize_run_selector_key(run_label)) +@dataclass(frozen=True) +class PrepareCategoryColumnMapping: + """One raw-to-canonical categorical column mapping.""" + + source_column: str + mapping: dict[str, object] = field(default_factory=dict) + output_type: Literal["string", "boolean"] = "string" + preserve_unmapped: bool = True + + +@dataclass(frozen=True) +class PrepareCategoryMappingsSettings: + """Run-aware categorical mappings applied before prepare enrichment.""" + + mappings: dict[ + str, dict[str, dict[str, PrepareCategoryColumnMapping]] + ] = field(default_factory=dict) + + @property + def enabled(self) -> bool: + return bool(self.mappings) + + def mapping_for_run( + self, run_label: str + ) -> dict[str, dict[str, PrepareCategoryColumnMapping]] | None: + return self.mappings.get(normalize_run_selector_key(run_label)) + + @dataclass(frozen=True) class PrepareAutoSufficiencySettings: """Configurable household comparison basis for AUTOSUFF derivation.""" @@ -341,11 +373,13 @@ class Config: config_path: str config_digest: str + base_prepare_config_digest: str prepare_config_digest: str summary_config_digest: str presentation_config_digest: str name: str dashboard_title: str + dashboard_logo: str | None log_level: str pipeline: PipelineSettings dashboard_pages: list[DashboardPageConfigEntry] | None @@ -363,6 +397,7 @@ class Config: export_html: ExportHTMLSettings skimjoin: SkimjoinSettings prepare_vot_bins: PrepareVotBinsSettings + prepare_category_mappings: PrepareCategoryMappingsSettings prepare_auto_sufficiency: PrepareAutoSufficiencySettings prepare_time_periods: PrepareTimePeriodsSettings prepare_non_motorized_distance_skim: PrepareNonMotorizedDistanceSkimSettings @@ -458,6 +493,11 @@ def prepare_signature_payload(self) -> dict[str, Any]: return prepare_signature_payload(self) + def base_prepare_signature_payload(self) -> dict[str, Any]: + from .signatures import base_prepare_signature_payload + + return base_prepare_signature_payload(self) + def summary_signature_payload(self) -> dict[str, Any]: from .signatures import summary_signature_payload @@ -485,7 +525,7 @@ def weighting_mode_label(self, mode_id: str) -> str: def skimjoin_step_enabled(self) -> bool: """Return whether the active pipeline includes integrated skimjoin.""" - return self.pipeline.has_step("skimjoin") + return self.pipeline.has_step("skimjoin") and self.skimjoin.enabled def summary_category_spec(self, category_id: str) -> CategorySpec | None: return self.summary_categories.get(str(category_id)) diff --git a/runtime/config/normalize_prepare.py b/runtime/config/normalize_prepare.py index b7d88ce..1911f3a 100644 --- a/runtime/config/normalize_prepare.py +++ b/runtime/config/normalize_prepare.py @@ -20,6 +20,8 @@ ) from .models import ( Config, + PrepareCategoryColumnMapping, + PrepareCategoryMappingsSettings, PrepareNonMotorizedDistanceSkimSettings, PrepareTimePeriodsSettings, PrepareVotBinsSettings, @@ -27,6 +29,115 @@ from .normalize_skimjoin import normalize_run_skimjoin_overrides, resolve_run_skimjoin_settings +_CATEGORY_MAPPING_TABLES = { + "households", + "persons", + "day", + "tours", + "trips", + "vehicles", + "joint_tour_participants", + "land_use", +} + + +def normalize_prepare_category_mappings( + raw_value, + *, + field_name: str, +) -> PrepareCategoryMappingsSettings: + """Normalize run-aware raw-to-canonical category mappings.""" + if raw_value in (None, {}): + return PrepareCategoryMappingsSettings() + if not isinstance(raw_value, dict): + raise ValueError(f"{field_name} must be a mapping when provided.") + + mappings: dict[ + str, dict[str, dict[str, PrepareCategoryColumnMapping]] + ] = {} + for raw_run_name, raw_tables in raw_value.items(): + run_name = normalize_run_selector_key(str(raw_run_name)) + if run_name in mappings: + raise ValueError( + f"{field_name} contains duplicate run key {raw_run_name!r} after normalization." + ) + if not isinstance(raw_tables, dict): + raise ValueError(f"{field_name}.{raw_run_name} must be a mapping.") + + table_mappings: dict[str, dict[str, PrepareCategoryColumnMapping]] = {} + for raw_table_name, raw_columns in raw_tables.items(): + table_name = str(raw_table_name).strip() + table_field = f"{field_name}.{raw_run_name}.{table_name}" + if table_name not in _CATEGORY_MAPPING_TABLES: + raise ValueError( + f"{table_field} uses unsupported table id {table_name!r}." + ) + if not isinstance(raw_columns, dict): + raise ValueError(f"{table_field} must be a mapping.") + + column_mappings: dict[str, PrepareCategoryColumnMapping] = {} + for raw_target, raw_spec in raw_columns.items(): + target = str(raw_target).strip() + column_field = f"{table_field}.{target}" + if not target: + raise ValueError(f"{table_field} contains an empty target column.") + if not isinstance(raw_spec, dict): + raise ValueError(f"{column_field} must be a mapping.") + + source_raw = raw_spec.get("source", target) + if not isinstance(source_raw, str) or not source_raw.strip(): + raise ValueError(f"{column_field}.source must be a non-empty string.") + source = source_raw.strip() + + output_type = str(raw_spec.get("output_type", "string")).strip().lower() + if output_type not in {"string", "boolean"}: + raise ValueError( + f"{column_field}.output_type must be 'string' or 'boolean'." + ) + preserve_unmapped = raw_spec.get("preserve_unmapped", True) + if not isinstance(preserve_unmapped, bool): + raise ValueError( + f"{column_field}.preserve_unmapped must be true or false." + ) + + raw_mapping = raw_spec.get("mapping") + if not isinstance(raw_mapping, dict) or not raw_mapping: + raise ValueError(f"{column_field}.mapping must be a non-empty mapping.") + if output_type == "boolean": + invalid_values = [ + value + for value in raw_mapping.values() + if value is not None and not isinstance(value, bool) + ] + if invalid_values: + raise ValueError( + f"{column_field}.mapping values must be true, false, or null " + "when output_type is 'boolean'." + ) + normalized_mapping = { + str(source_value): mapped_value + for source_value, mapped_value in raw_mapping.items() + } + else: + normalized_mapping = { + str(source_value): ( + None if mapped_value is None else str(mapped_value) + ) + for source_value, mapped_value in raw_mapping.items() + } + + column_mappings[target] = PrepareCategoryColumnMapping( + source_column=source, + mapping=normalized_mapping, + output_type=output_type, + preserve_unmapped=preserve_unmapped, + ) + table_mappings[table_name] = column_mappings + mappings[run_name] = table_mappings + + return PrepareCategoryMappingsSettings(mappings=mappings) + + def normalize_prepare_vot_bins( raw_value, *, diff --git a/runtime/config/normalize_skimjoin.py b/runtime/config/normalize_skimjoin.py index 7291ac4..b72d137 100644 --- a/runtime/config/normalize_skimjoin.py +++ b/runtime/config/normalize_skimjoin.py @@ -121,6 +121,9 @@ def normalize_run_skimjoin_overrides( f"{field_name}.create_hypothetical_skim_tables." ) + enabled = raw_value.get("enabled") + if enabled is not None and not isinstance(enabled, bool): + raise ValueError(f"{field_name}.enabled must be true or false when provided.") config_path = normalize_optional_path_string( raw_value.get("config_path"), field_name=f"{field_name}.config_path", @@ -156,6 +159,7 @@ def normalize_run_skimjoin_overrides( f"{field_name}.create_hypothetical_skim_tables must be true or false when provided." ) return RunSkimjoinOverrides( + enabled=enabled, config_path=config_path, skim_files=skim_files, network_los_file=network_los_file, @@ -301,6 +305,16 @@ def resolve_run_skimjoin_settings(config: Config, run_entry: dict[str, Any]) -> config_dir=Path(config.config_path).resolve().parent, ) + if overrides.enabled is False: + return replace( + config.skimjoin, + enabled=False, + config_digest=None, + normalized_config=None, + resolved_skim_files=(), + resolved_network_los_file=None, + ) + effective_config_path = overrides.config_path or config.skimjoin.config_path if effective_config_path is None: raise ValueError( diff --git a/runtime/config/schema.py b/runtime/config/schema.py index 9a3d764..2f41c1d 100644 --- a/runtime/config/schema.py +++ b/runtime/config/schema.py @@ -125,6 +125,7 @@ def validate_canonical_config(raw: Mapping[str, object]) -> None: field_name="dashboard", allowed={ "title", + "logo", "live", "export", "host", @@ -251,7 +252,7 @@ def validate_canonical_config(raw: Mapping[str, object]) -> None: _reject_unknown_keys( pipeline, field_name="pipeline", - allowed={"steps", "dashboard_mode", "overwrite"}, + allowed={"steps", "dashboard_mode", "refresh", "overwrite"}, ) display = _mapping(raw.get("display"), field_name="display") @@ -275,6 +276,7 @@ def validate_canonical_config(raw: Mapping[str, object]) -> None: "output", "validation", "distance_skim", + "category_mappings", "vot_bins", "time_periods", "non_motorized_distance_skim", diff --git a/runtime/config/sections.py b/runtime/config/sections.py index 32a5e69..7427cca 100644 --- a/runtime/config/sections.py +++ b/runtime/config/sections.py @@ -8,6 +8,7 @@ PIPELINE_STEP_ORDER = ("prepare", "skimjoin", "segment", "summarize", "dashboard") VALID_PIPELINE_STEPS = set(PIPELINE_STEP_ORDER) +REFRESHABLE_PIPELINE_STEPS = {"prepare", "skimjoin", "summarize"} VALID_DASHBOARD_MODES = {"none", "live", "export", "host"} @@ -60,9 +61,34 @@ def parse_pipeline(raw_value) -> PipelineSettings: "pipeline.dashboard_mode must be one of none, live, export, or host." ) - overwrite = raw_value.get("overwrite", False) - if not isinstance(overwrite, bool): - raise ValueError("pipeline.overwrite must be true or false when provided.") + if "overwrite" in raw_value: + raise ValueError( + "pipeline.overwrite has been replaced by pipeline.refresh. " + "Use refresh: all for a complete rebuild or refresh: [] for normal operation." + ) + + refresh_raw = raw_value.get("refresh", []) + if refresh_raw == "all": + refresh = [step for step in steps if step in REFRESHABLE_PIPELINE_STEPS] + elif isinstance(refresh_raw, list): + refresh = [] + for idx, raw_step in enumerate(refresh_raw): + if not isinstance(raw_step, str): + raise ValueError("pipeline.refresh entries must be strings.") + step = raw_step.strip() + if step not in REFRESHABLE_PIPELINE_STEPS: + raise ValueError( + f"pipeline.refresh[{idx}] must be one of prepare, skimjoin, or summarize." + ) + if step in refresh: + raise ValueError(f"pipeline.refresh contains duplicate step {step!r}.") + if step not in steps: + raise ValueError( + f"pipeline.refresh cannot include disabled step {step!r}." + ) + refresh.append(step) + else: + raise ValueError("pipeline.refresh must be a list or 'all' when provided.") if "skimjoin" in steps and "prepare" not in steps: raise ValueError("pipeline.steps cannot include 'skimjoin' without 'prepare'.") @@ -74,7 +100,7 @@ def parse_pipeline(raw_value) -> PipelineSettings: return PipelineSettings( steps=tuple(steps), dashboard_mode=dashboard_mode, - overwrite=overwrite, + refresh=tuple(refresh), ) diff --git a/runtime/config/sections_prepare.py b/runtime/config/sections_prepare.py index 2b10582..7b9b329 100644 --- a/runtime/config/sections_prepare.py +++ b/runtime/config/sections_prepare.py @@ -8,6 +8,7 @@ from .models import PrepareAutoSufficiencySettings from .normalize_categories import normalize_student_types from .normalize_prepare import ( + normalize_prepare_category_mappings, normalize_prepare_non_motorized_distance_skim, normalize_prepare_relationship_checks, normalize_prepare_time_periods, @@ -54,6 +55,10 @@ def parse_prepare(raw_value, *, config_dir: Path) -> ParsedPrepareSection: return ParsedPrepareSection( config_fields={ + "prepare_category_mappings": normalize_prepare_category_mappings( + prepare.get("category_mappings"), + field_name="prepare.category_mappings", + ), "prepare_vot_bins": normalize_prepare_vot_bins( prepare.get("vot_bins"), field_name="prepare.vot_bins" ), diff --git a/runtime/config/signatures.py b/runtime/config/signatures.py index d94e9a5..866ffa4 100644 --- a/runtime/config/signatures.py +++ b/runtime/config/signatures.py @@ -131,6 +131,26 @@ def prepare_signature_payload(config: Config) -> dict[str, Any]: "failure_policy": config.skimjoin.failure_policy, }, "prepare": { + "category_mappings": { + run_name: { + table_name: { + target_column: { + "source": spec.source_column, + "output_type": spec.output_type, + "preserve_unmapped": spec.preserve_unmapped, + "mapping": { + key: value + for key, value in sorted(spec.mapping.items()) + }, + } + for target_column, spec in sorted(column_mappings.items()) + } + for table_name, column_mappings in sorted(table_mappings.items()) + } + for run_name, table_mappings in sorted( + config.prepare_category_mappings.mappings.items() + ) + }, "auto_sufficiency": { "basis": config.prepare_auto_sufficiency.basis, }, @@ -179,47 +199,14 @@ def prepare_signature_payload(config: Config) -> dict[str, Any]: } +def base_prepare_signature_payload(config: Config) -> dict[str, Any]: + """Return preparation identity before optional skim enrichment.""" + payload = prepare_signature_payload(config) + payload.pop("skimjoin", None) + return payload + + def summary_signature_payload(config: Config) -> dict[str, Any]: - segmentation_payload: dict[str, Any] = {"enabled": config.segmentation.enabled} - if config.segmentation.enabled: - segmentation_payload["definitions"] = [ - { - "name": definition.name, - "include_full": definition.include_full, - "persist_segmented_prepared_tables": definition.persist_segmented_prepared_tables, - "allow_overlapping": definition.allow_overlapping, - "on_empty_segment": definition.on_empty_segment, - "source": ( - { - "type": "prepared_column", - "column": definition.source.column, - "source_table": definition.source.source_table, - } - if isinstance(definition.source, PreparedColumnSegmentationSource) - else { - "type": "csv_lookup", - "file": definition.source.file, - "join_source_table": definition.source.join_source_table, - "join_source_key_column": definition.source.join_source_key_column, - "csv_key_column": definition.source.csv_key_column, - "segment_value_column": definition.source.segment_value_column, - "lookup_rows": [ - {"key": key, "value": value} - for key, value in definition.source.lookup_rows - ], - } - ), - "segments": [ - { - "id": segment.id, - "label": segment.label, - "values": list(segment.values), - } - for segment in definition.segments - ], - } - for definition in config.segmentation.definitions - ] return { "weighting_modes": [ definition.signature_payload() @@ -276,13 +263,64 @@ def summary_signature_payload(config: Config) -> dict[str, Any]: "prepare": { "vot_bins": prepare_signature_payload(config)["prepare"]["vot_bins"], }, - "segmentation": segmentation_payload, + } + + +def segmentation_unit_signature_payload( + config: Config, + *, + segmentation_type: str, + segment_id: str, +) -> dict[str, Any]: + """Return the cache identity for one full or segmented analysis unit.""" + if segmentation_type == "full" and segment_id == "full": + return {"segmentation_type": "full", "segment_id": "full"} + + definition = config.segmentation.definition_by_name(segmentation_type) + if definition is None or definition.source is None: + return {"segmentation_type": segmentation_type, "segment_id": segment_id} + segment = next( + (candidate for candidate in definition.segments if candidate.id == segment_id), + None, + ) + if segment is None: + return {"segmentation_type": segmentation_type, "segment_id": segment_id} + + source = definition.source + if isinstance(source, PreparedColumnSegmentationSource): + source_payload: dict[str, Any] = { + "type": "prepared_column", + "column": source.column, + "source_table": source.source_table, + } + else: + segment_values = set(segment.values) + source_payload = { + "type": "csv_lookup", + "file": source.file, + "join_source_table": source.join_source_table, + "join_source_key_column": source.join_source_key_column, + "csv_key_column": source.csv_key_column, + "segment_value_column": source.segment_value_column, + "lookup_rows": [ + {"key": key, "value": value} + for key, value in source.lookup_rows + if value in segment_values + ], + } + return { + "segmentation_type": segmentation_type, + "segment_id": segment.id, + "segment_label": segment.label, + "segment_values": list(segment.values), + "source": source_payload, } def presentation_signature_payload(config: Config) -> dict[str, Any]: return { "dashboard_title": config.dashboard_title, + "dashboard_logo": config.dashboard_logo, "log_level": config.log_level, "dashboard_pages": ( [ diff --git a/runtime/run_lock.py b/runtime/run_lock.py new file mode 100644 index 0000000..947d310 --- /dev/null +++ b/runtime/run_lock.py @@ -0,0 +1,60 @@ +"""Cross-process lock for one visualizer output root.""" + +from __future__ import annotations + +import os +from pathlib import Path +from typing import BinaryIO + + +class RunLockError(RuntimeError): + """Raised when another process owns a visualizer output-root lock.""" + + +def acquire_run_lock(output_root: str | Path) -> tuple[Path, BinaryIO]: + lock_path = Path(output_root) / ".activitysim_visualizer.lock" + lock_path.parent.mkdir(parents=True, exist_ok=True) + stream = lock_path.open("a+b") + if lock_path.stat().st_size == 0: + stream.write(b"\0") + stream.flush() + stream.seek(0) + + try: + if os.name == "nt": + import msvcrt + + msvcrt.locking(stream.fileno(), msvcrt.LK_NBLCK, 1) + else: + import fcntl + + fcntl.flock(stream.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError as exc: + stream.close() + raise RunLockError( + f"another visualizer process is already using output root " + f"{lock_path.parent}. Lock file: {lock_path}" + ) from exc + + return lock_path, stream + + +def release_run_lock(lock: tuple[Path, BinaryIO] | None) -> None: + if lock is None: + return + _, stream = lock + try: + stream.seek(0) + if os.name == "nt": + import msvcrt + + msvcrt.locking(stream.fileno(), msvcrt.LK_UNLCK, 1) + else: + import fcntl + + fcntl.flock(stream.fileno(), fcntl.LOCK_UN) + finally: + stream.close() + + +__all__ = ["RunLockError", "acquire_run_lock", "release_run_lock"] diff --git a/runtime/workflows/artifacts.py b/runtime/workflows/artifacts.py index c963e41..ba6d082 100644 --- a/runtime/workflows/artifacts.py +++ b/runtime/workflows/artifacts.py @@ -15,7 +15,7 @@ class WorkflowPlan: logical_steps: tuple[str, ...] runtime_steps: tuple[str, ...] dashboard_mode: str = "none" - overwrite: bool = False + refresh_steps: tuple[str, ...] = () @classmethod def from_config(cls, config: Any) -> "WorkflowPlan": @@ -36,12 +36,15 @@ def for_steps(cls, config: Any, steps: Any) -> "WorkflowPlan": logical_steps=logical_steps, runtime_steps=tuple(runtime_steps), dashboard_mode=str(config.pipeline.dashboard_mode).lower(), - overwrite=bool(config.pipeline.overwrite), + refresh_steps=tuple(config.pipeline.refresh), ) def includes(self, step: str) -> bool: return step in self.logical_steps + def refreshes(self, step: str) -> bool: + return step in self.refresh_steps + @dataclass class PreparedRunsArtifact: @@ -66,5 +69,8 @@ class SummaryCacheInspection: """Reusable cached summaries and the table ids that still need rebuilding.""" runs: tuple[Any, ...] = () - reusable_summary_ids: tuple[str, ...] = () - stale_summary_ids: tuple[str, ...] = () + reusable_summary_ids_by_unit: dict[str, tuple[str, ...]] = field( + default_factory=dict + ) + stale_summary_ids_by_unit: dict[str, tuple[str, ...]] = field(default_factory=dict) + obsolete_unit_keys: tuple[str, ...] = () diff --git a/runtime/workflows/common.py b/runtime/workflows/common.py index 09f490f..31b39bd 100644 --- a/runtime/workflows/common.py +++ b/runtime/workflows/common.py @@ -6,13 +6,18 @@ from typing import Any from runtime.logging import get_logger -from processor.cache_identity import build_run_fingerprint, build_run_keys +from processor.cache_infra import read_manifest +from processor.cache_identity import ( + build_run_fingerprint, + build_run_keys, + optional_file_identity, +) from processor.models import ( PreparedTableName, prune_prepared_runs, ) from processor.prepare.cache import build_prepared_manifest_identity, prepared_root -from processor.prepare.reader import resolve_skim_path +from processor.prepare.reader import resolve_run_file_paths, resolve_skim_path from processor.summarize import cache as summary_cache from processor.summarize import builder as summary_builder from processor.summarize import cache_types as summary_types @@ -90,6 +95,7 @@ def load_summary_runs_from_cache( explicit_cache_dirs: list[str] | None, run_entries: list[dict] | None, required_summary_ids: list[str] | tuple[str, ...] | None = None, + optional_summary_ids: list[str] | tuple[str, ...] = (), ) -> list[Any]: """Load validated summary caches for dashboard or export workflows.""" from processor.summarize.external import ( @@ -110,6 +116,7 @@ def load_summary_runs_from_cache( if required_summary_ids is None else list(required_summary_ids) ) + optional_summary_ids = list(optional_summary_ids) if not cache_dirs and not run_entries: raise ValueError("no summary cache directories were found to load.") @@ -134,10 +141,11 @@ def load_summary_runs_from_cache( config=config, source_run_dir=entry.get("dir") or None, ) - ids_for_cache = [ + mapped_summary_ids = set(summary_table_map or {}) + required_ids_for_cache = [ summary_id for summary_id in required_summary_ids - if summary_id not in set(summary_table_map or {}) + if summary_id not in mapped_summary_ids ] expectations = ( shared.summary_cache_load_expectations( @@ -146,12 +154,31 @@ def load_summary_runs_from_cache( config=config, build_run_fingerprint_fn=build_run_fingerprint, resolve_skim_path_fn=resolve_skim_path, + resolve_run_file_paths_fn=resolve_run_file_paths, + optional_file_identity_fn=optional_file_identity, build_prepared_manifest_identity_fn=build_prepared_manifest_identity, ) or {} ) loaded_cache_runs: list[Any] = [] try: + optional_ids_for_cache: list[str] = [] + if optional_summary_ids and cache_dir.exists(): + manifest = read_manifest( + cache_dir, + error_cls=summary_types.SummaryCacheError, + ) + available_summary_ids = { + str(summary_id) + for summary_id in manifest.get("summary_ids", []) + } + optional_ids_for_cache = [ + summary_id + for summary_id in optional_summary_ids + if summary_id in available_summary_ids + and summary_id not in mapped_summary_ids + ] + ids_for_cache = [*required_ids_for_cache, *optional_ids_for_cache] if ids_for_cache or not external_summary_run: loaded_cache_runs = summary_cache.load_summary_run_bundle( cache_dir, diff --git a/runtime/workflows/prepare.py b/runtime/workflows/prepare.py index 46f7dd6..d2bd61e 100644 --- a/runtime/workflows/prepare.py +++ b/runtime/workflows/prepare.py @@ -6,8 +6,8 @@ from typing import Any, Callable from runtime.logging import get_logger -from processor.cache_identity import build_run_fingerprint -from processor.models import PreparedTableName, RunData +from processor.cache_identity import build_run_fingerprint, optional_file_identity +from processor.models import PreparedTableName, RunData, map_run_data_tables from processor.prepare.availability import ( failed_tables, has_usable_loaded_tables, @@ -21,7 +21,7 @@ write_prepared_run_cache, ) from processor.prepare.enrichment.pipeline import prepare_data -from processor.prepare.reader import read_run, resolve_skim_path +from processor.prepare.reader import read_run, resolve_run_file_paths, resolve_skim_path from processor.prepare.validation import ( PreparedRelationshipValidationError, validate_prepared_relationships, @@ -41,6 +41,11 @@ def prepared_cache_dir(prepared_root: Path, run_key: str) -> Path: return prepared_root / run_key / "prepared_tables" +def base_prepared_cache_dir(prepared_root: Path, run_key: str) -> Path: + """Return the pre-skim preparation cache directory for one run.""" + return prepared_root / run_key / "base_prepared_tables" + + def _run_cache_metadata( *, entry: dict, @@ -48,14 +53,21 @@ def _run_cache_metadata( config: Config, ) -> dict[str, object]: """Return the stable cache metadata for one resolved run entry.""" - return shared.run_cache_metadata( + metadata = shared.run_cache_metadata( entry=entry, run_key=run_key, config=config, resolve_skim_path_fn=resolve_skim_path, + resolve_run_file_paths_fn=resolve_run_file_paths, + optional_file_identity_fn=optional_file_identity, build_run_fingerprint_fn=build_run_fingerprint, build_prepared_manifest_identity_fn=build_prepared_manifest_identity, ) + metadata["base_run_fingerprint"] = { + **dict(metadata["run_fingerprint"]), + "skimjoin": None, + } + return metadata def _log_prepare_table_diagnostics(run_label: str, prepared_run: RunData) -> None: @@ -139,20 +151,26 @@ def _load_prepared_run_from_cache( run_key: str, label: str, run_fingerprint: dict[str, object], + prepare_config_digest: str | None = None, + stage: str = "prepare", ) -> tuple[str, RunData] | None: """Load one prepared run from cache when valid and usable.""" try: prepared_run = load_prepared_run_cache( prepared_dir, config, - expected_prepare_config_digest=config.prepare_config_digest, + expected_prepare_config_digest=( + prepare_config_digest or config.prepare_config_digest + ), expected_run_fingerprint=run_fingerprint, expected_label=label, expected_run_key=run_key, ) - LOGGER.info("Loaded prepared cache for run: %r", label) + LOGGER.info("Pipeline decision for %r / %s: REUSE", label, stage) except PreparedCacheError as exc: - LOGGER.info("Prepared cache miss for %r: %s", label, exc) + LOGGER.info( + "Pipeline decision for %r / %s: REBUILD — %s", label, stage, exc + ) return None if not has_usable_loaded_tables(prepared_run): @@ -177,11 +195,16 @@ def _build_prepared_run( metadata: dict[str, object], write_cache: bool, run_skimjoin: bool, + cache_name: str = "prepared_tables", + prepare_config_digest: str | None = None, + run_fingerprint: dict[str, object] | None = None, ) -> tuple[str, RunData] | None: """Read, prepare, skimjoin, and optionally cache one run.""" label = str(metadata["label"]) run_dir = str(metadata["run_dir"]) - run_fingerprint = dict(metadata["run_fingerprint"]) + resolved_run_fingerprint = dict( + run_fingerprint or metadata["run_fingerprint"] + ) prepared_table_map = entry.get("prepared_table_map") or None run_config = ( config if prepared_table_map is not None else config_for_run(config, entry) @@ -205,6 +228,11 @@ def _build_prepared_run( LOGGER.info("Prepared run: %r", label) return (label, prepared_run) + LOGGER.info( + "Pipeline decision for %r / %s: REBUILD", + label, + "prepare" if not run_skimjoin else "skimjoin", + ) LOGGER.info("Reading run %r from %s", label, run_dir) prepared_run = read_run( run_dir, @@ -215,6 +243,7 @@ def _build_prepared_run( hh_weight_col=entry.get("hh_weight_col") or None, person_weight_col=entry.get("person_weight_col") or None, trip_weight_col=entry.get("trip_weight_col") or None, + day_weight_col=entry.get("day_weight_col", "day_weight"), ) prepared_run = prepare_data(prepared_run, run_config) if run_skimjoin: @@ -235,8 +264,10 @@ def _build_prepared_run( run_config, run_key=run_key, output_root=prepared_root, - run_fingerprint=run_fingerprint, + run_fingerprint=resolved_run_fingerprint, file_format=config.prepare_output_file_format, + cache_name=cache_name, + prepare_config_digest=prepare_config_digest, ) LOGGER.info("Wrote prepared cache for run: %r", label) else: @@ -251,7 +282,8 @@ def _resolve_prepared_run( config: Config, prepared_root: Path, existing_prepared_runs_by_key: dict[str, tuple[str, RunData]], - prefer_cache: bool, + prefer_base_cache: bool, + prefer_skimjoin_cache: bool, write_cache: bool, run_skimjoin: bool, ) -> tuple[str, RunData] | None: @@ -282,7 +314,20 @@ def _resolve_prepared_run( existing_prepared_runs_by_key[run_key] = loaded_custom_prepared_run return loaded_custom_prepared_run - if prefer_cache: + if run_skimjoin and prefer_skimjoin_cache: + cached_prepared_run = _load_prepared_run_from_cache( + prepared_dir=prepared_dir, + config=config, + run_key=run_key, + label=label, + run_fingerprint=run_fingerprint, + stage="skimjoin", + ) + if cached_prepared_run is not None: + existing_prepared_runs_by_key[run_key] = cached_prepared_run + return cached_prepared_run + + if not run_skimjoin and prefer_base_cache: cached_prepared_run = _load_prepared_run_from_cache( prepared_dir=prepared_dir, config=config, @@ -294,6 +339,56 @@ def _resolve_prepared_run( existing_prepared_runs_by_key[run_key] = cached_prepared_run return cached_prepared_run + if run_skimjoin: + base_fingerprint = dict(metadata["base_run_fingerprint"]) + base_dir = base_prepared_cache_dir(prepared_root, run_key) + base_prepared_run = None + if prefer_base_cache: + base_prepared_run = _load_prepared_run_from_cache( + prepared_dir=base_dir, + config=config, + run_key=run_key, + label=label, + run_fingerprint=base_fingerprint, + prepare_config_digest=config.base_prepare_config_digest, + stage="prepare", + ) + if base_prepared_run is None: + base_prepared_run = _build_prepared_run( + entry=entry, + config=config, + run_key=run_key, + prepared_root=prepared_root, + metadata=metadata, + write_cache=write_cache, + run_skimjoin=False, + cache_name="base_prepared_tables", + prepare_config_digest=config.base_prepare_config_digest, + run_fingerprint=base_fingerprint, + ) + if base_prepared_run is None: + return None + + run_config = config_for_run(config, entry) + skimjoined_run = map_run_data_tables(base_prepared_run[1], lambda _name, frame: frame) + LOGGER.info("Pipeline decision for %r / skimjoin: REBUILD", label) + skimjoined_run = apply_skimjoin(skimjoined_run, run_config) + _log_prepare_table_diagnostics(label, skimjoined_run) + _validate_prepared_run(label, skimjoined_run, run_config) + if write_cache: + write_prepared_run_cache( + skimjoined_run, + run_config, + run_key=run_key, + output_root=prepared_root, + run_fingerprint=run_fingerprint, + file_format=config.prepare_output_file_format, + ) + LOGGER.info("Wrote skimjoin cache for run: %r", label) + result = (label, skimjoined_run) + existing_prepared_runs_by_key[run_key] = result + return result + rebuilt_prepared_run = _build_prepared_run( entry=entry, config=config, @@ -301,7 +396,7 @@ def _resolve_prepared_run( prepared_root=prepared_root, metadata=metadata, write_cache=write_cache, - run_skimjoin=run_skimjoin, + run_skimjoin=False, ) if rebuilt_prepared_run is None: return None @@ -338,6 +433,14 @@ def run_prepare_workflow( plan=plan, ) prepared_root = prepared_root or prepared_cache_root(config, create=write_cache) + prefer_base_cache = prefer_cache + prefer_skimjoin_cache = prefer_cache + if plan.refreshes("skimjoin") and not plan.refreshes("prepare"): + prefer_base_cache = True + prefer_skimjoin_cache = False + if plan.refreshes("prepare"): + prefer_base_cache = False + prefer_skimjoin_cache = False ( existing_prepared_runs_by_key, prepared_runs_by_key, @@ -357,6 +460,10 @@ def run_prepare_workflow( label, ) continue + run_skimjoin = bool( + not entry.get("prepared_table_map") + and config_for_run(config, entry).skimjoin_step_enabled() + ) metadata = _run_cache_metadata(entry=entry, run_key=run_key, config=config) prepared_loaded = _resolve_prepared_run( entry=entry, @@ -364,9 +471,10 @@ def run_prepare_workflow( config=config, prepared_root=prepared_root, existing_prepared_runs_by_key=existing_prepared_runs_by_key, - prefer_cache=prefer_cache, + prefer_base_cache=prefer_base_cache, + prefer_skimjoin_cache=prefer_skimjoin_cache, write_cache=write_cache, - run_skimjoin=config.skimjoin_step_enabled(), + run_skimjoin=run_skimjoin, ) if prepared_loaded is None: continue diff --git a/runtime/workflows/shared.py b/runtime/workflows/shared.py index 8f1c694..52700a7 100644 --- a/runtime/workflows/shared.py +++ b/runtime/workflows/shared.py @@ -83,6 +83,9 @@ def effective_processor_config( ) if effective is config: return config + effective.base_prepare_config_digest = digest_payload( + effective.base_prepare_signature_payload() + ) effective.prepare_config_digest = digest_payload(effective.prepare_signature_payload()) effective.summary_config_digest = digest_payload(effective.summary_signature_payload()) return effective @@ -123,6 +126,8 @@ def summary_cache_load_expectations( config: Any, build_run_fingerprint_fn: Callable[..., dict[str, object]], resolve_skim_path_fn: Callable[[str | None, str | None, str | Path], str | None], + resolve_run_file_paths_fn: Callable[..., dict[str, str | None]], + optional_file_identity_fn: Callable[[str | Path | None], dict[str, object] | None], build_prepared_manifest_identity_fn: Callable[..., dict[str, object]], ) -> dict[str, object] | None: """Return cache-load expectations for a cache dir when raw run inputs exist.""" @@ -146,7 +151,18 @@ def summary_cache_load_expectations( "config_path": resolved_skimjoin.config_path, "config_digest": resolved_skimjoin.config_digest, "resolved_skim_files": list(resolved_skimjoin.resolved_skim_files), + "resolved_skim_file_identities": [ + optional_file_identity_fn(path) + for path in resolved_skimjoin.resolved_skim_files + ], "resolved_network_los_file": resolved_skimjoin.resolved_network_los_file, + "resolved_network_los_identity": optional_file_identity_fn( + resolved_skimjoin.resolved_network_los_file + ), + "create_hypothetical_skim_tables": ( + resolved_skimjoin.create_hypothetical_skim_tables + ), + "failure_policy": resolved_skimjoin.failure_policy, } base_run_fingerprint = build_run_fingerprint_fn( label=expected_label, @@ -173,6 +189,26 @@ def summary_cache_load_expectations( else config.fallback_files or None ), skimjoin=expected_skimjoin, + raw_file_identities=( + None + if (uses_custom_prepared_tables or uses_summary_table_map_only) + else { + table_id: identity + for table_id, path in resolve_run_file_paths_fn( + run_dir, + config, + entry.get("file_map") or None, + ).items() + if (identity := optional_file_identity_fn(path)) is not None + } + ), + skim_file_identity=( + None + if (uses_custom_prepared_tables or uses_summary_table_map_only) + else optional_file_identity_fn( + resolve_skim_path_fn(entry.get("skim_file") or None, config.skim_file, run_dir) + ) + ), hh_weight_col=None if (uses_custom_prepared_tables or uses_summary_table_map_only) else entry.get("hh_weight_col") or None, @@ -182,6 +218,9 @@ def summary_cache_load_expectations( trip_weight_col=None if (uses_custom_prepared_tables or uses_summary_table_map_only) else entry.get("trip_weight_col") or None, + day_weight_col=None + if (uses_custom_prepared_tables or uses_summary_table_map_only) + else entry.get("day_weight_col", "day_weight"), ) expected_run_fingerprint = summary_run_fingerprint( base_run_fingerprint, @@ -320,6 +359,8 @@ def run_cache_metadata( run_key: str, config: Any, resolve_skim_path_fn: Callable[[str | None, str | None, str | Path], str | None], + resolve_run_file_paths_fn: Callable[..., dict[str, str | None]], + optional_file_identity_fn: Callable[[str | Path | None], dict[str, object] | None], build_run_fingerprint_fn: Callable[..., dict[str, object]], build_prepared_manifest_identity_fn: Callable[..., dict[str, object]], ) -> dict[str, object]: @@ -340,7 +381,18 @@ def run_cache_metadata( "config_path": resolved_skimjoin.config_path, "config_digest": resolved_skimjoin.config_digest, "resolved_skim_files": list(resolved_skimjoin.resolved_skim_files), + "resolved_skim_file_identities": [ + optional_file_identity_fn(path) + for path in resolved_skimjoin.resolved_skim_files + ], "resolved_network_los_file": resolved_skimjoin.resolved_network_los_file, + "resolved_network_los_identity": optional_file_identity_fn( + resolved_skimjoin.resolved_network_los_file + ), + "create_hypothetical_skim_tables": ( + resolved_skimjoin.create_hypothetical_skim_tables + ), + "failure_policy": resolved_skimjoin.failure_policy, } resolved_skim = ( None @@ -355,6 +407,24 @@ def run_cache_metadata( else run_dir ), skim_file=resolved_skim, + raw_file_identities=( + None + if (uses_custom_prepared_tables or uses_summary_table_map_only) + else { + table_id: identity + for table_id, path in resolve_run_file_paths_fn( + run_dir, + config, + entry.get("file_map") or None, + ).items() + if (identity := optional_file_identity_fn(path)) is not None + } + ), + skim_file_identity=( + None + if (uses_custom_prepared_tables or uses_summary_table_map_only) + else optional_file_identity_fn(resolved_skim) + ), skimjoin=resolved_skimjoin_payload, file_map=None if (uses_custom_prepared_tables or uses_summary_table_map_only) @@ -373,6 +443,9 @@ def run_cache_metadata( trip_weight_col=None if (uses_custom_prepared_tables or uses_summary_table_map_only) else entry.get("trip_weight_col") or None, + day_weight_col=None + if (uses_custom_prepared_tables or uses_summary_table_map_only) + else entry.get("day_weight_col", "day_weight"), ) prepared_manifest_identity = None if not uses_summary_table_map_only: diff --git a/runtime/workflows/summarize.py b/runtime/workflows/summarize.py index 06a2ba8..8f966c6 100644 --- a/runtime/workflows/summarize.py +++ b/runtime/workflows/summarize.py @@ -59,6 +59,7 @@ def _load_summary_run_from_cache( run_key: str, run_fingerprint: dict[str, object], prepared_manifest_identity: dict[str, object], + analysis_units: list[AnalysisUnit] | None = None, ) -> SummaryCacheInspection | None: """Load one summary run from cache when valid.""" try: @@ -72,33 +73,65 @@ def _load_summary_run_from_cache( expected_prepared_manifest_identity=prepared_manifest_identity, expected_label=label, expected_run_key=run_key, + expected_analysis_units=analysis_units, ) - reusable_summary_ids = list(inspection["reusable_summary_ids"]) - stale_summary_ids = list(inspection["stale_summary_ids"]) + reusable_by_unit = { + str(unit_key): tuple(summary_ids) + for unit_key, summary_ids in dict( + inspection["reusable_summary_ids_by_unit"] + ).items() + } + stale_by_unit = { + str(unit_key): tuple(summary_ids) + for unit_key, summary_ids in dict( + inspection["stale_summary_ids_by_unit"] + ).items() + } + obsolete_unit_keys = tuple(inspection["obsolete_unit_keys"]) cached_runs = ( summary_cache.load_summary_run_bundle( cache_dir, config, expected_modes=config.weighting_modes, - expected_summary_ids=reusable_summary_ids, - expected_summary_config_digest=config.summary_config_digest, + expected_summary_ids_by_unit={ + unit_key: list(summary_ids) + for unit_key, summary_ids in reusable_by_unit.items() + if summary_ids + }, + # Per-summary digests were validated by the inspection above. + # Requiring the bundle-wide digest here would discard otherwise + # reusable tables whenever only one builder changed. + expected_summary_config_digest=None, expected_run_fingerprint=run_fingerprint, expected_prepared_manifest_identity=prepared_manifest_identity, expected_label=label, expected_run_key=run_key, ) - if reusable_summary_ids + if any(reusable_by_unit.values()) else [] ) + stale_count = sum(len(summary_ids) for summary_ids in stale_by_unit.values()) + reusable_count = sum( + len(summary_ids) for summary_ids in reusable_by_unit.values() + ) LOGGER.info( - "Loaded reusable summary cache tables for run %r: %s", + "Pipeline decision for %r / summarize: %s (%s)", label, - ", ".join(reusable_summary_ids) if reusable_summary_ids else "(none)", + "REUSE" if not stale_count and not obsolete_unit_keys else "REBUILD", + ( + "all analysis-unit summary tables reusable" + if not stale_count and not obsolete_unit_keys + else ( + f"{stale_count} stale; {reusable_count} reusable; " + f"{len(obsolete_unit_keys)} obsolete analysis units" + ) + ), ) return SummaryCacheInspection( runs=tuple(cached_runs), - reusable_summary_ids=tuple(reusable_summary_ids), - stale_summary_ids=tuple(stale_summary_ids), + reusable_summary_ids_by_unit=reusable_by_unit, + stale_summary_ids_by_unit=stale_by_unit, + obsolete_unit_keys=obsolete_unit_keys, ) except summary_types.SummaryCacheError as exc: LOGGER.info("Cache miss for %r: %s", label, exc) @@ -167,9 +200,8 @@ def _merge_summary_runs( *, cached_runs: list[Any], rebuilt_runs: list[Any], + analysis_units: list[AnalysisUnit], ) -> list[Any]: - if not cached_runs: - return rebuilt_runs cached_by_segment = { (run.segmentation_type, run.segment_id): run for run in cached_runs } @@ -177,9 +209,14 @@ def _merge_summary_runs( (run.segmentation_type, run.segment_id): run for run in rebuilt_runs } merged: list[Any] = [] - for segment_key in rebuilt_by_segment: - rebuilt = rebuilt_by_segment[segment_key] + for unit in analysis_units: + segment_key = (unit.segmentation_type, unit.segment_id) + rebuilt = rebuilt_by_segment.get(segment_key) cached = cached_by_segment.get(segment_key) + if rebuilt is None: + if cached is not None: + merged.append(cached) + continue if cached is None: merged.append(rebuilt) continue @@ -286,6 +323,30 @@ def run_summary_workflow( run_keys.append(run_key) run_fingerprints_by_key[run_key] = run_fingerprint cached_run = None + analysis_units: list[AnalysisUnit] | None = None + prepared_loaded: tuple[str, RunData] | None = None + has_buildable_inputs = bool(entry.get("dir") or entry.get("prepared_table_map")) + + if config.segmentation.enabled and has_buildable_inputs: + prepare_artifact = run_prepare_workflow( + config=config, + prepared_root=prepared_root, + run_entries=[entry], + prefer_cache=prepared_prefer_cache, + write_cache=True, + existing=prepare_artifact, + plan=plan, + ) + if run_key in prepare_artifact.by_key: + prepared_loaded = prepare_artifact.by_key[run_key] + existing_prepared_runs_by_key = dict(prepare_artifact.by_key) + prepared_runs_by_key[run_key] = prepared_loaded + analysis_units = build_analysis_units_for_run( + run_key=run_key, + run_name=label, + prepared_run=prepared_loaded[1], + config=config, + ) if prefer_cache: cached_run = _load_summary_run_from_cache( @@ -295,10 +356,13 @@ def run_summary_workflow( run_key=run_key, run_fingerprint=run_fingerprint, prepared_manifest_identity=prepared_manifest_identity, + analysis_units=analysis_units, ) if cached_run is not None: - stale_summary_ids = list(cached_run.stale_summary_ids) - if not stale_summary_ids: + if ( + not any(cached_run.stale_summary_ids_by_unit.values()) + and not cached_run.obsolete_unit_keys + ): summary_runs.extend( merge_summary_table_map_run( list(cached_run.runs), @@ -309,18 +373,13 @@ def run_summary_workflow( if cached_prepared_run is not None: prepared_runs_by_key[run_key] = cached_prepared_run continue + else: + LOGGER.info( + "Pipeline decision for %r / summarize: REBUILD — refresh requested or cache reuse disabled", + label, + ) cached_summary_runs = list(cached_run.runs) if cached_run else [] - summary_ids_to_build = list(summary_builder.DEFAULT_SUMMARY_IDS) - if cached_run is not None: - summary_ids_to_build = list(cached_run.stale_summary_ids) - summary_ids_to_build = [ - summary_id - for summary_id in summary_ids_to_build - if summary_id not in external_summary_ids - ] - - has_buildable_inputs = bool(entry.get("dir") or entry.get("prepared_table_map")) if not has_buildable_inputs: run_summary_runs = merge_summary_table_map_run( cached_summary_runs, @@ -346,16 +405,19 @@ def run_summary_workflow( ) continue - prepare_artifact = run_prepare_workflow( - config=config, - prepared_root=prepared_root, - run_entries=[entry], - prefer_cache=prepared_prefer_cache, - write_cache=True, - existing=prepare_artifact, - plan=plan, - ) - if run_key not in prepare_artifact.by_key: + if prepared_loaded is None: + prepare_artifact = run_prepare_workflow( + config=config, + prepared_root=prepared_root, + run_entries=[entry], + prefer_cache=prepared_prefer_cache, + write_cache=True, + existing=prepare_artifact, + plan=plan, + ) + if run_key in prepare_artifact.by_key: + prepared_loaded = prepare_artifact.by_key[run_key] + if prepared_loaded is None: run_summary_runs = merge_summary_table_map_run( cached_summary_runs, external_summary_run, @@ -379,19 +441,35 @@ def run_summary_workflow( label, ) continue - prepared_loaded = prepare_artifact.by_key[run_key] existing_prepared_runs_by_key = dict(prepare_artifact.by_key) prepared_runs_by_key[run_key] = prepared_loaded - analysis_units = build_analysis_units_for_run( - run_key=run_key, - run_name=label, - prepared_run=prepared_loaded[1], - config=config, - ) + if analysis_units is None: + analysis_units = build_analysis_units_for_run( + run_key=run_key, + run_name=label, + prepared_run=prepared_loaded[1], + config=config, + ) run_summary_runs = [] - if summary_ids_to_build: - for unit in analysis_units: + for unit in analysis_units: + unit_key = summary_cache.analysis_unit_key( + segmentation_type=unit.segmentation_type, + segment_id=unit.segment_id, + ) + summary_ids_to_build = ( + list(summary_builder.DEFAULT_SUMMARY_IDS) + if cached_run is None + else list( + cached_run.stale_summary_ids_by_unit.get(unit_key, ()) + ) + ) + summary_ids_to_build = [ + summary_id + for summary_id in summary_ids_to_build + if summary_id not in external_summary_ids + ] + if summary_ids_to_build: summaries_by_mode, summary_metadata_by_mode = _build_summary_tables_for_run( prepared_run=unit.prepared_run, config=config, @@ -418,13 +496,12 @@ def run_summary_workflow( source_run_dir=str(unit.prepared_run.run_dir), ) ) - if cached_summary_runs and run_summary_runs: + if cached_summary_runs or run_summary_runs: run_summary_runs = _merge_summary_runs( cached_runs=cached_summary_runs, rebuilt_runs=run_summary_runs, + analysis_units=analysis_units, ) - elif cached_summary_runs: - run_summary_runs = cached_summary_runs run_summary_runs = merge_summary_table_map_run( run_summary_runs, external_summary_run, diff --git a/scripts/generate_validation_demo_fixtures.py b/scripts/generate_validation_demo_fixtures.py new file mode 100644 index 0000000..de802b3 --- /dev/null +++ b/scripts/generate_validation_demo_fixtures.py @@ -0,0 +1,289 @@ +"""Generate deterministic estimated tables for validation-page demonstrations.""" + +from __future__ import annotations + +import csv +from pathlib import Path + + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +SOURCE_DIR = PROJECT_ROOT / "outside_summary_tables" +OUTPUT_DIR = SOURCE_DIR / "estimated_fixtures" +RUNS = ("unfiltered", "filtered", "override", "estimation-output") +RUN_BIASES = (-0.08, -0.03, 0.03, 0.08) +PERIOD_COLUMNS = ("am_vol", "md_vol", "pm_vol", "day_vol") + + +def _read_rows(path: Path) -> list[dict[str, str]]: + with path.open(newline="", encoding="utf-8-sig") as stream: + return list(csv.DictReader(stream)) + + +def _write_rows(path: Path, fieldnames: list[str], rows: list[dict[str, object]]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", newline="", encoding="utf-8") as stream: + writer = csv.DictWriter(stream, fieldnames=fieldnames) + writer.writeheader() + writer.writerows(rows) + + +def _estimated_value(value: str | float, run_index: int, row_index: int, column_index: int) -> float: + variation = (((row_index + 3) * (column_index + 5) * (run_index + 2)) % 17 - 8) / 100 + return max(0.0, float(value) * (1.0 + RUN_BIASES[run_index] + variation)) + + +def _write_count_locations(run: str, run_index: int) -> None: + source = _read_rows(SOURCE_DIR / "countLocCounts.csv") + rows: list[dict[str, object]] = [] + for row_index, row in enumerate(source): + output: dict[str, object] = {"id": row["id"], "FACTYPE": row["FACTYPE"]} + for column_index, column in enumerate(PERIOD_COLUMNS): + output[column] = _estimated_value( + row[column], run_index, row_index, column_index + ) + rows.append(output) + _write_rows( + OUTPUT_DIR / run / "count_location_volumes_validation_summary.csv", + ["id", "FACTYPE", *PERIOD_COLUMNS], + rows, + ) + + +def _write_links(run: str, run_index: int) -> None: + source = _read_rows(SOURCE_DIR / "allLinkSummary.csv")[:2000] + rows: list[dict[str, object]] = [] + for row_index, row in enumerate(source): + output: dict[str, object] = { + "id": row["id"], + "From_Node": row["From_Node"], + "To_Node": row["To_Node"], + "FACTYPE": row["FACTYPE"], + } + for column_index, column in enumerate(PERIOD_COLUMNS): + output[column] = _estimated_value( + row[column], run_index, row_index, column_index + ) + rows.append(output) + _write_rows( + OUTPUT_DIR / run / "link_validation_summary.csv", + ["id", "From_Node", "To_Node", "FACTYPE", *PERIOD_COLUMNS], + rows, + ) + + +def _write_screenlines(run: str, run_index: int) -> None: + counts = _read_rows(SOURCE_DIR / "countLocCounts.csv")[:16] + rows: list[dict[str, object]] = [] + for row_index, row in enumerate(counts): + for column_index, (period, column) in enumerate( + zip(("AM", "MD", "PM", "Day"), PERIOD_COLUMNS) + ): + observed = float(row[column]) + rows.append( + { + "screenline_id": f"SL-{row_index + 1:02d}", + "direction": "NB/EB" if row_index % 2 == 0 else "SB/WB", + "count_period": period, + "facility_type": row["FACTYPE"], + "observed_volume": observed, + "modeled_volume": _estimated_value( + observed, run_index, row_index, column_index + ), + } + ) + _write_rows( + OUTPUT_DIR / run / "screenline_flow_comparisons.csv", + [ + "screenline_id", + "direction", + "count_period", + "facility_type", + "observed_volume", + "modeled_volume", + ], + rows, + ) + + +def _write_commuting_flows(run: str, run_index: int) -> None: + rows: list[dict[str, object]] = [] + for source_name, geography_type in ( + ("countyFlows.csv", "district"), + ("countyFlows_JoJa.csv", "county"), + ): + for row_index, row in enumerate(_read_rows(SOURCE_DIR / source_name)): + origin = row.get("") or row.get("Origin") + if origin is None or origin.strip().lower() == "total": + continue + for column_index, (destination, value) in enumerate(row.items()): + if destination in {"", "Origin", "Total"}: + continue + rows.append( + { + "origin_geography_type": geography_type, + "origin_geography_id": origin, + "destination_geography_type": geography_type, + "destination_geography_id": destination, + "commuter_count": _estimated_value( + value, run_index, row_index, column_index + ), + } + ) + _write_rows( + OUTPUT_DIR / run / "commuting_flows.csv", + [ + "origin_geography_type", + "origin_geography_id", + "destination_geography_type", + "destination_geography_id", + "commuter_count", + ], + rows, + ) + + +def _write_wide_summary( + run: str, + run_index: int, + *, + source_name: str, + output_name: str, + category_column: str, + value_columns: list[str], +) -> None: + source = _read_rows(SOURCE_DIR / source_name) + rows: list[dict[str, object]] = [] + for row_index, row in enumerate(source): + output: dict[str, object] = {category_column: row[category_column]} + for column_index, column in enumerate(value_columns): + output[column] = _estimated_value( + row[column], run_index, row_index, column_index + ) + output["Total"] = sum(float(output[column]) for column in value_columns) + rows.append(output) + _write_rows( + OUTPUT_DIR / run / output_name, + [category_column, *value_columns, "Total"], + rows, + ) + + +def _write_observed_series() -> None: + observed_dir = OUTPUT_DIR / "observed" + _write_rows( + observed_dir / "transit_boardings_by_operator_and_technology.csv", + ["operator", "technology", "boardings"], + [ + {"operator": "City Transit", "technology": "Bus", "boardings": 18400.0}, + {"operator": "Regional Transit", "technology": "Bus", "boardings": 9200.0}, + {"operator": "Regional Transit", "technology": "Rail", "boardings": 6100.0}, + ], + ) + _write_rows( + observed_dir / "transit_transfer_rate.csv", + ["operator", "technology", "access_mode", "transfer_rate"], + [ + {"operator": "City Transit", "technology": "Bus", "access_mode": "Walk", "transfer_rate": 1.21}, + {"operator": "Regional Transit", "technology": "Bus", "access_mode": "Walk", "transfer_rate": 1.34}, + {"operator": "Regional Transit", "technology": "Rail", "access_mode": "PNR", "transfer_rate": 1.47}, + ], + ) + _write_rows( + observed_dir / "bicycle_vmt_by_facility_type.csv", + ["facility_type", "bicycle_vmt"], + [ + {"facility_type": "Protected Bike Lane", "bicycle_vmt": 12400.0}, + {"facility_type": "Bike Lane", "bicycle_vmt": 18750.0}, + {"facility_type": "Shared Roadway", "bicycle_vmt": 9300.0}, + {"facility_type": "Multi-Use Path", "bicycle_vmt": 15600.0}, + ], + ) + + +def _write_estimated_series(run: str, run_index: int) -> None: + observed_dir = OUTPUT_DIR / "observed" + table_specs = ( + ( + "transit_boardings_by_operator_and_technology.csv", + ["operator", "technology", "boardings"], + ["boardings"], + ), + ( + "transit_transfer_rate.csv", + ["operator", "technology", "access_mode", "transfer_rate"], + ["transfer_rate"], + ), + ( + "bicycle_vmt_by_facility_type.csv", + ["facility_type", "bicycle_vmt"], + ["bicycle_vmt"], + ), + ) + for filename, fieldnames, value_columns in table_specs: + source = _read_rows(observed_dir / filename) + rows: list[dict[str, object]] = [] + for row_index, row in enumerate(source): + output: dict[str, object] = dict(row) + for column_index, column in enumerate(value_columns): + output[column] = _estimated_value( + row[column], run_index, row_index, column_index + ) + rows.append(output) + _write_rows(OUTPUT_DIR / run / filename, fieldnames, rows) + + +def generate() -> None: + _write_observed_series() + for run_index, run in enumerate(RUNS): + _write_count_locations(run, run_index) + _write_links(run, run_index) + _write_screenlines(run, run_index) + _write_commuting_flows(run, run_index) + _write_estimated_series(run, run_index) + _write_wide_summary( + run, + run_index, + source_name="cvm_summary.csv", + output_name="commercial_vehicle_validation_summary.csv", + category_column="tod", + value_columns=["car", "mu", "su"], + ) + _write_wide_summary( + run, + run_index, + source_name="cvm_vmt_summary.csv", + output_name="commercial_vehicle_vmt_validation_summary.csv", + category_column="tod", + value_columns=["car", "mu", "su"], + ) + external_columns = [ + "hbcoll", + "hbo", + "hbr", + "hbs", + "hbsch", + "hbw", + "nhbnw", + "nhbw", + "truck", + ] + _write_wide_summary( + run, + run_index, + source_name="ext_summary.csv", + output_name="external_trip_validation_summary.csv", + category_column="tod", + value_columns=external_columns, + ) + _write_wide_summary( + run, + run_index, + source_name="ext_vmt_summary.csv", + output_name="external_vmt_validation_summary.csv", + category_column="tod", + value_columns=external_columns, + ) + + +if __name__ == "__main__": + generate() diff --git a/scripts/generate_wiki_catalogs.py b/scripts/generate_wiki_catalogs.py index 0bccbce..fad6b61 100644 --- a/scripts/generate_wiki_catalogs.py +++ b/scripts/generate_wiki_catalogs.py @@ -1,4 +1,4 @@ -"""Generate drift-prone wiki catalog sections from runtime registries.""" +"""Generate drift-prone documentation catalogs from runtime registries.""" from __future__ import annotations @@ -6,9 +6,16 @@ import sys from typing import Iterable +import yaml + ROOT = Path(__file__).resolve().parents[1] WIKI = ROOT / "wiki" +REFERENCE = ROOT / "reference" +SUMMARY_CATALOG_METADATA = Path(__file__).with_name("summary_catalog_metadata.yaml") +PROCESSOR_OUTPUT_CATALOG_METADATA = Path(__file__).with_name( + "processor_output_catalog_metadata.yaml" +) if str(ROOT) not in sys.path: sys.path.insert(0, str(ROOT)) @@ -46,56 +53,453 @@ def _type_name(dtype: object) -> str: return str(dtype).replace("Datetime(", "Datetime(") +def _summary_catalog_metadata() -> dict[str, object]: + return yaml.safe_load(SUMMARY_CATALOG_METADATA.read_text(encoding="utf-8")) + + +def _processor_output_catalog_metadata() -> dict[str, object]: + return yaml.safe_load( + PROCESSOR_OUTPUT_CATALOG_METADATA.read_text(encoding="utf-8") + ) + + +def _output_fields(definition, metadata: dict[str, object]) -> str: + return "
    ".join( + f"`{_escape_cell(name or chr(34) * 2)}` " + f"(`{_escape_cell(_type_name(dtype))}`): " + f"{_escape_cell(metadata['fields'][name])}" + for name, dtype in definition.contract.schema.items() + ) or "-" + + +def _required_inputs(definition, input_metadata: dict[str, object]) -> str: + contract = definition.contract + table_names = list(contract.required_tables) + table_names.extend( + table_name + for table_name in contract.required_columns + if table_name not in table_names + ) + if not table_names: + return "None declared." + + rendered_tables: list[str] = [] + for table_name in table_names: + table_metadata = input_metadata[table_name] + parts = [ + f"`{_escape_cell(table_name)}`: " + f"{_escape_cell(table_metadata['description'])}" + ] + for field_name in contract.required_columns.get(table_name, ()): + field_metadata = table_metadata["fields"][field_name] + parts.append( + f"• `{_escape_cell(field_name)}` " + f"(`{_escape_cell(field_metadata['type'])}`): " + f"{_escape_cell(field_metadata['description'])}" + ) + rendered_tables.append("
    ".join(parts)) + return "

    ".join(rendered_tables) + + +def _summary_function(definition) -> str: + builder_name = ( + f"{definition.builder.__module__}." + f"{getattr(definition.builder, '__name__', '')}" + ) + default = "yes" if definition.build_by_default else "no" + return f"`{_escape_cell(builder_name)}`
    Default build: **{default}**" + + +def _category_dashboard_pages(summary_ids: set[str]) -> list[str]: + from dashboard.page_registry import all_page_definitions + + titles: list[str] = [] + for page in all_page_definitions(): + page_summary_ids = { + *page.required_summary_ids, + *page.optional_summary_ids, + } + if summary_ids.intersection(page_summary_ids) and page.title not in titles: + titles.append(page.title) + return titles + + +def _sidecar_fields(schema, descriptions: dict[str, str]) -> str: + return "
    ".join( + f"`{_escape_cell(name)}` (`{_escape_cell(_type_name(dtype))}`): " + f"{_escape_cell(descriptions[name])}" + for name, dtype in schema.items() + ) + + def build_summary_catalog() -> str: from processor.summarize.catalog import SUMMARY_DEFINITIONS + catalog_metadata = _summary_catalog_metadata() + definitions = { + definition.summary_id: definition for definition in SUMMARY_DEFINITIONS + } + default_count = sum( + definition.build_by_default for definition in SUMMARY_DEFINITIONS + ) lines: list[str] = [ - "_Generated from `processor.summarize.catalog.SUMMARY_DEFINITIONS`._", + "_Generated from `processor.summarize.catalog.SUMMARY_DEFINITIONS` and " + "`scripts/summary_catalog_metadata.yaml`._", "", - f"Total registered summaries: **{len(SUMMARY_DEFINITIONS)}**", + f"Registered summaries: **{len(SUMMARY_DEFINITIONS)}**; " + f"built by default: **{default_count}**.", "", - "| Summary ID | Filename | Builder | Output schema | Required inputs |", - "|---|---|---|---|---|", ] - for definition in sorted( - SUMMARY_DEFINITIONS, key=lambda item: item.summary_id - ): - contract = definition.contract - builder_name = ( - f"{definition.builder.__module__}." - f"{getattr(definition.builder, '__name__', '')}" + for category in catalog_metadata["categories"]: + lines.extend([f"### {category['name']}", ""]) + dashboard_pages = _category_dashboard_pages( + {summary["id"] for summary in category["summaries"]} + ) + if dashboard_pages: + pages = ", ".join(_escape_cell(title) for title in dashboard_pages) + lines.extend( + [ + "_Related dashboard pages: " + f"{pages}. See the " + "[Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._", + "", + ] + ) + lines.extend( + [ + "| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function |", + "|---|---|---|---|---|", + ] + ) + for metadata in category["summaries"]: + definition = definitions[metadata["id"]] + lines.append( + "| " + + " | ".join( + [ + f"`{_escape_cell(definition.summary_id)}`
    " + f"`{_escape_cell(definition.filename)}.csv`", + _escape_cell(metadata["description"]), + _required_inputs( + definition, + catalog_metadata["required_inputs"], + ), + _output_fields(definition, metadata), + _summary_function(definition), + ] + ) + + " |" + ) + lines.append("") + + return "\n".join(lines) + + +def build_processor_output_reference() -> str: + from processor.summarize.catalog import SUMMARY_DEFINITIONS + from processor.skimjoin.hypothetical_sidecars import ( + TOUR_HYPOTHETICAL_SIDECAR_SCHEMA, + TRIP_HYPOTHETICAL_SIDECAR_SCHEMA, + ) + + catalog_metadata = _summary_catalog_metadata() + output_metadata = _processor_output_catalog_metadata() + definitions = { + definition.summary_id: definition for definition in SUMMARY_DEFINITIONS + } + external_category = next( + category + for category in catalog_metadata["categories"] + if category["name"] == "Externally Supplied Validation Contracts" + ) + processor_categories = [ + category + for category in catalog_metadata["categories"] + if category is not external_category + ] + processor_summary_ids = { + summary["id"] + for category in processor_categories + for summary in category["summaries"] + } + default_count = sum( + definition.build_by_default + for definition in SUMMARY_DEFINITIONS + if definition.summary_id in processor_summary_ids + ) + lines: list[str] = [ + "_Generated from the ActivitySim Visualizer processor contracts and " + "analytical metadata._", + "", + "## 1. Prepared Output Tables", + "", + "Prepared caches contain eight canonical tables in the configured " + "Parquet or CSV format. Each file is named " + "`.`. Prepare preserves the source columns " + "described earlier in the Model Outputs documentation; the field tables " + "below list the stable canonical and processor-derived additions. " + "Configuration-specific category mappings and skim outputs can add " + "further columns.", + "", + ] + for output in output_metadata["prepared_outputs"]: + shared_fields = catalog_metadata["required_inputs"].get( + output["runtime_name"], {} + ).get("fields", {}) + fields = {**shared_fields, **output["fields"]} + lines.extend( + [ + f"### `{_escape_cell(output['table_id'])}` Prepared Table", + "", + f"_Processor runtime name: " + f"`{_escape_cell(output['runtime_name'])}`._", + "", + f"**Information and analytical use.** " + f"{_escape_cell(output['description'])}", + "", + "Retained source fields keep their source meaning and generally " + "keep their source type.", + "", + "| Field | Type | Description |", + "|---|---|---|", + ] + ) + for name, field_metadata in fields.items(): + lines.append( + f"| `{_escape_cell(name)}` | " + f"`{_escape_cell(field_metadata['type'])}` | " + f"{_escape_cell(field_metadata['description'])} |" + ) + lines.append("") + lines.extend( + [ + "", + "## 2. Summary Output Tables", + "", + f"The processor calculates **{len(processor_summary_ids)}** summary " + f"contracts: **{default_count}** in the standard workflow and " + f"**{len(processor_summary_ids) - default_count}** optional skim ECDF " + "tables. The same schemas apply to weighted, unweighted, and " + "segmented cache variants.", + "", + "Every summary table is saved as a CSV file named " + "`.csv`.", + "", + ] + ) + for category in processor_categories: + lines.extend( + [ + f"### {category['name']}", + "", + "| Summary Table | Information and Analytical Use | Fields |", + "|---|---|---|", + ] ) - schema = "
    ".join( - f"`{_escape_cell(name)}: {_escape_cell(_type_name(dtype))}`" - for name, dtype in contract.schema.items() - ) or "-" - required_parts: list[str] = [] - if contract.required_tables: - required_parts.append("tables: " + _inline_list(contract.required_tables)) - for table_name, columns in sorted(contract.required_columns.items()): - required_parts.append( - f"{_escape_cell(table_name)}: " + _inline_list(columns) + for metadata in category["summaries"]: + definition = definitions[metadata["id"]] + lines.append( + "| " + + " | ".join( + [ + f"`{_escape_cell(definition.summary_id)}`", + _escape_cell(metadata["description"]), + _output_fields(definition, metadata), + ] + ) + + " |" ) - required = "
    ".join(required_parts) if required_parts else "-" + lines.append("") + sidecar_schemas = { + "trip_hypothetical_skims": TRIP_HYPOTHETICAL_SIDECAR_SCHEMA, + "tour_hypothetical_skims": TOUR_HYPOTHETICAL_SIDECAR_SCHEMA, + } + lines.extend( + [ + "## 3. Hypothetical All-Modes Skim Tables", + "", + "These optional long-form tables evaluate skim components for all " + "configured modes, not only the observed mode. Each file uses the " + "prepared-cache format and is named " + "`.`. Operational skimjoin QA " + "report CSVs are outside the scope of this reference.", + "", + "| Hypothetical Skim Table | Information and Analytical Use | Fields |", + "|---|---|---|", + ] + ) + for sidecar in output_metadata["conditional_sidecars"]: + lines.append( + "| " + + " | ".join( + [ + f"`{_escape_cell(sidecar['attribute'])}`", + _escape_cell(sidecar["description"]), + _sidecar_fields( + sidecar_schemas[sidecar["attribute"]], + sidecar["fields"], + ), + ] + ) + + " |" + ) + + lines.extend( + [ + "", + "## 4. Externally Supplied Table Schemas", + "", + "**These tables are not created by the processor.** They are " + "registered schemas for validation files supplied through " + "`summary_table_map`. A mapped file can be CSV or Parquet and its " + "configured filename does not need to match the schema name.", + "", + "| Externally Supplied Table Schema | Information and Analytical Use | Fields |", + "|---|---|---|", + ] + ) + for metadata in external_category["summaries"]: + definition = definitions[metadata["id"]] lines.append( "| " + " | ".join( [ f"`{_escape_cell(definition.summary_id)}`", - f"`{_escape_cell(definition.filename)}.csv`", - f"`{_escape_cell(builder_name)}`", - schema, - required, + _escape_cell(metadata["description"]), + _output_fields(definition, metadata), ] ) + " |" ) + lines.append("") return "\n".join(lines) +def _validate_summary_reference() -> None: + """Keep the analytical metadata aligned with runtime declarations.""" + from processor.summarize.catalog import SUMMARY_DEFINITIONS + + definitions = { + definition.summary_id: definition for definition in SUMMARY_DEFINITIONS + } + catalog_metadata = _summary_catalog_metadata() + metadata = [ + summary + for category in catalog_metadata["categories"] + for summary in category["summaries"] + ] + metadata_ids = [summary["id"] for summary in metadata] + problems: list[str] = [] + duplicate_ids = sorted( + summary_id + for summary_id in set(metadata_ids) + if metadata_ids.count(summary_id) > 1 + ) + if duplicate_ids: + problems.append("duplicate summaries: " + ", ".join(duplicate_ids)) + + missing_ids = sorted(set(definitions) - set(metadata_ids)) + extra_ids = sorted(set(metadata_ids) - set(definitions)) + if missing_ids: + problems.append("missing summaries: " + ", ".join(missing_ids)) + if extra_ids: + problems.append("unknown summaries: " + ", ".join(extra_ids)) + + for summary in metadata: + definition = definitions.get(summary["id"]) + if definition is None: + continue + documented_fields = tuple(summary["fields"]) + declared_fields = tuple(definition.contract.schema) + if documented_fields != declared_fields: + problems.append(f"{summary['id']} fields do not match its schema") + + contract = definition.contract + required_tables = { + *contract.required_tables, + *contract.required_columns, + } + for table_name in sorted(required_tables): + table_metadata = catalog_metadata["required_inputs"].get(table_name) + if table_metadata is None: + problems.append(f"{summary['id']} missing input table {table_name}") + continue + for field_name in contract.required_columns.get(table_name, ()): + if field_name not in table_metadata["fields"]: + problems.append( + f"{summary['id']} missing input field " + f"{table_name}.{field_name}" + ) + + if problems: + raise ValueError( + "Summary analytical reference is incomplete: " + "; ".join(problems) + ) + + +def _validate_processor_output_reference() -> None: + """Keep prepared and sidecar metadata aligned with runtime cache contracts.""" + from processor.prepare.cache import PREPARED_TABLE_ATTRS, SIDECAR_TABLE_ATTRS + from processor.skimjoin.hypothetical_sidecars import ( + TOUR_HYPOTHETICAL_SIDECAR_SCHEMA, + TRIP_HYPOTHETICAL_SIDECAR_SCHEMA, + ) + + metadata = _processor_output_catalog_metadata() + shared_input_metadata = _summary_catalog_metadata()["required_inputs"] + prepared = { + output["runtime_name"]: output for output in metadata["prepared_outputs"] + } + declared_prepared = { + attr_name: (table_id, filename) + for attr_name, table_id, filename in PREPARED_TABLE_ATTRS + } + problems: list[str] = [] + if set(prepared) != set(declared_prepared): + problems.append("prepared table inventory does not match the cache contract") + for runtime_name, (table_id, filename) in declared_prepared.items(): + output = prepared.get(runtime_name) + if output is None: + continue + if output["table_id"] != table_id or output["filename"] != filename: + problems.append(f"{runtime_name} prepared table identity does not match") + shared_fields = shared_input_metadata.get(runtime_name, {}).get("fields", {}) + duplicated_fields = set(shared_fields).intersection(output["fields"]) + if duplicated_fields: + problems.append( + f"{runtime_name} duplicates shared field metadata: " + + ", ".join(sorted(duplicated_fields)) + ) + + sidecars = { + output["attribute"]: output + for output in metadata["conditional_sidecars"] + } + declared_sidecars = dict(SIDECAR_TABLE_ATTRS) + if set(sidecars) != set(declared_sidecars): + problems.append("sidecar table inventory does not match the cache contract") + schemas = { + "trip_hypothetical_skims": TRIP_HYPOTHETICAL_SIDECAR_SCHEMA, + "tour_hypothetical_skims": TOUR_HYPOTHETICAL_SIDECAR_SCHEMA, + } + for attribute, filename in declared_sidecars.items(): + output = sidecars.get(attribute) + if output is None: + continue + if output["filename"] != filename: + problems.append(f"{attribute} sidecar filename does not match") + if tuple(output["fields"]) != tuple(schemas[attribute]): + problems.append(f"{attribute} fields do not match its schema") + + if problems: + raise ValueError( + "Processor output reference is incomplete: " + "; ".join(problems) + ) + + def build_dashboard_page_catalog() -> str: from dashboard.page_registry import all_group_definitions, all_page_definitions @@ -166,8 +570,10 @@ def build_dashboard_page_catalog() -> str: def main() -> None: + _validate_summary_reference() + _validate_processor_output_reference() _replace_generated_section( - WIKI / "24-summary-catalog.md", + WIKI / "26-summary-catalog.md", marker="SUMMARY-CATALOG", generated=build_summary_catalog(), ) @@ -176,6 +582,11 @@ def main() -> None: marker="DASHBOARD-PAGE-CATALOG", generated=build_dashboard_page_catalog(), ) + _replace_generated_section( + REFERENCE / "processor-output-table-reference.md", + marker="PROCESSOR-OUTPUT-REFERENCE", + generated=build_processor_output_reference(), + ) if __name__ == "__main__": diff --git a/scripts/processor_output_catalog_metadata.yaml b/scripts/processor_output_catalog_metadata.yaml new file mode 100644 index 0000000..07fa2fe --- /dev/null +++ b/scripts/processor_output_catalog_metadata.yaml @@ -0,0 +1,398 @@ +prepared_outputs: + - runtime_name: hh + table_id: households + filename: households + description: >- + One row per household. This core prepared table preserves the ActivitySim + household record and adds stable weights, household-size and vehicle-count + aliases, auto-sufficiency inputs, and home-geography fields. Use it for + household controls, demographic profiles, vehicle availability, and + residential geography analysis. The cache file is always present, but it + contains an empty sentinel when the source table is unavailable or empty. + fields: + home_zone_id: + type: source-compatible identifier + description: Household home-zone identifier, normally a MAZ when MAZ processing is enabled. + WORKERS: + type: Int32 + description: Number of workers copied from the configured household worker-count field. + LICENSEDDRIVERS: + type: Int32 + description: Licensed household members derived from person records when license data are available. + ADULTS: + type: Int32 + description: Number of adults copied from the configured household adult-count field. + home_taz: + type: source-compatible identifier + description: TAZ corresponding to home_zone_id; copied directly in TAZ models or looked up from land use in MAZ models. + HGEO: + type: String + description: Legacy configured home-geography label used by compatible summaries. + home_geo__: + type: String + description: Home-geography identifier for each configured named geography aggregation. + + - runtime_name: per + table_id: persons + filename: persons + description: >- + One row per person. This core table preserves ActivitySim person attributes + and choices and supplies canonical identifiers, worker/student indicators, + home/work/school geography, mandatory-distance measures, and analysis + weights. Use it for population segmentation, long-term choice analysis, + accessibility, and person-level rates. The cache file is always present, + with an empty sentinel when the source is unavailable or empty. + fields: + is_university: + type: Boolean-like source value + description: Prepared university-student indicator selected from the configured source alias. + adult: + type: String + description: Prepared adult-status value used when deriving household auto-sufficiency inputs. + mandatory_tour_frequency: + type: String + description: Mandatory-tour-frequency choice used directly by summaries when available. + imf_choice: + type: integer + description: Legacy numeric mandatory-tour-frequency code derived from named ActivitySim alternatives. + num_joint_tours: + type: Int32 + description: Number of unique joint tours linked to the person through the joint-participant table. + home_taz: + type: source-compatible identifier + description: TAZ corresponding to the person's home zone. + work_taz: + type: source-compatible identifier + description: TAZ corresponding to the person's internal workplace zone. + school_taz: + type: source-compatible identifier + description: TAZ corresponding to the person's school zone. + HGEO: + type: String + description: Legacy configured home-geography label. + WGEO: + type: String + description: Legacy configured workplace-geography label. + home_geo__: + type: String + description: Home-geography identifier for each configured named geography aggregation. + work_geo__: + type: String + description: Workplace-geography identifier for each configured named geography aggregation. + school_geo__: + type: String + description: School-geography identifier for each configured named geography aggregation. + + - runtime_name: day + table_id: day + filename: day + description: >- + Optional person-day or household-day records retained from multi-day model + or survey output. Use this table for daily activity-pattern and frequency + analysis where a person can contribute more than one observed day. An + empty sentinel is written when no day table is available. + fields: + day_id: + type: Int64 + description: Canonical day-record identifier when supplied. + household_id: + type: Int64 + description: Household identifier for the observed day when supplied. + person_id: + type: Int64 + description: Person identifier for the observed day when supplied. + person_type: + type: String + description: Person type retained from the source or joined from the prepared person table. + travel_date: + type: String + description: Source travel-date label when available. + day_num: + type: Int32 + description: Sequential day number within the person's or household's observation period. + travel_dow: + type: Int32 + description: Source day-of-week code when available. + daily_activity_pattern: + type: String + description: Daily activity-pattern alternative, such as mandatory, nonmandatory, or home. + day_weight: + type: source-compatible number + description: Configured source day weight retained for provenance when available. + finalweight: + type: Float64 + description: Day expansion weight from day, person, or household weights, with a 1.0 fallback. + + - runtime_name: tours + table_id: tours + filename: tours + description: >- + One row per source tour representation. This core table preserves + ActivitySim tour choices and adds standardized purpose, household context, + stop counts, joint-party size, distance, time-period, and geography fields. + Use it for tour frequency, mode, destination, scheduling, stop-pattern, + joint-travel, and vehicle-allocation analysis. Some ActivitySim outputs + repeat joint tours by participant; summary descriptions state when those + rows are fractionally adjusted. The cache file is always present. + fields: + person_type: + type: source-compatible category + description: Person type copied from a configured tour-table alias when present. + summary_tour_purpose: + type: String + description: Purpose normalized to the configured summary-purpose categories. + origin: + type: source-compatible identifier + description: Tour origin zone; home-based and at-work origins are repaired from household or parent-tour context when possible. + start_hour: + type: Int32 + description: Tour start period, using the source period system or the survey adapter's 48-period conversion. + end_hour: + type: Int32 + description: Tour end period in the same period system as start_hour. + start_period: + type: String + description: Configured named network period corresponding to the tour start period. + end_period: + type: String + description: Configured named network period corresponding to the tour end period. + first_inbound_trip_depart: + type: Int32 + description: Departure period of the first inbound trip on the tour when it can be derived. + first_inbound_trip_period: + type: String + description: Named network period of the first inbound trip. + tourdur: + type: Int32 + description: Inclusive tour duration, end_hour minus start_hour plus one, clipped to 1 through 48 periods. + stop_frequency: + type: source-compatible category + description: ActivitySim stop-frequency choice retained from the configured source field. + composition: + type: String + description: Joint-party composition retained from the source or derived as adults, children, or mixed from participant attributes. + income_segment: + type: Int64 + description: Household income segment copied to the tour when available. + o_maz: + type: Int64 + description: Tour-origin MAZ copied from origin when MAZ processing is enabled. + d_maz: + type: Int64 + description: Tour-destination MAZ copied from destination when MAZ processing is enabled. + OTAZ: + type: Int32 + description: Origin TAZ copied or looked up from the origin MAZ. + DTAZ: + type: Int32 + description: Destination TAZ copied or looked up from the destination MAZ. + pnr_zone_id: + type: Int64 + description: Park-and-ride lot or zone selected for a drive-transit tour when supplied. + pnr_taz: + type: Int32 + description: TAZ corresponding to pnr_zone_id. + origin_geo__: + type: String + description: Origin-geography identifier for each configured named geography aggregation. + destination_geo__: + type: String + description: Destination-geography identifier for each configured named geography aggregation. + tour_distance: + type: Float64 + description: Supplied tour distance with missing values filled from SKIMDIST when both fields are available. + vot_bin: + type: String + description: Configured value-of-time category used to select segmented skims. + : + type: normally Float64 + description: Skimjoin result whose name, units, direction, and mode applicability are defined by the normalized skimjoin configuration. + + - runtime_name: trips + table_id: trips + filename: trips + description: >- + One row per source trip representation. This core table preserves + ActivitySim trip choices and adds parent-tour context, canonical endpoints, + distances, periods, stop indicators, escort-event positions, geography, + and weights. Use it for trip and stop rates, mode and purpose analysis, + parking demand, time-of-day profiles, VMT, and travel-impedance analysis. + Joint records may represent several travelers; summary descriptions state + how participant expansion is applied. The cache file is always present. + fields: + household_id: + type: Int64 + description: Canonical household identifier associated with the trip. + trip_id: + type: Int64 + description: Canonical trip identifier. + summary_tour_purpose: + type: String + description: Parent-tour purpose normalized to configured summary-purpose categories. + tour_category: + type: String + description: Category of the parent tour, filled from the prepared tour table when necessary. + origin: + type: source-compatible identifier + description: Trip origin zone; tour-boundary endpoints are repaired from the prepared tour origin when possible. + destination: + type: source-compatible identifier + description: Trip destination zone; tour-boundary endpoints are repaired from the prepared tour origin when possible. + outbound: + type: Boolean-like source value + description: Direction flag indicating the outbound half of the tour. + inbound: + type: Int32 + description: Derived inverse direction indicator, 1 for inbound records and 0 otherwise. + trip_num: + type: Int32 + description: Sequential trip number within a tour direction. + max_trip_num: + type: integer + description: Largest trip number on the same tour half, used to identify intermediate stops. + depart_hour: + type: Int32 + description: Trip departure period; defaults to one only when no departure field is available. + trip_period: + type: String + description: Configured named network period corresponding to the trip departure period. + income_segment: + type: Int64 + description: Household income segment copied to the trip when available. + AUTOSUFF: + type: Int32 + description: Parent-household auto-sufficiency category under the configured comparison basis. + o_maz: + type: Int64 + description: Trip-origin MAZ copied from origin when MAZ processing is enabled. + d_maz: + type: Int64 + description: Trip-destination MAZ copied from destination when MAZ processing is enabled. + OTAZ: + type: Int32 + description: Origin TAZ copied or looked up from the origin MAZ. + DTAZ: + type: Int32 + description: Destination TAZ copied or looked up from the destination MAZ. + pnr_zone_id: + type: Int64 + description: Park-and-ride zone inherited from the parent tour when supplied. + pnr_taz: + type: Int32 + description: TAZ corresponding to pnr_zone_id. + origin_parking_zone: + type: Int64 + description: Parking zone associated with the trip origin when supplied. + origin_geo__: + type: String + description: Origin-geography identifier for each configured named geography aggregation. + destination_geo__: + type: String + description: Destination-geography identifier for each configured named geography aggregation. + prepared_non_motorized_distance: + type: Float64 + description: Optional walk/bicycle/e-bike distance from the configured non-motorized skim source. + escort_event_trip_num: + type: Int32 + description: Trip number at which the matched escort event occurs. + escort_event_match_status: + type: String + description: Whether escort-event matching was matched, ambiguous, or unmatched. + vot_bin: + type: String + description: Configured value-of-time category used to select segmented skims. + : + type: normally Float64 + description: Skimjoin result whose name, units, lookup direction, and mode applicability are defined by the normalized skimjoin configuration. + + - runtime_name: vehicles + table_id: vehicles + filename: vehicles + description: >- + Optional one-row-per-household-vehicle table. It preserves ActivitySim + vehicle records and decodes the standard compound vehicle type into body, + age, and fuel attributes. Use it for fleet composition, vehicle technology, + energy, emissions, and vehicle-allocation analysis. An empty sentinel is + written when no vehicle table is available. + fields: + household_id: + type: Int64 + description: Canonical identifier of the household that owns the vehicle. + vehicle_id: + type: Int64 + description: Canonical vehicle identifier. + vehicle_num: + type: Int32 + description: Sequential vehicle number within the household. + vehicle_type: + type: String + description: Source vehicle-type choice, commonly encoded as body type, age, and fuel type separated by underscores. + + - runtime_name: joint_participants + table_id: joint_tour_participants + filename: joint_tour_participants + description: >- + Optional participation records linking people to fully joint tours. The + usual grain is one person-tour participation, although additional source + attributes are retained. Use it to determine joint-tour party size, + composition, and person participation without treating a household tour + as independent person tours. An empty sentinel is written when the source + table is unavailable. + fields: + tour_id: + type: Int64 + description: Canonical identifier of the joint tour. + + - runtime_name: land_use + table_id: land_use + filename: land_use + description: >- + Optional zone and land-use attribute table, normally one row per source + MAZ or TAZ before any enrollment overlays are added. It supports zone + translation, named geography joins, employment and enrollment controls, + parking capacity, and destination-based analysis. When configured student + markets are expanded, additional zone/student-type rows carry enrollment + counts. An empty sentinel is written when land use is unavailable. + fields: + TAZ: + type: Int64 + description: Canonical transportation-analysis-zone identifier. + EMPLOYMENT: + type: Float64 + description: Canonical total employment copied from the configured land-use employment field. + land_use_geo__: + type: String + description: Geography identifier for each configured named land-use aggregation. + +conditional_sidecars: + - attribute: trip_hypothetical_skims + filename: trip_hypothetical_skims + description: >- + Optional long-form trip skim values evaluated under configured modes other + than each trip's observed mode. One row represents a trip, hypothetical + mode, and skim component. Use it to compare travel impedance across modal + alternatives without duplicating the full prepared trip table. It is + written only when hypothetical skim tables are enabled and populated. + fields: + trip_id: Canonical identifier of the evaluated trip. + observed_mode: Mode selected in the source trip record. + hypothetical_mode: Alternative mode used for the skim lookup. + component: Configured skim output name, which determines the value's units and interpretation. + value: Skim value for the trip and hypothetical mode; null when the lookup is unresolved. + finalweight: Expansion weight inherited from the prepared trip record. + - attribute: tour_hypothetical_skims + filename: tour_hypothetical_skims + description: >- + Optional long-form tour skim values evaluated under configured modes other + than each tour's observed mode. One row represents a tour, hypothetical + mode, direction, and skim component. Use it to compare outbound, inbound, + and combined tour impedance across modal alternatives. It is written only + when hypothetical skim tables are enabled and populated. + fields: + tour_id: Canonical identifier of the evaluated tour. + observed_mode: Main mode selected in the source tour record. + hypothetical_mode: Alternative mode used for the skim lookup. + direction: Outbound or inbound for directional components; null for nondirectional components. + component: Configured skim output name, which determines the value's units and interpretation. + value: Skim value for the tour, direction, and hypothetical mode; null when unresolved. + finalweight: Expansion weight inherited from the prepared tour record. diff --git a/scripts/run_simor_scenarios.py b/scripts/run_simor_scenarios.py new file mode 100644 index 0000000..9918ff1 --- /dev/null +++ b/scripts/run_simor_scenarios.py @@ -0,0 +1,289 @@ +"""Build the SIMOR area dashboards in parallel, then build their comparison.""" + +from __future__ import annotations + +import argparse +from collections import deque +from dataclasses import dataclass +from datetime import datetime +from pathlib import Path +import subprocess +import sys +import time + + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +RUN_SCRIPT = PROJECT_ROOT / "run.py" +AREA_CONFIGS = ( + ("metro", PROJECT_ROOT / "simor_configs/metro_configs/metro_config.yaml"), + ("lcog", PROJECT_ROOT / "simor_configs/lcog_configs/lcog_config.yaml"), + ("skats", PROJECT_ROOT / "simor_configs/skats_configs/skats_config.yaml"), +) +COMPARISON_CONFIG = ( + "comparison", + PROJECT_ROOT + / "simor_configs/comparison_configs/estimation_mode_outputs_comparison.yaml", +) +DEFAULT_LOG_ROOT = PROJECT_ROOT / "simor_project_outputs/logs/scenario_runner" + + +@dataclass(frozen=True) +class ScenarioJob: + name: str + config_path: Path + runtime_log: Path + console_log: Path + command: tuple[str, ...] + + +def _positive_int(value: str) -> int: + parsed = int(value) + if parsed < 1: + raise argparse.ArgumentTypeError("must be at least 1") + return parsed + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=( + "Run Metro, LCOG, and SKATS dashboard exports concurrently, then " + "run the comparison dashboard after all three succeed." + ) + ) + parser.add_argument( + "--max-parallel", + type=_positive_int, + default=2, + help="Maximum concurrent area dashboards (default: 2).", + ) + parser.add_argument( + "--log-root", + type=Path, + default=DEFAULT_LOG_ROOT, + help="Parent directory for timestamped per-scenario logs.", + ) + parser.add_argument( + "--refresh-caches", + action="store_true", + help="Pass --refresh-caches to every dashboard build.", + ) + pipeline_mode = parser.add_mutually_exclusive_group() + pipeline_mode.add_argument( + "--exports-only", + action="store_true", + help="Pass --dashboard --export-html to every dashboard build.", + ) + pipeline_mode.add_argument( + "--no-skimjoin", + action="store_true", + help=( + "Run prepare, summarize, and HTML export while skipping the " + "skimjoin step." + ), + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="Print the staged commands without running them.", + ) + return parser.parse_args() + + +def _build_job( + *, + name: str, + config_path: Path, + run_log_dir: Path, + refresh_caches: bool, + exports_only: bool, + no_skimjoin: bool, +) -> ScenarioJob: + runtime_log = run_log_dir / f"{name}.runtime.log" + console_log = run_log_dir / f"{name}.console.log" + command = [ + sys.executable, + str(RUN_SCRIPT), + "--config", + str(config_path), + "--log-path", + str(runtime_log), + ] + if refresh_caches: + command.append("--refresh-caches") + if exports_only: + command.extend(("--dashboard", "--export-html")) + if no_skimjoin: + command.extend( + ("--prepare", "--summarize", "--dashboard", "--export-html") + ) + return ScenarioJob( + name=name, + config_path=config_path, + runtime_log=runtime_log, + console_log=console_log, + command=tuple(command), + ) + + +def _tail(path: Path, line_count: int = 20) -> str: + if not path.exists(): + return "" + with path.open("r", encoding="utf-8", errors="replace") as stream: + return "".join(deque(stream, maxlen=line_count)).rstrip() + + +def _stop_active(active: dict[subprocess.Popen, tuple[ScenarioJob, object, float]]) -> None: + for process in active: + if process.poll() is None: + process.terminate() + for process, (_, console_stream, _) in active.items(): + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + process.wait() + console_stream.close() + + +def _run_batch( + jobs: list[ScenarioJob], + *, + max_parallel: int, + poll_interval: float = 0.25, +) -> bool: + pending = deque(jobs) + active: dict[subprocess.Popen, tuple[ScenarioJob, object, float]] = {} + + try: + while pending or active: + while pending and len(active) < max_parallel: + job = pending.popleft() + job.console_log.parent.mkdir(parents=True, exist_ok=True) + console_stream = job.console_log.open( + "w", + encoding="utf-8", + newline="", + ) + try: + process = subprocess.Popen( + job.command, + cwd=PROJECT_ROOT, + stdout=console_stream, + stderr=subprocess.STDOUT, + ) + except Exception: + console_stream.close() + raise + active[process] = (job, console_stream, time.perf_counter()) + print( + f"[runner] START {job.name} (pid={process.pid})", + flush=True, + ) + + completed = [ + process for process in active if process.poll() is not None + ] + if not completed: + time.sleep(poll_interval) + continue + + for process in completed: + job, console_stream, started = active.pop(process) + console_stream.close() + elapsed = time.perf_counter() - started + if process.returncode == 0: + print( + f"[runner] PASS {job.name} ({elapsed / 60:.2f} min)", + flush=True, + ) + continue + + print( + f"[runner] FAIL {job.name} (exit={process.returncode}, " + f"{elapsed / 60:.2f} min)", + file=sys.stderr, + flush=True, + ) + detail = _tail(job.console_log) + if detail: + print(detail, file=sys.stderr, flush=True) + _stop_active(active) + return False + except KeyboardInterrupt: + print("\n[runner] Interrupted; stopping active dashboards.", file=sys.stderr) + _stop_active(active) + raise + + return True + + +def run_scenarios(args: argparse.Namespace) -> int: + configured = (*AREA_CONFIGS, COMPARISON_CONFIG) + missing_configs = [path for _, path in configured if not path.is_file()] + if missing_configs: + for path in missing_configs: + print(f"[runner] Missing config: {path}", file=sys.stderr) + return 2 + + timestamp = datetime.now().astimezone().strftime("%Y%m%d_%H%M%S_%f") + run_log_dir = args.log_root.resolve() / timestamp + area_jobs = [ + _build_job( + name=name, + config_path=config_path, + run_log_dir=run_log_dir, + refresh_caches=args.refresh_caches, + exports_only=args.exports_only, + no_skimjoin=args.no_skimjoin, + ) + for name, config_path in AREA_CONFIGS + ] + comparison_job = _build_job( + name=COMPARISON_CONFIG[0], + config_path=COMPARISON_CONFIG[1], + run_log_dir=run_log_dir, + refresh_caches=args.refresh_caches, + exports_only=args.exports_only, + no_skimjoin=args.no_skimjoin, + ) + + print( + f"[runner] Area concurrency: {min(args.max_parallel, len(area_jobs))}", + flush=True, + ) + print(f"[runner] Logs: {run_log_dir}", flush=True) + + if args.dry_run: + for job in (*area_jobs, comparison_job): + print( + f"[runner] {job.name}: {subprocess.list2cmdline(job.command)}", + flush=True, + ) + return 0 + + run_log_dir.mkdir(parents=True, exist_ok=False) + started = time.perf_counter() + if not _run_batch(area_jobs, max_parallel=args.max_parallel): + print( + "[runner] Comparison skipped because an area dashboard failed.", + file=sys.stderr, + ) + return 1 + + if not _run_batch([comparison_job], max_parallel=1): + return 1 + + print( + f"[runner] All dashboards completed in " + f"{(time.perf_counter() - started) / 60:.2f} minutes.", + flush=True, + ) + return 0 + + +def main() -> None: + raise SystemExit(run_scenarios(parse_args())) + + +if __name__ == "__main__": + main() diff --git a/scripts/summary_catalog_metadata.yaml b/scripts/summary_catalog_metadata.yaml new file mode 100644 index 0000000..5be54b6 --- /dev/null +++ b/scripts/summary_catalog_metadata.yaml @@ -0,0 +1,1010 @@ +required_inputs: + hh: + description: Prepared household table. + fields: + HHSIZE: + type: Int32 + description: Household person count clipped to 1 through 5, where 5 represents households of five or more people. + HHVEH: + type: Int32 + description: Household vehicle count clipped to 0 through 4 for standard summary categories. + av_ownership: + type: Boolean-like + description: Whether the household owns an autonomous vehicle. + finalweight: + type: Float64 + description: Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback. + household_id: + type: Int64 + description: Canonical household identifier used to join people, tours, trips, days, and vehicles. + joint_participants: + description: Prepared joint-tour participant table. + fields: + person_id: + type: Int64 + description: Canonical identifier of a person participating in a joint tour. + land_use: + description: Prepared land-use and zone-attribute table. + fields: + MAZ: + type: Int64 + description: Canonical micro-analysis-zone identifier. + employment_count: + type: Float64 + description: Employment opportunities associated with the zone. + enrollment_count: + type: Float64 + description: Enrollment opportunities associated with the zone and student type. + student_type: + type: String + description: Student or enrollment market segment. + per: + description: Prepared person table. + fields: + age: + type: integer + description: Person age in years. + bike_comfort: + type: category + description: Modeled bicycle-comfort alternative. + distance_to_school: + type: Float64 + description: Prepared home-to-school distance in the configured distance units. + distance_to_work: + type: Float64 + description: Prepared home-to-work distance in the configured distance units. + external_workplace_zone_id: + type: identifier + description: Assigned external workplace zone identifier. + finalweight: + type: Float64 + description: Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. + free_parking_at_work: + type: Boolean-like or category + description: Free-workplace-parking eligibility or modeled alternative. + has_license: + type: Boolean-like + description: Whether the person holds a driver's license. + home_zone_id: + type: identifier + description: Prepared home-zone identifier. + household_id: + type: Int64 + description: Canonical household identifier for the person. + is_external_worker: + type: Boolean-like + description: Whether the person's assigned workplace is external to the modeled area. + is_student: + type: Boolean-like + description: Whether the person is classified as a student. + is_worker: + type: String + description: Whether the person is classified as a worker. + person_id: + type: Int64 + description: Canonical person identifier. + person_type: + type: source-compatible category + description: ActivitySim person-type code. + school_zone_id: + type: identifier + description: Assigned school-zone identifier. + student_type: + type: String + description: Prepared school or enrollment market segment. + telecommute_frequency: + type: category + description: Modeled telecommute-frequency alternative. + transit_pass_ownership: + type: Boolean-like + description: Whether the person owns a transit pass. + transit_pass_subsidy: + type: category + description: Modeled transit-pass-subsidy alternative. + work_from_home: + type: String + description: Whether the worker has a home workplace. + workplace_zone_id: + type: identifier + description: Assigned internal workplace-zone identifier. + tours: + description: Prepared tour table. + fields: + AUTOSUFF: + type: Int32 + description: Prepared household auto-sufficiency category for the tour. + NUMBER_HH: + type: Int32 + description: Joint-party size from participant rows, number_of_participants, an existing NUMBER_HH value, or a fallback of one. + SKIMDIST: + type: Float64 + description: Prepared tour distance in the configured skim distance units. + atwork_subtour_frequency: + type: category + description: Modeled at-work subtour-frequency alternative. + destination: + type: identifier + description: Tour destination-zone identifier. + finalweight: + type: Float64 + description: Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. + household_id: + type: Int64 + description: Canonical household identifier associated with the tour. + is_external_tour: + type: Boolean-like + description: Whether the tour destination is external to the modeled area. + num_escortees: + type: Int64 + description: Number of students escorted on the tour. + num_ib_stops: + type: Int32 + description: Number of inbound intermediate stops parsed from stop_frequency. + num_ob_stops: + type: Int32 + description: Number of outbound intermediate stops parsed from stop_frequency. + num_tot_stops: + type: Int32 + description: Total intermediate stops on the tour. + person_id: + type: Int64 + description: Canonical person identifier associated with an individual tour. + school_esc_inbound: + type: String + description: Inbound school-escort alternative or status. + school_esc_outbound: + type: String + description: Outbound school-escort alternative or status. + tour_category: + type: String + description: Tour category such as mandatory, nonmandatory, joint, or at-work. + tour_id: + type: Int64 + description: Canonical tour identifier. + tour_mode: + type: String + description: Main mode selected for the tour. + tour_purpose: + type: String + description: Primary purpose of the tour. + vehicle_occup_1: + type: vehicle-type code + description: Vehicle type allocated to single-occupant auto tours. + vehicle_occup_2: + type: vehicle-type code + description: Vehicle type allocated to two-occupant auto tours. + vehicle_occup_3.5: + type: vehicle-type code + description: Vehicle type allocated to auto tours with three or more occupants. + trips: + description: Prepared trip table. + fields: + escort_event_role: + type: String + description: Adult escort-event role assigned to the trip. + escort_stops_after_event: + type: Int32 + description: Stops occurring after the associated escort event. + escort_stops_before_event: + type: Int32 + description: Stops occurring before the associated escort event. + finalweight: + type: Float64 + description: Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. + num_participants: + type: Int32 + description: Number of travelers represented by a joint trip record. + od_dist: + type: Float64 + description: Origin-to-destination trip distance from the configured distance skim or supported survey distance, in source units. + out_dir_dist: + type: Float64 + description: Additional distance caused by visiting the stop rather than traveling directly to the half-tour endpoint. + parking_zone: + type: Int64 + description: Zone where the auto trip parks. + person_id: + type: Int64 + description: Canonical person identifier associated with the trip. + stops: + type: Int32 + description: Intermediate-stop indicator, 1 before the last trip on a half-tour and 0 for the final leg. + tour_id: + type: Int64 + description: Canonical identifier of the parent tour. + tour_mode: + type: String + description: Main mode of the parent tour. + tour_purpose: + type: String + description: Primary purpose of the parent tour. + trip_mode: + type: String + description: Mode used for the trip leg. + trip_purpose: + type: String + description: Destination purpose of the trip leg. + vehicles: + description: Prepared household-vehicle table. + fields: + body_type: + type: String + description: Vehicle body-style category parsed from vehicle_type. + finalweight: + type: Float64 + description: Vehicle expansion weight inherited from the owning household or set to 1.0 when unavailable. + fuel_type: + type: String + description: Vehicle fuel or powertrain category parsed from vehicle_type. + vehicle_age: + type: Int64 + description: Vehicle age in years parsed from vehicle_type. + +categories: +- name: Population and Demographics + summaries: + - id: household_size_distribution + description: Household totals by modeled household size. Use it to profile household composition, calculate size shares, and compare population-synthesis results across runs. + fields: + household_size: number of people in the household. + household_count: weighted households in that size category. + - id: person_type_distribution + description: Person totals by ActivitySim person type, with a display label. Use it to compare demographic market segments and as a denominator for person-type travel rates. + fields: + person_type: stable person-type code. + person_type_label: configured readable label for the code. + person_count: weighted people of that type. + - id: population_totals + description: One run-level control-total row for people, households, person-tours, person-trips, and person-stops. Joint tour and trip records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback; sentinel, missing, or nonpositive fallback counts resolve to one. Use it for reasonableness checks and top-level comparisons; the measures use their own table weights and are not additive to one another. + fields: + person_count: weighted persons. + household_count: weighted households. + tour_count: weighted person-tours; joint records are participant-expanded. + trip_count: weighted person-trips; joint records are participant-expanded. + stop_count: weighted person-trip records flagged as intermediate stops; joint records are participant-expanded. +- name: Person Attributes and Long-Term Choices + summaries: + - id: license_holding_status_distribution + description: Licensed and unlicensed people age 16 or older by person type, including an all-person-types rollup. Use it to assess access to driving and explain auto-mode availability. + fields: + person_type: person-type code or `all_person_types` rollup. + license_holding_status: '`has_license` or `no_license`.' + person_type_label: configured readable person-type label. + person_count: weighted people in the group. + - id: bicycle_comfort_level_distribution + description: Bicycle comfort categories by person type, including an all-person-types rollup. Use it to understand the population assumed willing to use different bicycle facilities. + fields: + person_type: person-type code or rollup. + bicycle_comfort_level: prepared bicycle-comfort category. + person_type_label: configured readable person-type label. + person_count: weighted people in the group. + - id: transit_pass_ownership_by_person_type + description: Transit-pass ownership status by person type, including an all-person-types rollup. Use it to evaluate transit market eligibility and pass-ownership model results. + fields: + person_type: person-type code or rollup. + transit_pass_ownership_status: '`has_transit_pass` or `no_transit_pass`.' + person_type_label: configured readable person-type label. + person_count: weighted people in the group. + - id: transit_subsidy_by_person_type + description: Transit-pass subsidy alternatives for workers, by person type and with an all-person-types rollup. Use it to examine employer or institutional transit-benefit assumptions. + fields: + person_type: person-type code or rollup. + transit_subsidy_status: prepared subsidy alternative code. + transit_subsidy_label: configured readable subsidy label. + person_type_label: configured readable person-type label. + person_count: weighted eligible workers in the group. + - id: telecommute_frequency_distribution + description: Non-work-from-home workers by telecommute-frequency category and home geography, plus an all-geographies rollup. Use it to analyze recurring telecommuting among workers who still have an external workplace. + fields: + geography_type: home-geography system or rollup. + geography_id: home-geography identifier or rollup value. + telecommute_frequency: prepared telecommute-frequency alternative. + person_count: weighted workers in the group. +- name: Household Vehicles and Vehicle Characteristics + summaries: + - id: autonomous_vehicle_ownership_totals + description: One run-level total of households modeled as owning an autonomous vehicle. Use it to report scenario penetration and compare AV-ownership assumptions. + fields: + household_with_autonomous_vehicle_count: weighted households with `av_ownership` true. + - id: auto_ownership_distribution + description: Household totals jointly classified by household size and vehicle count; household sizes of five or more are grouped as `5+`. Use it to assess motorization and auto sufficiency. + fields: + household_size: household-size category, with `5+` as the terminal group. + household_vehicle_count: vehicles available to the household. + household_count: weighted households in the joint category. + - id: vehicle_age_distribution + description: Household vehicles by age, with ages 20 and older grouped as `20+`. Use it for fleet turnover, emissions, and technology analyses. + fields: + age: vehicle age in years or `20+`. + vehicle_count: weighted vehicles in the age category. + - id: vehicle_fuel_type_distribution + description: Household vehicles by prepared fuel or powertrain type. Use it for fleet composition, energy, and emissions analysis. + fields: + fuel_type: prepared vehicle fuel/powertrain category. + vehicle_count: weighted vehicles of that type. + - id: vehicle_body_type_distribution + description: Household vehicles by prepared body type. Use it to characterize the light-duty fleet and support occupancy or emissions comparisons. + fields: + body_type: prepared vehicle body-style category. + vehicle_count: weighted vehicles of that type. +- name: Long-Term Geography, Location, and Shadow Pricing + summaries: + - id: work_from_home_rate_by_geography + description: All workers and work-from-home workers by home geography, plus a regional rollup. Divide the WFH count by the worker count to calculate the WFH rate and map its spatial pattern. + fields: + geography_type: home-geography system or rollup. + geography_id: home-geography identifier or rollup value. + worker_count: weighted workers living in the geography. + work_from_home_worker_count: weighted workers flagged as working from home. + - id: internal_external_worker_by_geography + description: Internal and external workers by home geography, plus a regional rollup. Use it to understand external-worker incidence and its residential distribution. + fields: + geography_type: home-geography system or rollup. + geography_id: home-geography identifier or rollup value. + internal_worker_count: weighted workers with an internal workplace. + external_worker_count: weighted workers classified as external. + - id: external_worker_workplace_locations + description: External workers by their external workplace geography, plus a regional rollup. Use it to analyze external commute orientation; the all-worker denominator is repeated to support share calculations. + fields: + geography_type: external-workplace geography system or rollup. + geography_id: external-workplace geography identifier or rollup value. + external_worker_count: weighted external workers assigned to that destination. + all_worker_count: weighted workers in the full run, repeated on every row. + - id: workplace_location_employment_comparison + description: Land-use employment and modeled worker workplace choices aligned by workplace geography. Use it to compare attraction targets with assigned workers and diagnose location-choice balance. + fields: + geography_type: workplace-geography system. + geography_id: workplace-geography identifier. + employment_count: employment opportunities from land use. + worker_count: weighted workers assigned to the geography. + - id: commuting_flows + description: Worker flows from home geography to workplace geography at matching configured geography levels, plus a regional total. Use it as an origin-destination matrix for commute sheds, self-containment, and interjurisdictional flows. + fields: + origin_geography_type: home-geography system. + origin_geography_id: home-geography identifier. + destination_geography_type: workplace-geography system. + destination_geography_id: workplace-geography identifier. + commuter_count: weighted workers in the OD pair. + - id: school_location_enrollment_comparison + description: Land-use enrollment and modeled student school locations aligned by geography and student type. Use it to compare school-location targets with assigned students. + fields: + geography_type: school-geography system. + geography_id: school-geography identifier. + student_type: prepared school/enrollment market segment. + enrollment_count: target enrollment from land use. + student_count: weighted students assigned to the geography and type. + - id: workplace_shadow_pricing_residuals + description: Zone-level workplace target-versus-modeled residuals. Use positive residuals to find over-assigned workplace geographies and negative residuals to find under-assigned ones. + fields: + geography_type: workplace-geography system. + geography_id: workplace-geography identifier. + target_count: land-use employment target. + modeled_count: weighted assigned workers. + residual_count: '`modeled_count - target_count`.' + absolute_residual_count: absolute residual magnitude. + percent_error: residual divided by target, times 100; null when target is zero. + - id: school_shadow_pricing_residuals + description: Zone-level school target-versus-modeled residuals by student type. Use it to diagnose school-location shadow-pricing convergence and segment-specific imbalance. + fields: + geography_type: school-geography system. + geography_id: school-geography identifier. + student_type: school/enrollment market segment. + target_count: land-use enrollment target. + modeled_count: weighted assigned students. + residual_count: modeled minus target. + absolute_residual_count: absolute residual magnitude. + percent_error: residual divided by target, times 100; null for zero targets. + - id: workplace_shadow_pricing_residual_histogram + description: Distribution of workplace residuals by geography system. Use it to assess convergence across all zones without inspecting each zone separately; zero residuals receive their own zero-width bin. + fields: + geography_type: workplace-geography system. + bin_start: inclusive lower residual bound. + bin_end: upper residual bound; both bounds are zero for the exact-zero bin. + geography_count: number of geography records in the bin. + - id: school_shadow_pricing_residual_histogram + description: Distribution of school residuals by geography system and student type. Use it to compare convergence across student markets. + fields: + geography_type: school-geography system. + student_type: school/enrollment market segment. + bin_start: lower residual bound. + bin_end: upper residual bound, or zero for the exact-zero bin. + geography_count: number of geography/student-type records in the bin. + - id: park_and_ride_location_residuals + description: Modeled park-and-ride tour use compared with lot capacity by lot geography. Use it to identify over-capacity or underused PNR locations. + fields: + geography_type: PNR-lot geography system. + geography_id: PNR-lot geography identifier. + pnr_tour_count: weighted PNR tours assigned to the location. + pnr_lot_capacity: supplied lot capacity target. + residual_count: tours minus capacity. + absolute_residual_count: absolute residual magnitude. + percent_error: residual divided by capacity, times 100; null for zero capacity. + - id: park_and_ride_location_residual_histogram + description: Distribution of PNR use-minus-capacity residuals by geography system. Use it for systemwide capacity-fit assessment. + fields: + geography_type: PNR-lot geography system. + bin_start: lower residual bound. + bin_end: upper residual bound, or zero for the exact-zero bin. + geography_count: number of PNR geography records in the bin. + - id: free_parking_eligibility_by_workplace_geography + description: Workers with and without free workplace parking by workplace geography. Use it to analyze parking-cost exposure and its effect on commute mode choice. + fields: + geography_type: workplace-geography system. + geography_id: workplace-geography identifier. + workers_without_free_parking_count: weighted workers not eligible for free parking. + workers_with_free_parking_count: weighted workers eligible for free parking. +- name: Long-Term Location Distance + summaries: + - id: work_location_distance_distribution_by_geography + description: Internal, non-work-from-home workers with valid workplace locations by home geography and home-to-work distance. Use it for commute-length distributions and spatial comparisons. + fields: + distance_bin: exact-zero, positive-sub-mile, whole-mile lower-bound, or `51+` category. + geography_type: home-geography system or rollup. + geography_id: home-geography identifier or rollup value. + person_count: weighted workers in the bin. + - id: university_location_distance_distribution_by_geography + description: University students, identified by person type 3, by home geography and home-to-school distance. Use it to examine university travel markets and campus catchments. + fields: + distance_bin: exact-zero, positive-sub-mile, whole-mile lower-bound, or `51+` category. + geography_type: home-geography system or rollup. + geography_id: home-geography identifier or rollup value. + person_count: weighted university students in the bin. + - id: school_location_distance_distribution_by_geography + description: School students, identified by person types 6 and higher, by home geography and home-to-school distance. Use it for K--12 travel-distance and school-catchment analysis. + fields: + distance_bin: exact-zero, positive-sub-mile, whole-mile lower-bound, or `51+` category. + geography_type: home-geography system or rollup. + geography_id: home-geography identifier or rollup value. + person_count: weighted school students in the bin. +- name: Daily Activity Patterns and Travel Rates + summaries: + - id: daily_activity_pattern_by_person_type + description: Daily activity pattern alternatives by person type, using each person's non-null person-table CDAP value and falling back only for remaining persons to all observed day rows with day weights when available, including an all-person-types rollup. No day rows are deduplicated. Use it to compare mandatory, nonmandatory, and home-stay behavior. + fields: + person_type: person-type code or rollup. + daily_activity_pattern: prepared CDAP/activity-pattern category. + person_count: weighted people or observed person-days in the pattern. + - id: mandatory_tour_frequency_by_person_type + description: Positive mandatory-tour-frequency choice by person type, using each person's non-null prepared choice and falling back only for remaining persons to choices derived from work and school tours on every person-day, with day weights when available, plus an all-person-types rollup. Use it to analyze how many mandatory tours travelers make. The table excludes records with a choice of zero. + fields: + person_type: person-type code or rollup. + mandatory_tour_frequency: prepared or tour-derived positive mandatory-tour frequency alternative. + person_count: weighted people or person-days choosing that frequency. + - id: nonmandatory_tour_frequency_by_person_type + description: Count of individual nonmandatory tours plus joint-tour participation, grouped as 0, 1, 2, or 3+, by person type and for all types. Genuinely multi-day inputs are classified per person-day only when every tour and participant has a safe day key; otherwise the one-day/person fallback is retained rather than dropping records or guessing keys. Use it to compare discretionary travel propensity. + fields: + person_type: person-type code or rollup. + nonmandatory_tour_frequency: combined nonmandatory-tour category `0`, `1`, `2`, or `3+`. + person_count: weighted people or person-days in the category. + - id: tour_rates_by_person_type_and_tour_purpose + description: Person-tours per person-day by person type and tour purpose under the selected weighting mode, plus all-person-types rates. Every eligible source row contributes and at-work subtours are excluded. Joint tours are attributed using each known participant's person weight or expanded by a valid party size when participant records are unavailable. Use it to compare tour-generation rates while controlling for population composition. + fields: + person_type: person-type code or rollup. + tour_purpose: prepared tour-purpose category. + tour_rate: sum of eligible person-tour weights, divided by summed person or person-day weights for that person type. + - id: trip_rates_by_person_type_and_trip_purpose + description: Person-trips per person-day by person type and trip purpose under the selected weighting mode, plus all-person-types rates. Every eligible source row contributes. Joint trips are attributed using each known participant's person weight or expanded by a valid party size when participant records are unavailable. Use it to compare trip-generation rates across demographic markets. + fields: + person_type: person-type code or rollup. + trip_purpose: destination purpose of the trip. + trip_rate: sum of eligible person-trip weights, divided by summed person or person-day weights for that person type. +- name: School Escorting + summaries: + - id: escorted_tour_totals + description: One run-level total of adult-side tours with an outbound or inbound school-escort condition. Use it as the top-level escorted-tour control total. + fields: + tour_count: weighted distinct eligible tours with at least one escorted direction. + - id: school_escorted_tours_by_escort_type_and_direction + description: Adult-side escorted tours by escort arrangement and direction, with an `all_directions` incidence rollup. Use it to compare ride-share and pure-escort patterns. + fields: + escort_type: prepared escort arrangement label. + direction: '`outbound`, `inbound`, or `all_directions`.' + tour_count: weighted escorted-tour incidences. + - id: adult_escorted_tour_purposes_by_direction + description: Purposes of the adult tours that do school escorting, by direction and with an all-directions incidence rollup. Use it to see how escorting connects with work or other adult activities. + fields: + tour_purpose: adult tour's primary purpose. + direction: escorted half or `all_directions`. + tour_count: weighted escorted-tour incidences. + - id: adult_escorted_tours_by_person_type_and_direction + description: Adult-side escorted tours by the adult traveler's person type and escorted direction. Use it to identify who performs school escorting. + fields: + person_type: adult traveler person-type code. + direction: '`outbound`, `inbound`, or `both`.' + tour_count: weighted tours meeting that directional condition. + - id: student_school_escort_status_by_direction + description: Student school tours classified by normalized escort type for each direction and for tours escorted both ways. Use it to measure the student-side escort experience. + fields: + direction: '`outbound`, `inbound`, or `both`.' + escort_type: normalized escort arrangement, including unescorted alternatives where present. + tour_count: weighted student school tours in the group. + - id: student_households_by_student_count + description: Households by the number of school-age/student household members recognized by the escort logic. Use it as a denominator for household escort participation. + fields: + student_count: students in the household. + household_count: weighted households with that count. + - id: households_with_school_escorting_by_student_count_and_direction + description: Unique households with at least one escorted student school tour, by number of students and directional condition. Use it to calculate escort-participation rates by household composition. + fields: + student_count: students in the household. + direction: '`outbound`, `inbound`, or `both`.' + household_count: weighted unique households meeting the condition. + - id: schoolkids_per_escorted_tour_by_student_count_and_direction + description: Average number of escorted children on adult-side escorted tours by household student count and direction. Use it to analyze escorting efficiency and child grouping. + fields: + student_count: students in the adult traveler's household. + direction: '`outbound`, `inbound`, or `both`.' + avg_schoolkids_per_tour: weighted mean number of escortees per eligible tour. + tour_count: weighted eligible tours used as the mean denominator. + - id: adult_escorted_tour_distance_distribution_by_direction + description: Adult-side escorted tours by lower-bound tour-distance bin and directional escort condition. Use it to compare the length of outbound-only, inbound-only, and both-way escort tours. + fields: + distance_bin: exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`. + direction: '`outbound`, `inbound`, or `both`.' + tour_count: weighted eligible tours in the bin. + - id: adult_escorted_trip_distance_distribution_by_direction + description: Trips on adult tours marked as escorted, by outbound or inbound half and lower-bound trip-distance bin. Use it to examine the trip-leg distance for escorting. + fields: + distance_bin: exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`. + direction: '`outbound`, `inbound`, or `both` condition.' + trip_count: weighted eligible trips in the bin. + - id: adult_escort_event_stop_distribution + description: Intermediate stops before and after school drop-off or pickup on adult tours marked as escorted. Use it to analyze trip chains around escort events. + fields: + segment: one of `outbound_before_dropoff`, `outbound_after_dropoff`, `inbound_before_pickup`, or `inbound_after_pickup`. + stop_count: prepared count of stops in that segment. + tour_count: weighted escort-event records with that stop count. + - id: adult_escort_trip_stop_frequency + description: Adult-side escorted tours jointly classified by purpose and outbound, inbound, and total stop counts. Use it to compare stop-making complexity on escort tours. + fields: + tour_purpose: adult tour purpose. + outbound_stop_count: outbound stops capped at 3. + inbound_stop_count: inbound stops capped at 3. + total_stop_count: total stops capped at 6. + tour_count: weighted escorted tours in the combination. +- name: Joint Travel + summaries: + - id: jtf_distribution + description: Household observations across the legacy 21 joint-tour-frequency labels, selected by minimum counts in five fixed purpose slots covering shopping, maintenance, eating out, visiting/social, and other discretionary. Purposes outside those slots do not affect the label, and counts above a label's threshold are not rejected, so the labels are not strict total-tour counts. When the day table shows a genuinely multi-day diary and tours contain `day_num`, observations use `(household_id, day_num)`; null or unmatched tour days do not contribute. Without the tour day column, the result is empty rather than pooled into a weekly choice. On a one-day input, the declaration does not require tours, so an unavailable tour table can instead classify every household as `No Joint Tours`; verify tour availability before using that result. Use it to validate compatible joint-tour generation inputs. + fields: + jtf_code: integer alternative 1--21. + jtf_label: readable minimum-frequency/purpose-combination label. + household_count: sum of `hh.finalweight` across households or household-day observations assigned to the alternative. + - id: joint_tours_by_household_size + description: All household observations and observations making at least one joint tour by household size. A genuinely multi-day diary uses household-days when tours contain `day_num`; null or unmatched tour days do not contribute, and a missing tour day column produces an empty result rather than pooling days. Valid shared joint-tour or tour IDs collapse repeated records; records without either valid ID remain distinct. Use the two counts to calculate joint-tour participation rates. + fields: + household_size: number of household members. + household_count: sum of `hh.finalweight` across households or household-day observations of that size. + joint_tour_hh_count: sum of `hh.finalweight` for those observations with a joint tour. + - id: joint_tour_party_size_distribution + description: Household joint tours by valid participant count, with parties of five or more stored in bin 5. Verified survey participant-row groups retain every row at a fractional representation weight that sums to one household tour; ordinary model joint-tour rows retain full weight. Sentinel values of 995 or greater, missing values, and nonpositive values are excluded. Use it to assess joint-tour occupancy. + fields: + party_size: household participants; value 5 represents `5+`. + joint_tour_count: weighted joint tours in the party-size bin. + - id: joint_tour_composition_distribution + description: Household joint tours by prepared party-composition category. Verified participant-row representations retain every row fractionally so each complete group contributes one household tour. Use it to compare adult-only, child-inclusive, and other modeled compositions. + fields: + tour_composition: prepared joint-party composition. + joint_tour_count: weighted joint tours in the category. + - id: joint_tour_composition_by_party_size + description: Household joint tours jointly classified by party composition and exact valid participant count. Verified participant-row representations retain every row fractionally so each complete group contributes one household tour. Sentinel values of 995 or greater, missing values, and nonpositive values are excluded. Use it to study how household makeup and group size interact. + fields: + tour_composition: prepared party-composition category. + party_size: number of tour participants. + joint_tour_count: weighted joint tours in the combination. + - id: person_jtp_by_household_size + description: All person observations and observations participating in one or more joint tours by household size. When a multi-day input has person-day records and each joint-tour identity maps to one day, the observations are person-days and participation comes from linked joint participants. A missing or ambiguous tour-day mapping produces an empty result; if the day table lacks household, person, or day fields, the person-level fallback remains. The multi-day path's participant-table limitations are documented in chapter 25. Use the two counts to calculate participation rates. + fields: + household_size: size of the person's household. + joint_tour_person_count: sum of the applicable person or day weight for people or person-days participating in a joint tour. + total_person_count: sum of the applicable person or day weight for people or person-days in households of that size. + - id: household_jtp_by_household_size_and_jtf + description: For household observations of size two or more, weighted percentage distribution across 0, 1, and 2+ joint-tour identities within each household size. A genuinely multi-day diary uses household-days when tours contain `day_num`; null or unmatched tour days do not contribute, and a missing tour day column produces an empty result rather than pooling days. Valid shared joint-tour or tour IDs collapse repeated records; records without either valid ID remain distinct. Use it to compare joint-tour propensity independent of household-size totals across compatible runs. + fields: + jtf: joint-tour count category `0`, `1`, or `2+`. + household_size: household size as a category. + household_percent: percent of summed `hh.finalweight` for household observations of that size in the JTF category. +- name: Basic Tour Distributions + summaries: + - id: tour_category_distribution + description: Household-tour-unit counts by ActivitySim category, such as mandatory, nonmandatory, at-work, or joint. Verified participant-row representations of one joint tour are retained fractionally rather than collapsed. Use it for high-level tour-system composition. + fields: + tour_category: prepared tour category. + tour_count: weighted tours in the category. + - id: tour_purpose_distribution + description: Household-tour-unit counts by configured summary purpose. Verified participant-row representations of one joint tour are retained fractionally rather than collapsed. Use it to compare the volume and share of work, school, escort, shopping, and other travel. + fields: + tour_purpose: canonical summary tour purpose. + tour_count: weighted tours for that purpose. +- name: Vehicles Allocated to Tours + summaries: + - id: allocated_vehicle_age_by_occupancy + description: Allocated vehicle ages by occupancy condition. Use it to analyze how fleet age is associated with single- and shared-occupant travel. + fields: + age: decoded vehicle age, with `20+` terminal. + occupancy: allocation condition `1`, `2`, or `3+`. + vehicle_count: weighted tour allocation incidences. + - id: allocated_vehicle_fuel_type_by_occupancy + description: Allocated vehicle fuel/powertrain type by occupancy condition. Use it for energy or emissions segmentation of auto travel. + fields: + fuel_type: decoded fuel/powertrain category. + occupancy: allocation condition `1`, `2`, or `3+`. + vehicle_count: weighted tour allocation incidences. + - id: allocated_vehicle_body_type_by_occupancy + description: Allocated vehicle body type by occupancy condition. Use it to relate party size to the modeled vehicle used. + fields: + body_type: decoded vehicle body-style category. + occupancy: allocation condition `1`, `2`, or `3+`. + vehicle_count: weighted tour allocation incidences. +- name: Tour Mode, Stops, Time, and Distance + summaries: + - id: tour_mode_by_tour_purpose_and_auto_sufficiency + description: Person-tour mode counts by purpose and household auto sufficiency, with all-purpose rows. Joint records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback; sentinel, missing, or nonpositive fallback sizes resolve to one. Use it to compare mode choice across vehicle-availability markets. + fields: + tour_mode: prepared tour mode. + tour_purpose: tour purpose or `all_tour_purposes`. + tour_count_zero_auto: weighted person-tours from zero-auto households. + tour_count_auto_deficient: weighted person-tours from households with fewer autos than workers. + tour_count_auto_sufficient: weighted person-tours from auto-sufficient households. + tour_count_all_households: sum of the three auto-sufficiency counts. + - id: tour_stop_frequency_by_tour_purpose + description: Household-tour-unit counts jointly classified by purpose and outbound, inbound, and total intermediate-stop counts. Verified participant-row joint-tour groups retain every row at fractional weight summing to one tour. Use it to measure tour complexity and stop-generation patterns. + fields: + tour_purpose: canonical tour purpose. + outbound_stop_count: outbound stops capped at 3. + inbound_stop_count: inbound stops capped at 3. + total_stop_count: total stops capped at 6. + tour_count: weighted tours in the combination. + - id: atwork_subtour_frequency_distribution + description: Mandatory work tours by their at-work-subtour-frequency alternative. Use it to validate subtour generation from the workplace. + fields: + atwork_subtour_frequency_category: prepared at-work subtour-frequency category. + atwork_subtour_count: weighted parent work tours choosing the category. + - id: tour_time_of_day_by_tour_purpose + description: Dense household-tour-unit departure, arrival, and duration profiles by tour purpose plus all-purpose totals. Every eligible row is retained; verified participant-row representations of one joint tour contribute fractionally so their combined weight equals one tour. Use it to compare scheduling and duration distributions. + fields: + time_bin: ActivitySim period index. + tour_purpose: tour purpose or `all_tour_purposes`. + departure_tour_count: weighted tours starting in the bin. + arrival_tour_count: weighted tours ending in the bin. + duration_tour_count: weighted tours whose prepared duration falls in the bin. + - id: tour_distance_by_tour_purpose + description: Nonmandatory, joint, and at-work household-tour-unit records by lower-bound `SKIMDIST` bin and purpose, plus all-purpose totals; mandatory tours are excluded to match the legacy distribution. The metric is fixed to prepared origin-to-primary-destination `SKIMDIST` and does not switch to `tour_distance` or generic `distance_miles`. Every row is retained, with verified participant-row joint-tour groups contributing fractional weight that sums to one tour. + fields: + distance_bin: exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`. + tour_purpose: purpose or `all_tour_purposes`. + tour_count: sum of eligible household-tour representation weight in the bin. +- name: Tour Geography + summaries: + - id: average_mandatory_tour_distance_by_purpose_and_geography + description: Weighted average home-to-work or home-to-school distance for internal non-WFH workers, university students, and school students, by home geography and regionwide. Use it to compare mandatory destination accessibility. + fields: + mandatory_tour_purpose: '`work`, `university`, or `school`.' + geography_type: home-geography system or rollup. + geography_id: home-geography identifier or rollup. + average_tour_distance: mean person-level mandatory distance weighted by `per.finalweight`. + person_count: sum of `per.finalweight` for people contributing to the mean. + - id: average_nonmandatory_tour_distance_by_purpose_and_geography + description: Weighted average prepared origin-to-primary-destination `SKIMDIST` for individual nonmandatory tours by purpose and traveler home geography, plus regional rows. The metric does not switch to `tour_distance` or generic `distance_miles`. Use it to compare discretionary travel reach. + fields: + nonmandatory_tour_purpose: nonmandatory purpose. + geography_type: home-geography system or rollup. + geography_id: home-geography identifier or rollup. + average_tour_distance: mean prepared `SKIMDIST` weighted by `tours.finalweight`. + tour_count: sum of `tours.finalweight` for tours contributing to the mean. + - id: internal_external_nonmandatory_tour_frequency_by_home_geography + description: Internal and external nonmandatory tours by traveler home geography, plus regional totals. Use it to calculate external-tour shares and locate households producing external discretionary travel. + fields: + geography_type: home-geography system or rollup. + geography_id: home-geography identifier or rollup. + internal_nonmandatory_tour_count: weighted internal nonmandatory tours. + external_nonmandatory_tour_count: weighted external nonmandatory tours. + - id: external_nonmandatory_tour_locations + description: External nonmandatory tours by destination geography, plus a regional total. Use it to analyze external destination orientation and gateway demand. + fields: + geography_type: destination-geography system or rollup. + geography_id: destination-geography identifier or rollup. + external_nonmandatory_tour_count: weighted external nonmandatory tours ending there. +- name: Trip Purpose, Mode, and Parking + summaries: + - id: trip_purpose_distribution + description: Person-trip destination purpose cross-classified by parent tour purpose, with all-tour-purpose rows. Joint records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to analyze activity chains within different kinds of tours. + fields: + tour_purpose: parent tour purpose or `all_tour_purposes`. + trip_purpose: destination purpose of the trip leg. + trip_count: weighted person-trips in the combination. + - id: stop_destination_purpose_by_tour_purpose + description: Person-stop destination purposes by parent tour purpose. Joint records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to understand what activities are chained into tours. + fields: + stop_destination_purpose: destination purpose of trip records flagged as stops. + tour_purpose: parent tour purpose. + stop_count: weighted person-stops in the combination. + - id: trip_mode_by_tour_purpose_and_tour_mode + description: Person-trip mode counts by parent tour purpose and main tour mode, including all-purpose, all-tour-mode, and grand rollups. Joint records use the sum of all matching participant person weights when available; valid party-size expansion is the fallback, and sentinel, missing, or nonpositive party sizes fall back to one. Use it to examine access/egress and mode combinations within tours. + fields: + tour_purpose: parent purpose or `all_tour_purposes`. + tour_mode: main tour mode or `all_tour_modes`. + trip_mode: mode of the individual trip leg. + trip_count: weighted person-trips in the combination. + - id: parking_locations + description: Auto-trip parking events by configured parking geography. Use it to map modeled parking demand and compare locations across runs. + fields: + geography_type: parking-geography system. + geography_id: valid positive parking-zone identifier at that geography. + trip_count: weighted trips parking there. +- name: Trip Time and Distance + summaries: + - id: trip_departure_time_by_purpose + description: Dense person-trip and person-stop departure-period profiles by parent tour purpose and for all purposes. Joint records use all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to compare trip timing with stop timing. + fields: + tour_purpose: parent purpose or `all_tour_purposes`. + time_bin: prepared trip departure period index. + departure_trip_count: weighted person-trips departing in the bin. + departure_stop_count: weighted departing person-stops. + - id: trip_distance_by_purpose + description: Person-trips by lower-bound OD-distance bin and parent tour purpose, plus all-purpose totals. Joint records use the sum of all matching participant person weights when available; valid party-size expansion is the fallback, non-joint trips contribute once, and sentinel, missing, or nonpositive fallback sizes resolve to one. Use it for purpose-specific trip length distributions. + fields: + distance_bin: exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`. + tour_purpose: parent purpose or `all_tour_purposes`. + trip_count: sum of participant-adjusted trip weight in the bin. + - id: stop_out_of_direction_distance_by_tour_purpose + description: Person-stops by lower-bound out-of-direction-distance bin, with a dense distribution for each purpose and all purposes. Joint records use all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to quantify detour burden from stop-making. + fields: + distance_bin: exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`. + tour_purpose: parent purpose or `all_tour_purposes`. + stop_count: weighted person-stops in the bin. +- name: Skimjoin Diagnostics + summaries: + - id: skimjoin_trip_component_stats + description: Weighted descriptive statistics for every numeric `skim_` component on trips, by trip mode and skim scenario. Use it to QA joined time, distance, and cost values and identify missing or zero-heavy components. + fields: + skim_scenario: chosen or hypothetical evaluation scenario. + trip_mode: trip mode or `All Modes`. + component: numeric skim column name. + n_total: total trip weight eligible for the component/mode group. + n_valid: trip weight with non-null component values, including zeros. + mean: weighted mean including zeros. + mean_nonzero: weighted mean excluding zeros. + std: weighted population standard deviation. + min: minimum observed value. + max: maximum observed value. + median: weighted 50th percentile. + mode: value with greatest total weight, using the smaller value on ties. + zero_share: valid weight at exactly zero divided by `n_valid`. + missing_share: missing weight divided by `n_total`. + - id: skimjoin_trip_component_ecdf + description: Optional 0th-through-100th weighted percentile curves for trip skim components by mode and scenario. Use it for distribution comparison when the compact stats table is insufficient. + fields: + skim_scenario: chosen or hypothetical scenario. + trip_mode: trip mode or `All Modes`. + component: numeric skim column name. + percentile: cumulative probability from 0.00 through 1.00 in 0.01 steps. + value: weighted quantile at that probability. + n_valid: total valid trip weight behind the curve. + - id: skimjoin_tour_component_stats + description: Weighted descriptive statistics for numeric tour `skim_` components by tour mode and scenario. Use it to QA tour-level round-trip or composite skims. + fields: + skim_scenario: chosen or hypothetical evaluation scenario. + tour_mode: tour mode or `All Modes`. + component: numeric skim column name. + n_total: total tour weight in scope. + n_valid: tour weight with a non-null value, including zeros. + mean: weighted mean including zeros. + mean_nonzero: weighted mean excluding zeros. + std: weighted population standard deviation. + min: minimum value. + max: maximum value. + median: weighted median. + mode: highest-weight value, smaller on ties. + zero_share: valid weight at zero divided by `n_valid`. + missing_share: missing weight divided by `n_total`. + - id: skimjoin_tour_component_ecdf + description: Optional weighted percentile curves for tour skim components by mode and scenario. Use it to compare complete tour-level distributions across runs. + fields: + skim_scenario: chosen or hypothetical scenario. + tour_mode: tour mode or `All Modes`. + component: numeric skim column name. + percentile: probability from 0.00 through 1.00. + value: weighted quantile at that probability. + n_valid: valid tour weight behind the curve. +- name: Processor-Built Validation Summaries + summaries: + - id: traffic_count_comparisons + description: Observed and modeled traffic counts that agree at count-location, direction, and period level. Use it for count scatterplots, percent differences, RMSE, and facility calibration. The table keeps only keys in both sources. + fields: + count_location_id: traffic-count station/location identifier. + direction: observed/modeled direction label. + count_period: count time-period label. + observed_volume: summed observed count for the key. + modeled_volume: summed assigned volume for the matching key. + - id: screenline_flow_comparisons + description: Observed and modeled screenline flows matched by screenline, direction, and period, with a representative facility type. Use it for corridor-level flow validation and regression analysis. + fields: + screenline_id: screenline/cutline identifier. + direction: flow direction. + count_period: comparison period. + facility_type: supplied facility class, or `All` if absent. + observed_volume: summed observed flow. + modeled_volume: summed modeled flow for the matching key. + - id: transit_boardings_by_operator_and_technology + description: Assigned transit boardings summed by operator and transit technology. Use it to compare ridership scale across agencies and modes. + fields: + operator: supplied transit operator identifier or name. + technology: supplied transit mode/technology category. + boardings: total assigned unlinked passenger boardings. + - id: transit_transfer_rate + description: Assigned boardings divided by linked transit trips by operator, technology, and access mode. The value is boardings per linked trip, so values above one indicate transfers; subtract one if a transfers-per-trip measure is needed. + fields: + operator: transit operator. + technology: transit technology/mode. + access_mode: mode used to access transit. + transfer_rate: assigned boardings divided by linked trips; null for a zero linked-trip denominator. + - id: auto_vmt_totals + description: One run-level personal-auto VMT total using the same eligible records, distance source, and legacy-compatible occupancy calculation as the detailed personal-auto VMT summary. Every eligible row is retained. Household-level joint rows use one vehicle, while repeated person-level joint rows use their fixed auto-mode occupancy. A prepared auto skim is preferred; otherwise eligible non-drive-transit trips use `od_dist`. Use it for overview controls and scenario comparison. + fields: + auto_vmt: total weighted personal-auto vehicle miles traveled. + - id: auto_vmt_by_home_geography_income_hhsize_time_period + description: Personal-auto VMT and trip counts by home geography, income, household size, time period, and auto mode. When component periods exist, Daily rows are derived only from those non-Daily rows and replace any stale source Daily row; Daily-only groups are retained. Use it for equity, temporal, modal, and spatial VMT analysis. + fields: + geography_type: home-geography system or `all_geographies`. + geography_id: home-geography identifier or rollup. + income_segment: prepared household income segment or fallback rollup. + household_size: prepared household size or fallback rollup. + time_period: configured period or `Daily`. + mode: trip mode or `All Auto` fallback. + auto_vmt: sum of distance times `trips.finalweight`, divided by legacy-compatible vehicle occupancy. + trip_count: sum of `trips.finalweight` for eligible auto trips. + distance_source: provenance of the distance used, such as a skim or OD-distance field. + time_period_source: provenance of the time-period assignment. + - id: non_motorized_vmt_by_home_geography_income_hhsize_time_period + description: Walk, bicycle, and e-bike weighted miles and trip counts by home geography, income, household size, period, and mode, including derived daily rows. Use it for active-travel exposure and equity analysis. + fields: + geography_type: home-geography system or rollup. + geography_id: home-geography identifier or rollup. + income_segment: household income segment or fallback rollup. + household_size: household size or fallback rollup. + time_period: configured period or `Daily`. + mode: '`WALK`, `BIKE`, or `EBIKE` as available.' + non_motorized_vmt: distance times `trips.finalweight`; despite the VMT name, this is weighted traveler mileage. + trip_count: sum of `trips.finalweight` for eligible trips. + distance_source: prepared or skim distance source used for the mode. + time_period_source: provenance of the period assignment. + - id: commercial_vmt_totals + description: Commercial-vehicle VMT by vehicle type, split between internal and external travel. Use it for freight VMT totals and internal/external shares. + fields: + commercial_vehicle_type: supplied commercial vehicle/truck class. + external_vmt: VMT from records classified as external. + internal_vmt: VMT from records classified as internal. + - id: bicycle_vmt_by_facility_type + description: Bicycle VMT by facility type, read directly when supplied or calculated as assigned bicycle trips times link distance. Use it to evaluate bicycle use by facility class. + fields: + facility_type: supplied bicycle/network facility category. + bicycle_vmt: summed bicycle vehicle/traveler miles on that facility type. +- name: Externally Supplied Validation Contracts + summaries: + - id: link_validation_summary + description: Modeled network link volumes with link endpoints and facility class. Use it to aggregate modeled flow by facility or inspect high-volume links. + fields: + id: link identifier. + From_Node: upstream node identifier. + To_Node: downstream node identifier. + FACTYPE: facility-type code. + am_vol: AM-period modeled link volume. + md_vol: midday modeled link volume. + pm_vol: PM-period modeled link volume. + day_vol: daily modeled link volume. + - id: count_location_counts_validation_summary + description: Observed traffic-count volumes by count location and facility class. Use it as the observed side of location-level modeled-versus-observed comparisons. + fields: + id: count-location identifier. + FACTYPE: facility-type code. + am_vol: observed AM volume. + md_vol: observed midday volume. + pm_vol: observed PM volume. + day_vol: observed daily volume. + - id: count_location_volumes_validation_summary + description: Modeled volumes at the traffic-count locations. Use it as the modeled side of location-level count comparisons. + fields: + id: count-location identifier matching the observed table. + FACTYPE: facility-type code. + am_vol: modeled AM volume. + md_vol: modeled midday volume. + pm_vol: modeled PM volume. + day_vol: modeled daily volume. + - id: count_location_scatter_validation_summary + description: Long-form observed/modeled point pairs already prepared for count scatterplots. Use it when the source workflow supplies paired values directly. + fields: + id: count-location identifier. + facility_type: facility class code or label. + period: comparison period. + observed_volume: observed traffic volume. + modeled_volume: modeled traffic volume. + - id: count_location_fit_validation_summary + description: Precomputed linear-fit diagnostics for observed-versus-modeled counts by facility type and period. Use it to draw regression lines and report calibration fit. + fields: + facility_type: facility class used for the fit. + period: time period used for the fit. + slope: fitted slope for modeled volume as a function of observed volume. + intercept: fitted modeled-volume intercept. + r_squared: coefficient of determination. + n_locations: paired count locations in the fit. + observed_min: minimum observed volume in the fitting data. + observed_max: maximum observed volume. + equation_label: preformatted regression-equation text. + r_squared_label: preformatted R-squared text. + - id: district_commuting_flows_validation_summary + description: Supplied district-to-district commute-flow matrix for Albany, Corvallis, Lebanon, and Philomath. Use it as a local validation/control matrix. + fields: + '': origin district or row label. + Albany: commuters to Albany. + Corvallis: commuters to Corvallis. + Lebanon: commuters to Lebanon. + Philomath: commuters to Philomath. + Total: row total across destinations. + - id: county_commuting_flows_validation_summary + description: Supplied county-to-county commute-flow matrix for Benton, Linn, and Marion counties. Use it as a regional commute-flow validation/control matrix. + fields: + '': origin county or row label. + Benton: commuters to Benton County. + Linn: commuters to Linn County. + Marion: commuters to Marion County. + Total: row total across destinations. + - id: commercial_vehicle_validation_summary + description: Supplied commercial-vehicle trip totals by time of day and vehicle class. Use it to compare commercial demand composition and daily profiles. + fields: + tod: time-of-day row label. + car: commercial-car/light-vehicle trips. + mu: multi-unit truck trips. + su: single-unit truck trips. + Total: total commercial trips across classes. + - id: commercial_vehicle_vmt_validation_summary + description: Supplied commercial-vehicle VMT by time of day and vehicle class. Use it to compare freight mileage composition and temporal patterns. + fields: + tod: time-of-day row label. + car: commercial-car/light-vehicle VMT. + mu: multi-unit truck VMT. + su: single-unit truck VMT. + Total: total commercial VMT across classes. + - id: external_trip_validation_summary + description: Supplied external trip totals by time of day and purpose/class. Use it to analyze gateway demand by travel market. + fields: + tod: time-of-day row label. + hbcoll: home-based college trips. + hbo: home-based other trips. + hbr: home-based recreation trips. + hbs: home-based shopping trips. + hbsch: home-based school trips. + hbw: home-based work trips. + nhbnw: non-home-based non-work trips. + nhbw: non-home-based work trips. + truck: truck trips. + Total: total external trips across purposes/classes. + - id: external_vmt_validation_summary + description: Supplied external VMT by time of day and purpose/class. Use it to identify which external markets contribute mileage. + fields: + tod: time-of-day row label. + hbcoll: home-based college VMT. + hbo: home-based other VMT. + hbr: home-based recreation VMT. + hbs: home-based shopping VMT. + hbsch: home-based school VMT. + hbw: home-based work VMT. + nhbnw: non-home-based non-work VMT. + nhbw: non-home-based work VMT. + truck: truck VMT. + Total: total external VMT across purposes/classes. + - id: auto_vmt_validation_summary + description: Supplied auto and truck VMT by time of day and occupancy class. Use it as an independent control for modeled VMT. + fields: + TOD: time-of-day row label. + SOV: single-occupant-vehicle VMT. + HOV2: two-occupant shared-ride VMT. + HOV3: three-or-more-occupant shared-ride VMT. + Truck: truck VMT. + Total: total VMT across listed classes. + - id: work_from_home_validation_summary + description: Supplied worker and work-from-home controls by district. Use it to compare modeled WFH counts or rates with external targets. + fields: + District: district name or identifier. + Workers: total workers in the district. + WFH: workers who work from home. diff --git a/simor_configs/assets/lcog_logo.jpg b/simor_configs/assets/lcog_logo.jpg new file mode 100644 index 0000000..7656ee2 Binary files /dev/null and b/simor_configs/assets/lcog_logo.jpg differ diff --git a/simor_configs/assets/metro_logo.png b/simor_configs/assets/metro_logo.png new file mode 100644 index 0000000..a680983 Binary files /dev/null and b/simor_configs/assets/metro_logo.png differ diff --git a/simor_configs/assets/skats_logo_white_bg.png b/simor_configs/assets/skats_logo_white_bg.png new file mode 100644 index 0000000..c5b924c Binary files /dev/null and b/simor_configs/assets/skats_logo_white_bg.png differ diff --git a/simor_configs/comparison_configs/estimation_mode_outputs_comparison.yaml b/simor_configs/comparison_configs/estimation_mode_outputs_comparison.yaml new file mode 100644 index 0000000..59fa4d4 --- /dev/null +++ b/simor_configs/comparison_configs/estimation_mode_outputs_comparison.yaml @@ -0,0 +1,264 @@ +# Compare the final Estimation Mode Outputs from the three SIMOR model areas. + +name: "SIMOR Estimation Mode Outputs Comparison" +root: ../../simor_project_outputs/comparison_outputs +log_level: INFO + +pipeline: + steps: + - summarize + - dashboard + dashboard_mode: export + refresh: [] + +runs: + # The area-specific dashboards must run first and produce these prepared tables. + - label: Metro + prepared_table_map: + households: ../../simor_project_outputs/metro_outputs/estimation-mode-outputs/prepared_tables/households.parquet + persons: ../../simor_project_outputs/metro_outputs/estimation-mode-outputs/prepared_tables/persons.parquet + day: ../../simor_project_outputs/metro_outputs/estimation-mode-outputs/prepared_tables/day.parquet + tours: ../../simor_project_outputs/metro_outputs/estimation-mode-outputs/prepared_tables/tours.parquet + trips: ../../simor_project_outputs/metro_outputs/estimation-mode-outputs/prepared_tables/trips.parquet + trip_hypothetical_skims: ../../simor_project_outputs/metro_outputs/estimation-mode-outputs/prepared_tables/trip_hypothetical_skims.parquet + tour_hypothetical_skims: ../../simor_project_outputs/metro_outputs/estimation-mode-outputs/prepared_tables/tour_hypothetical_skims.parquet + vehicles: ../../simor_project_outputs/metro_outputs/estimation-mode-outputs/prepared_tables/vehicles.parquet + joint_tour_participants: ../../simor_project_outputs/metro_outputs/estimation-mode-outputs/prepared_tables/joint_tour_participants.parquet + land_use: ../../simor_project_outputs/metro_outputs/estimation-mode-outputs/prepared_tables/land_use.parquet + + - label: LCOG + prepared_table_map: + households: ../../simor_project_outputs/lcog_outputs/estimation-mode-outputs/prepared_tables/households.parquet + persons: ../../simor_project_outputs/lcog_outputs/estimation-mode-outputs/prepared_tables/persons.parquet + day: ../../simor_project_outputs/lcog_outputs/estimation-mode-outputs/prepared_tables/day.parquet + tours: ../../simor_project_outputs/lcog_outputs/estimation-mode-outputs/prepared_tables/tours.parquet + trips: ../../simor_project_outputs/lcog_outputs/estimation-mode-outputs/prepared_tables/trips.parquet + trip_hypothetical_skims: ../../simor_project_outputs/lcog_outputs/estimation-mode-outputs/prepared_tables/trip_hypothetical_skims.parquet + tour_hypothetical_skims: ../../simor_project_outputs/lcog_outputs/estimation-mode-outputs/prepared_tables/tour_hypothetical_skims.parquet + vehicles: ../../simor_project_outputs/lcog_outputs/estimation-mode-outputs/prepared_tables/vehicles.parquet + joint_tour_participants: ../../simor_project_outputs/lcog_outputs/estimation-mode-outputs/prepared_tables/joint_tour_participants.parquet + land_use: ../../simor_project_outputs/lcog_outputs/estimation-mode-outputs/prepared_tables/land_use.parquet + + - label: SKATS + prepared_table_map: + households: ../../simor_project_outputs/skats_outputs/estimation-mode-outputs/prepared_tables/households.parquet + persons: ../../simor_project_outputs/skats_outputs/estimation-mode-outputs/prepared_tables/persons.parquet + day: ../../simor_project_outputs/skats_outputs/estimation-mode-outputs/prepared_tables/day.parquet + tours: ../../simor_project_outputs/skats_outputs/estimation-mode-outputs/prepared_tables/tours.parquet + trips: ../../simor_project_outputs/skats_outputs/estimation-mode-outputs/prepared_tables/trips.parquet + trip_hypothetical_skims: ../../simor_project_outputs/skats_outputs/estimation-mode-outputs/prepared_tables/trip_hypothetical_skims.parquet + tour_hypothetical_skims: ../../simor_project_outputs/skats_outputs/estimation-mode-outputs/prepared_tables/tour_hypothetical_skims.parquet + vehicles: ../../simor_project_outputs/skats_outputs/estimation-mode-outputs/prepared_tables/vehicles.parquet + joint_tour_participants: ../../simor_project_outputs/skats_outputs/estimation-mode-outputs/prepared_tables/joint_tour_participants.parquet + land_use: ../../simor_project_outputs/skats_outputs/estimation-mode-outputs/prepared_tables/land_use.parquet + +zones: + use_maz: true + maz_col: [MAZ, zone_id] + taz_col: TAZ + +columns: + household_id: [household_id, hh_id] + trip_id: [trip_id, linked_trip_id] + trip_mode: [trip_mode, linked_trip_mode] + trip_num: [trip_num, linked_trip_num] + tour_purpose: [primary_purpose, tour_type, purpose] + +prepare: + output: + file_format: parquet + validation: + relationship_checks: warn + time_periods: + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + trip_period_number_column: depart + tour_start_period_number_column: start + tour_end_period_number_column: end + auto_sufficiency_basis: licensed_drivers + vot_bins: + source_column: income_segment + output_column: vot_bin + fallback_value: M + mappings: + metro: &estimation_vot_bins + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 995: M + 999: M + lcog: *estimation_vot_bins + skats: *estimation_vot_bins + +summarize: + weighting_modes: [unweighted, weighted] + pnr_tour_modes: + - PNR_TRANSIT + +dashboard: + title: "SIMOR Estimation Mode Outputs Comparison" + enable_maz_geographies: false + live: + pages: + - overview + - long_term_choices + - daily_travel + - joint_travel + - tour_summaries + - trip_summaries + - skim_summaries + export: + output_path: ../../simor_project_outputs/exports/estimation_mode_outputs_comparison.html + exclude_pages: + - tour_distance + - trip_stop_distance + dashboard: + weighting: [unweighted] + values: [percent, count] + pages: + long_term_choices: + mandatory_location_choice: + geography_level: [All Geography Types] + geography: [All] + +display: + bar_hover_mode: all + density_hover_mode: all + + # Color-blind-friendly Okabe-Ito colors, ordered to match the runs above. + run_colors: + - "#0072B2" # Metro: blue + - "#009E73" # LCOG: bluish green + - "#D55E00" # SKATS: vermillion + + labels: + person_type: + mapping: + all_person_types: All Person Types + 1: Full-time Worker + 2: Part-time Worker + 3: University Student + 4: Non-worker Adult + 5: Retired + 6: Student + 7: Preschool + + transit_subsidy: + mapping: + 0: No subsidy + 1: Discounted + 2: Free + + license_holding_status: + mapping: + has_license: Has License + no_license: No License + + transit_pass_ownership_status: + mapping: + has_transit_pass: Has Transit Pass + no_transit_pass: No Transit Pass + + telecommute_frequency: + mapping: + 1_day_week: 1 Day per Week + 2_days_week: 2 Days per Week + 3_days_week: 3 Days per Week + 2_3_days_week: 2-3 Days per Week + 4_days_week: 4+ Days per Week + No_Telecommute: No Telecommute + + mandatory_tour_frequency: + mapping: + 1: Work + 3: School + 2: 2 Work + 4: 2 School + 5: 1 Work + 1 School + + daily_activity_pattern: + mapping: + M: Mandatory + N: Non-Mandatory + H: At-Home + + tour_composition: + mapping: + adults: Adults + mixed: Mixed + children: Children + + atwork_subtour_frequency_category: + mapping: + no_subtours: None + eat: 1 Eating Out + business1: 1 Business + maint: 1 Maintenance + business2: 2 Business + eat_business: 1 Eating Out + 1 Business + + stop_purpose: + mapping: + work: Work + school: School + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + + mode: + mapping: + SOV: Drive Alone + HOV2: Shared Ride 2 + HOV3: Shared Ride 3+ + WALK: Walk + BIKE: Bike + EBIKE: E-Bike + ESCOOTER: E-Scooter + WALK_TRANSIT: Walk-Transit + BIKE_TRANSIT: Bike-Transit + PNR_TRANSIT: PNR-Transit + KNR_TRANSIT: KNR-Transit + TAXI: Taxi + TNC_SINGLE: TNC-Single + TNC_SHARED: TNC-Pool + SCHOOLBUS: School Bus + + tour_purpose: + mapping: + all_tour_purposes: All Tour Purposes + work: Work + school: School + joint: Joint + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + atwork: At-Work + loop: Loop + + trip_purpose: + mapping: + home: Home + work: Work + school: School + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + loop: Loop + + tour_category: + mapping: + mandatory: Mandatory + joint: Joint + non_mandatory: Non-Mandatory + atwork: At-Work diff --git a/simor_configs/lcog_configs/lcog_config.yaml b/simor_configs/lcog_configs/lcog_config.yaml new file mode 100644 index 0000000..d416caf --- /dev/null +++ b/simor_configs/lcog_configs/lcog_config.yaml @@ -0,0 +1,690 @@ +# ActivitySim Visualizer Configuration +# Adapt this file for each model deployment. +# Only the sections you need are required — see comments for which are optional. + +name: "LCOG Settings" +root: ../../simor_project_outputs/lcog_outputs # relative to the config's location +log_level: INFO + +pipeline: + steps: + - prepare + - skimjoin + # - segment + - summarize # will automatically overwrite summaries with a stale cache + - dashboard + dashboard_mode: export # live | export | host + refresh: [] # list stages here only when a forced rebuild is required + +# --------------------------------------------------------------------------- +# ActivitySim output file names +# Use stems (no extension) for automatic format detection: the reader will try +# .parquet first, then .csv. You may also specify an explicit extension to +# force a particular format (e.g. final_households.csv or final_tours.parquet). +# --------------------------------------------------------------------------- +files: + households: household + persons: person + day: day + tours: tour + trips: trip_linked + vehicles: vehicles + joint_tour_participants: joint_tour_participants + land_use: land_use + + + +# Optional shared fallback files for optional inputs that may be missing in some +# run folders. These must be explicit .csv or .parquet paths. +fallback_files: + land_use: U:/projects/clients/Oregon_HTS/SimOR/resident/outputs/estimation_lcog/final_land_use.csv + +# --------------------------------------------------------------------------- +# Runs to compare +# Each entry specifies a run directory, a display label, and optional overrides. +# +# Weight columns (all optional — leave as null or omit to use default rules): +# hh_weight_col: explicit household weight column in final_households +# person_weight_col: explicit person weight column in final_persons +# trip_weight_col: explicit trip weight column in final_trips +# (tour weight = average of its trips' weights when set) +# day_weight_col: explicit day-table weight column; set to null to inherit +# the corresponding person weight +# +# If none of the above are set and sample_rate is not in columns, all weights = 1. +# --------------------------------------------------------------------------- +runs: + - dir: U:/projects/clients/Oregon_HTS/hts_pyprocess/template/oregon/or_statewide_out + label: Raw Statewide Survey + hh_weight_col: hh_weight + person_weight_col: person_weight + file_map: + households: toc_household + persons: toc_person + day: toc_day + tours: toc_tour + trips: toc_trip_linked + joint_tour_participants: toc_joint_tour_participants + skimjoin: + enabled: false + + - dir: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/survey_data + label: "Regional: Filtered & Attributed" + hh_weight_col: hh_weight + person_weight_col: person_weight + day_weight_col: null + skimjoin: + config_path: lcog_skimjoin_config_alternate_id_col.yaml + + - dir: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/ + label: Estimation Mode Inputs + hh_weight_col: hh_weight + person_weight_col: person_weight + file_map: + households: override_households + persons: override_persons + tours: override_tours + trips: override_trips + joint_tour_participants: override_joint_tour_participants + + - dir: U:/projects/clients/Oregon_HTS/SimOR/resident/outputs/estimation_lcog/ + label: Estimation Mode Outputs + hh_weight_col: hh_weight + person_weight_col: person_weight + file_map: + households: final_households + persons: final_persons + tours: final_tours + trips: final_trips + vehicles: final_vehicles + joint_tour_participants: final_joint_tour_participants + land_use: final_land_use + + +# --------------------------------------------------------------------------- +# Zone system +# Set use_maz: false for TAZ-only models (zone IDs in outputs are already TAZs). +# For 2-zone (MAZ+TAZ) models, specify which columns in final_land_use +# hold the MAZ and TAZ IDs. +# --------------------------------------------------------------------------- +zones: + use_maz: true # false = TAZ-only model, no MAZ→TAZ conversion needed + maz_col: [MAZ, zone_id] # column in final_land_use containing MAZ IDs + taz_col: TAZ # column in final_land_use containing TAZ IDs + + +# --------------------------------------------------------------------------- +# Column names in the ActivitySim output files +# Change only if your outputs use non-standard column names. +# --------------------------------------------------------------------------- +columns: + household_id: [household_id, hh_id] + trip_id: [trip_id, linked_trip_id] + trip_mode: [trip_mode, linked_trip_mode] + trip_num: [trip_num, linked_trip_num] + ptype: ptype # person type column in persons file + hhsize: hhsize # HH size column in households file + auto_ownership: auto_ownership # vehicle count column in households file + num_workers: num_workers # workers per HH in households file + num_adults: num_adults # adults per HH in households file + # income_segment: [income_segment, income_broad] # OPTIONAL household income segment/level column for trip enrichment + # Prefer a readable purpose column per run. Some survey-style outputs store + # numeric codes in primary_purpose/tour_purpose and labels in tour_type. + tour_purpose: [primary_purpose, tour_type, purpose] + # sample_rate: sample_rate # OPTIONAL override; if omitted, code auto-detects + # a households column literally named 'sample_rate'. + # finalweight = 1/sample_rate (unless explicit run weight columns are provided) + # school_esc_outbound: [school_esc_outbound, out_escort_type] + # school_esc_inbound: [school_esc_inbound, inb_escort_type] + +prepare: + category_mappings: + raw_statewide_survey: + persons: + has_license: + source: can_drive + output_type: boolean + preserve_unmapped: false + mapping: + 1: true + 3: false + 995: false + transit_pass_ownership: + source: transit_pass + output_type: boolean + preserve_unmapped: false + mapping: + 1: false + 3: false + 4: false + 7: true + 8: true + 9: true + 995: null + telecommute_frequency: + source: telework_freq + preserve_unmapped: false + mapping: + 1: 4_days_week + 2: 4_days_week + 3: 4_days_week + 4: 3_days_week + 5: 2_days_week + 6: 1_day_week + 7: No_Telecommute + 8: No_Telecommute + 995: null + 996: null + work_from_home: + source: job_type + output_type: boolean + preserve_unmapped: false + mapping: + 1: false + 2: false + 3: true + 4: false + 5: false + 995: null + day: + cdap_activity: + source: daily_activity_pattern + preserve_unmapped: false + mapping: + 1: M + 2: N + 3: H + tours: + tour_mode: + mapping: &raw_statewide_tour_modes + 1: SCHOOLBUS + 2: WALK + 3: WALK_TRANSIT + 4: PNR_TRANSIT + 5: KNR_TRANSIT + 6: BIKE_TRANSIT + 7: BIKE + 8: EBIKE + 9: ESCOOTER + 10: TAXI + 11: TNC_SINGLE + 12: TNC_SHARED + 13: SOV + 14: HOV2 + 15: HOV3 + 16: OTHER + 995: MISSING + tour_purpose: + mapping: + -1: othdiscr + 1: loop + 2: work + 3: othmaint + 4: school + 5: othmaint + 6: escort + 7: shopping + 8: eatout + 9: social + 10: othmaint + 12: othdiscr + 13: othdiscr + 14: loop + 995: null + tour_category: + mapping: + 1: mandatory + 2: non_mandatory + 3: atwork + 4: joint + trips: + trip_mode: + source: linked_trip_mode + mapping: *raw_statewide_tour_modes + trip_purpose: + source: d_purpose_category + mapping: + -1: othdiscr + 1: home + 2: work + 3: othmaint + 4: school + 5: othmaint + 6: escort + 7: shopping + 8: eatout + 9: social + 10: othmaint + 12: othdiscr + 13: othdiscr + 14: home + 995: null + regional_filtered_attributed: + day: + cdap_activity: + source: daily_activity_pattern + preserve_unmapped: false + mapping: + 1: M + 2: N + 3: H + output: + file_format: parquet + validation: + relationship_checks: warn + distance_skim: + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/autoSkims__MD.omx # path relative to run directory, or absolute + matrix: SOV_H_DIST__MD + non_motorized_distance_skim: + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/maz_maz_walk.csv + matrix: null + time_periods: + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + trip_period_number_column: depart + tour_start_period_number_column: start + tour_end_period_number_column: end + auto_sufficiency_basis: licensed_drivers # licensed_drivers | workers | adults + vot_bins: + source_column: income_segment + output_column: vot_bin + fallback_value: M + mappings: + raw_statewide_survey: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 995: M + 999: M + survey: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 999: M + override: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 999: M + estimation: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 999: M + +skimjoin: + # Create optional long-form trip/tour tables with skim values for alternate modes. + create_hypothetical_skim_tables: true + defaults: + config_path: lcog_skimjoin_config.yaml + skim_files: + - 'U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/*.omx' + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/maz_stop_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/maz_maz_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/bike_maz_logsums_commute.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/bike_maz_logsums_noncommute.csv + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + +segment: + dashboard: + segmentation_type: person_sex + visibility: segments_only # full_only | segments_only | full_and_segments + definitions: + signup_platform: + include_full: true + persist_segmented_prepared_tables: false + allow_overlapping: false + on_empty_segment: warn + source: + type: prepared_column + source_table: hh + column: signup_platform + segments: + - id: rmove + label: RMove + values: ["rmove"] + - id: browser + label: Browser + values: ["browser"] + - id: call + label: Call + values: ["call"] + + person_sex: + include_full: false + persist_segmented_prepared_tables: false + allow_overlapping: false + on_empty_segment: warn + source: + type: prepared_column + source_table: per + column: SEX + segments: + - id: male + label: Male + values: [1] + - id: female + label: Female + values: [2] + +summarize: + weighting_modes: [unweighted, weighted] + pnr_tour_modes: + - PNR_TRANSIT + # OPTIONAL summary-time grouping for outputs with a tour_purpose dimension + # group_joint_tour_purposes: true + # group_atwork_tour_purposes: true + # group_school_tour_purposes: true + # Controls additional mapped geography aggregations only. Summaries may still + # emit all_geographies totals, and native prepared home geographies such as + # home_taz, home_county, and home_mpo can appear when those columns exist. + geography: + enabled: true + # Configured mappings create columns such as home_geo__school_district, + # work_geo__county, or land_use_geo__district. + aggregations: + school_district_k8: + source_zone_system: maz + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/land_use.csv + zone_id_col: MAZ + geography_col: DIST_KTO8 + school_district_9_12: + source_zone_system: maz + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/land_use.csv + zone_id_col: MAZ + geography_col: DIST_9TO12 + +dashboard: + title: "LCOG Estimation Visualizer" + logo: ../assets/lcog_logo.jpg + enable_maz_geographies: false + live: + pages: + - overview + - long_term_choices + - daily_travel + - joint_travel + - tour_summaries + - trip_summaries + - skim_summaries + # - validation: + # - traffic + # - transit + # - vmt + # - regional_validation + export: + output_path: ../../simor_project_outputs/exports/lcog_dashboard.html + # summary series are baked into one exported HTML file. + # `dashboard.export.pages` modifies matching live pages; it is not an + # inclusion list. Omit selector overrides to export all widget states. + # Export segmentation defaults to segmentation.dashboard when omitted. + # Use full_only, segments_only, or full_and_segments to choose which + dashboard: + weighting: [unweighted] + # segmentation_type: signup_platform + # segmentation_visibility: segments_only + pages: + long_term_choices: + mandatory_location_choice: + geography_level: + - All Geography Types + - County + - MPO + # omit TAZ + # parts: + # commuting_flows: + # enabled: false + + # shadow_pricing: + # enabled: true + # geography_level: [all] + # student_type: [all] + # parts: + # workplace_table: + # enabled: false + # school_table: + # enabled: false + # tour_summaries: + # internal_external_tours: + # enabled: false + validation: + vmt: + personal_auto_vmt_breakdown: all + personal_auto_vmt_geography_type: all + personal_auto_vmt_geography: default + personal_auto_vmt_time_period: default + personal_auto_vmt_mode: default + personal_auto_vmt_income_segment: default + personal_auto_vmt_household_size: default + non_motorized_vmt_breakdown: all + non_motorized_vmt_geography_type: all + non_motorized_vmt_geography: default + non_motorized_vmt_time_period: default + non_motorized_vmt_mode: default + non_motorized_vmt_income_segment: default + non_motorized_vmt_household_size: default + + external_travel_metric: all + external_travel_breakdown: all + external_travel_trip_purpose: default + external_travel_time_period: default + + demo_commercial_metric: all + demo_commercial_breakdown: all + demo_commercial_vehicle_type: default + demo_commercial_time_period: default + host: + account: my-connect-cloud-account + # app_id: 12345 + # title: Estimation Mode Comparison Visualizer + # First hosted publish may open a browser for Posit Connect Cloud login. + # Later runs usually reuse saved local rsconnect metadata. + verify: true + + +# Summary-affecting regrouping/normalization belongs under summarize.category_normalization. +# Cosmetic labels and ordering belong under display.labels. +display: + bar_hover_mode: all # closest | all + density_hover_mode: all # closest | all + labels: + person_type: + mapping: + all_person_types: All Person Types + 1: Full-time worker + 2: Part-time worker + 3: University student + 4: Non-worker adult + 5: Retired + 6: Student + 7: Preschool + + transit_subsidy: + mapping: + 0: No subsidy + 1: Discounted + 2: Free + + license_holding_status: + mapping: + has_license: Has License + no_license: No License + + transit_pass_ownership_status: + mapping: + has_transit_pass: Has Transit Pass + no_transit_pass: No Transit Pass + + telecommute_frequency: + mapping: + 1_day_week: 1 Day per Week + 2_days_week: 2 Days per Week + 3_days_week: 3 Days per Week + 2_3_days_week: 2-3 Days per Week + 4_days_week: 4+ Days per Week + No_Telecommute: No Telecommute + + mode: + mapping: + SOV: Drive Alone + HOV2: Shared Ride 2 + HOV3: Shared Ride 3+ + WALK: Walk + BIKE: Bike + EBIKE: E-Bike + ESCOOTER: E-Scooter + WALK_TRANSIT: Walk-Transit + BIKE_TRANSIT: Bike-Transit + PNR_TRANSIT: PNR-Transit + KNR_TRANSIT: KNR-Transit + TAXI: Taxi + TNC_SINGLE: TNC-Single + TNC_SHARED: TNC-Pool + SCHOOLBUS: School Bus + + + # escort: + # mapping: + # not_escorted: No Escort + # pure_escort: Pure Escort + # ride_share: Ride Share + # 0: No Escort + # 1: Pure Escort + # 2: Ride Share + # "": No Escort + + tour_purpose: + mapping: + all_tour_purposes: All Tour Purposes + work: Work + school: School + joint: Joint + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + atwork: At-Work + loop: Loop + + trip_purpose: + mapping: + home: Home + work: Work + school: School + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + loop: Loop + + stop_purpose: + mapping: + work: Work + school: School + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + + tour_composition: + mapping: + adults: Adults + mixed: Mixed + children: Children + + tour_category: + mapping: + mandatory: Mandatory + joint: Joint + non_mandatory: Non-Mandatory + atwork: At-Work + + atwork_subtour_frequency_category: + mapping: + no_subtours: None + eat: 1 Eating Out + business1: 1 Business + maint: 1 Maintenance + business2: 2 Business + eat_business: 1 Eating Out + 1 Business + + geography: + mapping: + all_geographies: All Geographies + county: County + home_county: County + home_mpo: MPO + home_taz: TAZ + district: District + taz: TAZ + maz: MAZ + school_district: School District + + mandatory_tour_frequency: + # mapping: + # work1: Work + # school1: School + # work2: 2 Work + # school2: 2 School + # work_and_school: 1 Work + 1 School + mapping: + 1: Work + 3: School + 2: 2 Work + 4: 2 School + 5: 1 Work + 1 School + + daily_activity_pattern: + mapping: + M : Mandatory + N : Non-Mandatory + H : At-Home + + facility_type: + mapping: + 1: "Interstate" + 3: "Principal Arterial" + 4: "Minor Arterial" + 5: "Major Collector" + 6: "Minor Collector" + 7: "Local Road" + 30: "Ramp" + 998: "Bike/Walk" + + commercial_vehicle_type: + mapping: + car: Car + su: Single-Unit Truck + mu: Multi-Unit Truck + + run_colors: + - "#298c8c" # Teal + - "#a00000" # Red + - "#b8b8b8" # Light gray + - "#384860" # Dark blue-gray + # - "#ff7f0e" # Orange + # - "#1f77b4" + # - "#ff7f0e" + # - "#2ca02c" + # - "#d62728" + # - "#9467bd" + # - "#8c564b" + # - "#e377c2" + # - "#7f7f7f" diff --git a/simor_configs/lcog_configs/lcog_skimjoin_config.yaml b/simor_configs/lcog_configs/lcog_skimjoin_config.yaml new file mode 100644 index 0000000..2a5d5c2 --- /dev/null +++ b/simor_configs/lcog_configs/lcog_skimjoin_config.yaml @@ -0,0 +1,356 @@ +project: + skim_files: + - 'U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/*.omx' + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/maz_stop_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/maz_maz_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/bike_maz_logsums_commute.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/bike_maz_logsums_noncommute.csv + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + +activitysim: + trip_mode_column: trip_mode + trip_id_column: trip_id + tour_mode_column: tour_mode + tour_id_column: tour_id + outbound_column: outbound + +defaults: + origin: OTAZ + destination: DTAZ + output_prefix: skim_ + sentinel_values: + - 9999 + - 999999 + +zone_mapping: + lookup_name: taz + file_lookup_names: + fares.omx: taz + missing_zone_policy: error + +dimensions: + PERIOD: + source_columns: + trip_source_column: trip_period + outbound_tour_source_column: start_period + inbound_tour_source_column: first_inbound_trip_period + VOT: + source_columns: + trip_source_column: vot_bin + outbound_tour_source_column: vot_bin + inbound_tour_source_column: vot_bin + values: + L: L + M: M + H: H + BIKE_PURPOSE: + source_columns: + trip_source_column: tour_purpose + outbound_tour_source_column: tour_purpose + inbound_tour_source_column: tour_purpose + values: + work: commute + school: commute + univ: commute + atwork: noncommute + business: noncommute + eat: noncommute + eatout: noncommute + escort: noncommute + loop: noncommute + maint: noncommute + othdiscr: noncommute + othmaint: noncommute + shopping: noncommute + social: noncommute + +ignore_modes: + - ESCOOTER + - MISSING + - OTHER + +modes: + SOV: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + HOV2: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + HOV3: + output_prefix: skim_auto_ + time: "SR3_{VOT}_TIME__{PERIOD}" + cost: "SR3_{VOT}_COST__{PERIOD}" + distance: "SR3_{VOT}_DIST__{PERIOD}" + KNR_TRANSIT: + output_prefix: skim_ + segment_on: outbound + segments: + true: + auto_time: "KTW_ACC__{PERIOD}" + auto_distance: "KTW_TDD__{PERIOD}" + transit_auxiliary_walk_time: "KTW_AUX__{PERIOD}" + transit_brt_ivtt: "KTW_BRT__{PERIOD}" + transit_bus_ivtt: "KTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "KTW_CRT__{PERIOD}" + walk_time: "KTW_EGR__{PERIOD}" + transit_first_wait_time: "KTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "KTW_LRT__{PERIOD}" + transit_tiv: "KTW_TIV__{PERIOD}" + transit_num_transfers: "KTW_XFR__{PERIOD}" + transit_transfer_wait_time: "KTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + false: + walk_time: "WTK_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTK_AUX__{PERIOD}" + transit_brt_ivtt: "WTK_BRT__{PERIOD}" + transit_bus_ivtt: "WTK_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTK_CRT__{PERIOD}" + auto_time: "WTK_EGR__{PERIOD}" + auto_distance: "WTK_TDD__{PERIOD}" + transit_first_wait_time: "WTK_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTK_LRT__{PERIOD}" + transit_tiv: "WTK_TIV__{PERIOD}" + transit_num_transfers: "WTK_XFR__{PERIOD}" + transit_transfer_wait_time: "WTK_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + PNR_TRANSIT: + output_prefix: skim_ + segment_on: outbound + segments: + true: + auto_time: "KTW_ACC__{PERIOD}" + auto_distance: "KTW_TDD__{PERIOD}" + transit_auxiliary_walk_time: "KTW_AUX__{PERIOD}" + transit_brt_ivtt: "KTW_BRT__{PERIOD}" + transit_bus_ivtt: "KTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "KTW_CRT__{PERIOD}" + walk_time: "KTW_EGR__{PERIOD}" + transit_first_wait_time: "KTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "KTW_LRT__{PERIOD}" + transit_tiv: "KTW_TIV__{PERIOD}" + transit_num_transfers: "KTW_XFR__{PERIOD}" + transit_transfer_wait_time: "KTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + false: + walk_time: "WTK_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTK_AUX__{PERIOD}" + transit_brt_ivtt: "WTK_BRT__{PERIOD}" + transit_bus_ivtt: "WTK_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTK_CRT__{PERIOD}" + auto_time: "WTK_EGR__{PERIOD}" + auto_distance: "WTK_TDD__{PERIOD}" + transit_first_wait_time: "WTK_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTK_LRT__{PERIOD}" + transit_tiv: "WTK_TIV__{PERIOD}" + transit_num_transfers: "WTK_XFR__{PERIOD}" + transit_transfer_wait_time: "WTK_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + SCHOOLBUS: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_ACC__{PERIOD}" + origin: OTAZ + destination: DTAZ + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_EGR__{PERIOD}" + origin: OTAZ + destination: DTAZ + TAXI: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + TNC_SHARED: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + TNC_SINGLE: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + WALK: + output_prefix: skim_walk_ + distance: + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__DISTWALK + fallbacks: + - matrix: WLK_DIST + origin: OTAZ + destination: DTAZ + maz_walk_actual: + output: skim_walk_maz_actual + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__actual + WALK_TRANSIT: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + BIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + logsum: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__logsum_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::logsum_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + EBIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + BIKE_TRANSIT: + output_prefix: skim_ + bike_o_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_d_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_o_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + bike_d_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" diff --git a/simor_configs/lcog_configs/lcog_skimjoin_config_alternate_id_col.yaml b/simor_configs/lcog_configs/lcog_skimjoin_config_alternate_id_col.yaml new file mode 100644 index 0000000..f80fa4c --- /dev/null +++ b/simor_configs/lcog_configs/lcog_skimjoin_config_alternate_id_col.yaml @@ -0,0 +1,356 @@ +project: + skim_files: + - 'U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/*.omx' + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/maz_stop_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/maz_maz_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/bike_maz_logsums_commute.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/lcog/data_full/bike_maz_logsums_noncommute.csv + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + +activitysim: + trip_mode_column: trip_mode + trip_id_column: linked_trip_id + tour_mode_column: tour_mode + tour_id_column: tour_id + outbound_column: outbound + +defaults: + origin: OTAZ + destination: DTAZ + output_prefix: skim_ + sentinel_values: + - 9999 + - 999999 + +zone_mapping: + lookup_name: taz + file_lookup_names: + fares.omx: taz + missing_zone_policy: error + +dimensions: + PERIOD: + source_columns: + trip_source_column: trip_period + outbound_tour_source_column: start_period + inbound_tour_source_column: first_inbound_trip_period + VOT: + source_columns: + trip_source_column: vot_bin + outbound_tour_source_column: vot_bin + inbound_tour_source_column: vot_bin + values: + L: L + M: M + H: H + BIKE_PURPOSE: + source_columns: + trip_source_column: tour_purpose + outbound_tour_source_column: tour_purpose + inbound_tour_source_column: tour_purpose + values: + work: commute + school: commute + univ: commute + atwork: noncommute + business: noncommute + eat: noncommute + eatout: noncommute + escort: noncommute + loop: noncommute + maint: noncommute + othdiscr: noncommute + othmaint: noncommute + shopping: noncommute + social: noncommute + +ignore_modes: + - ESCOOTER + - MISSING + - OTHER + +modes: + SOV: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + HOV2: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + HOV3: + output_prefix: skim_auto_ + time: "SR3_{VOT}_TIME__{PERIOD}" + cost: "SR3_{VOT}_COST__{PERIOD}" + distance: "SR3_{VOT}_DIST__{PERIOD}" + KNR_TRANSIT: + output_prefix: skim_ + segment_on: outbound + segments: + true: + auto_time: "KTW_ACC__{PERIOD}" + auto_distance: "KTW_TDD__{PERIOD}" + transit_auxiliary_walk_time: "KTW_AUX__{PERIOD}" + transit_brt_ivtt: "KTW_BRT__{PERIOD}" + transit_bus_ivtt: "KTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "KTW_CRT__{PERIOD}" + walk_time: "KTW_EGR__{PERIOD}" + transit_first_wait_time: "KTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "KTW_LRT__{PERIOD}" + transit_tiv: "KTW_TIV__{PERIOD}" + transit_num_transfers: "KTW_XFR__{PERIOD}" + transit_transfer_wait_time: "KTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + false: + walk_time: "WTK_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTK_AUX__{PERIOD}" + transit_brt_ivtt: "WTK_BRT__{PERIOD}" + transit_bus_ivtt: "WTK_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTK_CRT__{PERIOD}" + auto_time: "WTK_EGR__{PERIOD}" + auto_distance: "WTK_TDD__{PERIOD}" + transit_first_wait_time: "WTK_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTK_LRT__{PERIOD}" + transit_tiv: "WTK_TIV__{PERIOD}" + transit_num_transfers: "WTK_XFR__{PERIOD}" + transit_transfer_wait_time: "WTK_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + PNR_TRANSIT: + output_prefix: skim_ + segment_on: outbound + segments: + true: + auto_time: "KTW_ACC__{PERIOD}" + auto_distance: "KTW_TDD__{PERIOD}" + transit_auxiliary_walk_time: "KTW_AUX__{PERIOD}" + transit_brt_ivtt: "KTW_BRT__{PERIOD}" + transit_bus_ivtt: "KTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "KTW_CRT__{PERIOD}" + walk_time: "KTW_EGR__{PERIOD}" + transit_first_wait_time: "KTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "KTW_LRT__{PERIOD}" + transit_tiv: "KTW_TIV__{PERIOD}" + transit_num_transfers: "KTW_XFR__{PERIOD}" + transit_transfer_wait_time: "KTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + false: + walk_time: "WTK_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTK_AUX__{PERIOD}" + transit_brt_ivtt: "WTK_BRT__{PERIOD}" + transit_bus_ivtt: "WTK_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTK_CRT__{PERIOD}" + auto_time: "WTK_EGR__{PERIOD}" + auto_distance: "WTK_TDD__{PERIOD}" + transit_first_wait_time: "WTK_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTK_LRT__{PERIOD}" + transit_tiv: "WTK_TIV__{PERIOD}" + transit_num_transfers: "WTK_XFR__{PERIOD}" + transit_transfer_wait_time: "WTK_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + SCHOOLBUS: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_ACC__{PERIOD}" + origin: OTAZ + destination: DTAZ + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_EGR__{PERIOD}" + origin: OTAZ + destination: DTAZ + TAXI: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + TNC_SHARED: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + TNC_SINGLE: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + WALK: + output_prefix: skim_walk_ + distance: + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__DISTWALK + fallbacks: + - matrix: WLK_DIST + origin: OTAZ + destination: DTAZ + maz_walk_actual: + output: skim_walk_maz_actual + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__actual + WALK_TRANSIT: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + BIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + logsum: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__logsum_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::logsum_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + EBIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + BIKE_TRANSIT: + output_prefix: skim_ + bike_o_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_d_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_o_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + bike_d_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" diff --git a/simor_configs/metro_configs/metro_config.yaml b/simor_configs/metro_configs/metro_config.yaml new file mode 100644 index 0000000..0334be2 --- /dev/null +++ b/simor_configs/metro_configs/metro_config.yaml @@ -0,0 +1,681 @@ +# ActivitySim Visualizer Configuration +# Adapt this file for each model deployment. +# Only the sections you need are required — see comments for which are optional. + +name: "Metro Settings" +root: ../../simor_project_outputs/metro_outputs # relative to the config's location +log_level: INFO + +pipeline: + steps: + - prepare + - skimjoin + # - segment + - summarize # will automatically overwrite summaries with a stale cache + - dashboard + dashboard_mode: export # live | export | host + refresh: [] # list stages here only when a forced rebuild is required + +# --------------------------------------------------------------------------- +# ActivitySim output file names +# Use stems (no extension) for automatic format detection: the reader will try +# .parquet first, then .csv. You may also specify an explicit extension to +# force a particular format (e.g. final_households.csv or final_tours.parquet). +# --------------------------------------------------------------------------- +files: + households: household + persons: person + day: day + tours: tour + trips: trip_linked + vehicles: vehicles + joint_tour_participants: joint_tour_participants + land_use: land_use + + + +# Optional shared fallback files for optional inputs that may be missing in some +# run folders. These must be explicit .csv or .parquet paths. +fallback_files: + land_use: U:/projects/clients/Oregon_HTS/SimOR/resident/outputs/estimation_metro/final_land_use.csv + +# --------------------------------------------------------------------------- +# Runs to compare +# Each entry specifies a run directory, a display label, and optional overrides. +# +# Weight columns (all optional — leave as null or omit to use default rules): +# hh_weight_col: explicit household weight column in final_households +# person_weight_col: explicit person weight column in final_persons +# trip_weight_col: explicit trip weight column in final_trips +# (tour weight = average of its trips' weights when set) +# day_weight_col: explicit day-table weight column; set to null to inherit +# the corresponding person weight +# +# If none of the above are set and sample_rate is not in columns, all weights = 1. +# --------------------------------------------------------------------------- +runs: + + - dir: U:/projects/clients/Oregon_HTS/hts_pyprocess/template/oregon/or_statewide_out + label: Raw Statewide Survey + hh_weight_col: hh_weight + person_weight_col: person_weight + file_map: + households: toc_household + persons: toc_person + day: toc_day + tours: toc_tour + trips: toc_trip_linked + joint_tour_participants: toc_joint_tour_participants + skimjoin: + enabled: false + + - dir: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/survey_data + label: "Regional: Filtered & Attributed" + hh_weight_col: hh_weight + person_weight_col: person_weight + day_weight_col: null + skimjoin: + config_path: metro_skimjoin_config_alternate_id_col.yaml + + + + - dir: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full + label: Estimation Mode Inputs + hh_weight_col: hh_weight + person_weight_col: person_weight + file_map: + households: override_households + persons: override_persons + tours: override_tours + trips: override_trips + joint_tour_participants: override_joint_tour_participants + + - dir: U:/projects/clients/Oregon_HTS/SimOR/resident/outputs/estimation_metro + label: Estimation Mode Outputs + hh_weight_col: hh_weight + person_weight_col: person_weight + file_map: + households: final_households + persons: final_persons + tours: final_tours + trips: final_trips + vehicles: final_vehicles + joint_tour_participants: final_joint_tour_participants + land_use: final_land_use + +# --------------------------------------------------------------------------- +# Zone system +# Set use_maz: false for TAZ-only models (zone IDs in outputs are already TAZs). +# For 2-zone (MAZ+TAZ) models, specify which columns in final_land_use +# hold the MAZ and TAZ IDs. +# --------------------------------------------------------------------------- +zones: + use_maz: true # false = TAZ-only model, no MAZ→TAZ conversion needed + maz_col: [MAZ, zone_id] # column in final_land_use containing MAZ IDs + taz_col: TAZ # column in final_land_use containing TAZ IDs + + +# --------------------------------------------------------------------------- +# Column names in the ActivitySim output files +# Change only if your outputs use non-standard column names. +# --------------------------------------------------------------------------- +columns: + household_id: [household_id, hh_id] + trip_id: [trip_id, linked_trip_id] + trip_mode: [trip_mode, linked_trip_mode] + trip_num: [trip_num, linked_trip_num] + ptype: ptype # person type column in persons file + hhsize: hhsize # HH size column in households file + auto_ownership: auto_ownership # vehicle count column in households file + num_workers: num_workers # workers per HH in households file + num_adults: num_adults # adults per HH in households file + # income_segment: [income_segment, income_broad] # OPTIONAL household income segment/level column for trip enrichment + # Prefer a readable purpose column per run. Some survey-style outputs store + # numeric codes in primary_purpose/tour_purpose and labels in tour_type. + tour_purpose: [primary_purpose, tour_type, purpose] + # sample_rate: sample_rate # OPTIONAL override; if omitted, code auto-detects + # a households column literally named 'sample_rate'. + # finalweight = 1/sample_rate (unless explicit run weight columns are provided) + # school_esc_outbound: [school_esc_outbound, out_escort_type] + # school_esc_inbound: [school_esc_inbound, inb_escort_type] + +prepare: + category_mappings: + raw_statewide_survey: + persons: + has_license: + source: can_drive + output_type: boolean + preserve_unmapped: false + mapping: + 1: true + 3: false + 995: false + transit_pass_ownership: + source: transit_pass + output_type: boolean + preserve_unmapped: false + mapping: + 1: false + 3: false + 4: false + 7: true + 8: true + 9: true + 995: null + telecommute_frequency: + source: telework_freq + preserve_unmapped: false + mapping: + 1: 4_days_week + 2: 4_days_week + 3: 4_days_week + 4: 3_days_week + 5: 2_days_week + 6: 1_day_week + 7: No_Telecommute + 8: No_Telecommute + 995: null + 996: null + work_from_home: + source: job_type + output_type: boolean + preserve_unmapped: false + mapping: + 1: false + 2: false + 3: true + 4: false + 5: false + 995: null + day: + cdap_activity: + source: daily_activity_pattern + preserve_unmapped: false + mapping: + 1: M + 2: N + 3: H + tours: + tour_mode: + mapping: &raw_statewide_tour_modes + 1: SCHOOLBUS + 2: WALK + 3: WALK_TRANSIT + 4: PNR_TRANSIT + 5: KNR_TRANSIT + 6: BIKE_TRANSIT + 7: BIKE + 8: EBIKE + 9: ESCOOTER + 10: TAXI + 11: TNC_SINGLE + 12: TNC_SHARED + 13: SOV + 14: HOV2 + 15: HOV3 + 16: OTHER + 995: MISSING + tour_purpose: + mapping: + -1: othdiscr + 1: loop + 2: work + 3: othmaint + 4: school + 5: othmaint + 6: escort + 7: shopping + 8: eatout + 9: social + 10: othmaint + 12: othdiscr + 13: othdiscr + 14: loop + 995: null + tour_category: + mapping: + 1: mandatory + 2: non_mandatory + 3: atwork + 4: joint + trips: + trip_mode: + source: linked_trip_mode + mapping: *raw_statewide_tour_modes + trip_purpose: + source: d_purpose_category + mapping: + -1: othdiscr + 1: home + 2: work + 3: othmaint + 4: school + 5: othmaint + 6: escort + 7: shopping + 8: eatout + 9: social + 10: othmaint + 12: othdiscr + 13: othdiscr + 14: home + 995: null + regional_filtered_attributed: + day: + cdap_activity: + source: daily_activity_pattern + preserve_unmapped: false + mapping: + 1: M + 2: N + 3: H + output: + file_format: parquet + validation: + relationship_checks: warn + distance_skim: + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/autoSkims__MD.omx # path relative to run directory, or absolute + matrix: SOV_H_DIST__MD + non_motorized_distance_skim: + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/maz_maz_walk.csv + matrix: null + time_periods: + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + trip_period_number_column: depart + tour_start_period_number_column: start + tour_end_period_number_column: end + auto_sufficiency_basis: licensed_drivers # licensed_drivers | workers | adults + vot_bins: + source_column: income_segment + output_column: vot_bin + fallback_value: M + mappings: + raw_statewide_survey: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 995: M + 999: M + survey: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 999: M + override: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 999: M + estimation: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 999: M + +skimjoin: + # Create optional long-form trip/tour tables with skim values for alternate modes. + create_hypothetical_skim_tables: true + defaults: + config_path: metro_skimjoin_config.yaml + skim_files: + - 'U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/*.omx' + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/maz_stop_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/maz_maz_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/bike_maz_logsums_commute.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/bike_maz_logsums_noncommute.csv + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + +segment: + dashboard: + segmentation_type: person_sex + visibility: segments_only # full_only | segments_only | full_and_segments + definitions: + signup_platform: + include_full: true + persist_segmented_prepared_tables: false + allow_overlapping: false + on_empty_segment: warn + source: + type: prepared_column + source_table: hh + column: signup_platform + segments: + - id: rmove + label: RMove + values: ["rmove"] + - id: browser + label: Browser + values: ["browser"] + - id: call + label: Call + values: ["call"] + + +summarize: + weighting_modes: [unweighted, weighted] + pnr_tour_modes: + - PNR_TRANSIT + # OPTIONAL summary-time grouping for outputs with a tour_purpose dimension + # group_joint_tour_purposes: true + # group_atwork_tour_purposes: true + # group_school_tour_purposes: true + # Controls additional mapped geography aggregations only. Summaries may still + # emit all_geographies totals, and native prepared home geographies such as + # home_taz, home_county, and home_mpo can appear when those columns exist. + geography: + enabled: true + # Configured mappings create columns such as home_geo__school_district, + # work_geo__county, or land_use_geo__district. + aggregations: + regional_district: + source_zone_system: maz + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/land_use.csv + zone_id_col: MAZ + geography_col: DISTRICT9 + school_district_k8: + source_zone_system: maz + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/land_use.csv + zone_id_col: MAZ + geography_col: DIST_Kto8 + school_district_9_12: + source_zone_system: maz + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/land_use.csv + zone_id_col: MAZ + geography_col: DIST_9to12 + +dashboard: + title: "Metro Estimation Visualizer" + logo: ../assets/metro_logo.png + enable_maz_geographies: false + live: + pages: + - overview + - long_term_choices + - daily_travel + - joint_travel + - tour_summaries + - trip_summaries + - skim_summaries + # - validation: + # - traffic + # - transit + # - vmt + # - regional_validation + export: + output_path: ../../simor_project_outputs/exports/metro_dashboard.html + # summary series are baked into one exported HTML file. + # `dashboard.export.pages` modifies matching live pages; it is not an + # inclusion list. Omit selector overrides to export all widget states. + # Export segmentation defaults to segmentation.dashboard when omitted. + # Use full_only, segments_only, or full_and_segments to choose which + dashboard: + weighting: [unweighted] + # segmentation_type: signup_platform + # segmentation_visibility: segments_only + pages: + long_term_choices: + mandatory_location_choice: + geography_level: + - All Geography Types + - County + - MPO + # omit TAZ + # parts: + # commuting_flows: + # enabled: false + + # shadow_pricing: + # enabled: true + # geography_level: [all] + # student_type: [all] + # parts: + # workplace_table: + # enabled: false + # school_table: + # enabled: false + # tour_summaries: + # internal_external_tours: + # enabled: false + validation: + vmt: + personal_auto_vmt_breakdown: all + personal_auto_vmt_geography_type: all + personal_auto_vmt_geography: default + personal_auto_vmt_time_period: default + personal_auto_vmt_mode: default + personal_auto_vmt_income_segment: default + personal_auto_vmt_household_size: default + non_motorized_vmt_breakdown: all + non_motorized_vmt_geography_type: all + non_motorized_vmt_geography: default + non_motorized_vmt_time_period: default + non_motorized_vmt_mode: default + non_motorized_vmt_income_segment: default + non_motorized_vmt_household_size: default + + external_travel_metric: all + external_travel_breakdown: all + external_travel_trip_purpose: default + external_travel_time_period: default + + demo_commercial_metric: all + demo_commercial_breakdown: all + demo_commercial_vehicle_type: default + demo_commercial_time_period: default + host: + account: my-connect-cloud-account + # app_id: 12345 + # title: Estimation Mode Comparison Visualizer + # First hosted publish may open a browser for Posit Connect Cloud login. + # Later runs usually reuse saved local rsconnect metadata. + verify: true + + +# Summary-affecting regrouping/normalization belongs under summarize.category_normalization. +# Cosmetic labels and ordering belong under display.labels. +display: + bar_hover_mode: all # closest | all + density_hover_mode: all # closest | all + labels: + person_type: + mapping: + all_person_types: All Person Types + 1: Full-time worker + 2: Part-time worker + 3: University student + 4: Non-worker adult + 5: Retired + 6: Student + 7: Preschool + + transit_subsidy: + mapping: + 0: No subsidy + 1: Discounted + 2: Free + + license_holding_status: + mapping: + has_license: Has License + no_license: No License + + transit_pass_ownership_status: + mapping: + has_transit_pass: Has Transit Pass + no_transit_pass: No Transit Pass + + telecommute_frequency: + mapping: + 1_day_week: 1 Day per Week + 2_days_week: 2 Days per Week + 3_days_week: 3 Days per Week + 2_3_days_week: 2-3 Days per Week + 4_days_week: 4+ Days per Week + No_Telecommute: No Telecommute + + mode: + mapping: + SOV: Drive Alone + HOV2: Shared Ride 2 + HOV3: Shared Ride 3+ + WALK: Walk + BIKE: Bike + EBIKE: E-Bike + ESCOOTER: E-Scooter + WALK_TRANSIT: Walk-Transit + BIKE_TRANSIT: Bike-Transit + PNR_TRANSIT: PNR-Transit + KNR_TRANSIT: KNR-Transit + TAXI: Taxi + TNC_SINGLE: TNC-Single + TNC_SHARED: TNC-Pool + SCHOOLBUS: School Bus + + + # escort: + # mapping: + # not_escorted: No Escort + # pure_escort: Pure Escort + # ride_share: Ride Share + # 0: No Escort + # 1: Pure Escort + # 2: Ride Share + # "": No Escort + + tour_purpose: + mapping: + all_tour_purposes: All Tour Purposes + work: Work + school: School + joint: Joint + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + atwork: At-Work + loop: Loop + + trip_purpose: + mapping: + home: Home + work: Work + school: School + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + loop: Loop + + stop_purpose: + mapping: + work: Work + school: School + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + + tour_composition: + mapping: + adults: Adults + mixed: Mixed + children: Children + + tour_category: + mapping: + mandatory: Mandatory + joint: Joint + non_mandatory: Non-Mandatory + atwork: At-Work + + atwork_subtour_frequency_category: + mapping: + no_subtours: None + eat: 1 Eating Out + business1: 1 Business + maint: 1 Maintenance + business2: 2 Business + eat_business: 1 Eating Out + 1 Business + + geography: + mapping: + all_geographies: All Geographies + county: County + home_county: County + home_mpo: MPO + home_taz: TAZ + district: District + taz: TAZ + maz: MAZ + school_district: School District + + mandatory_tour_frequency: + # mapping: + # work1: Work + # school1: School + # work2: 2 Work + # school2: 2 School + # work_and_school: 1 Work + 1 School + mapping: + 1: Work + 3: School + 2: 2 Work + 4: 2 School + 5: 1 Work + 1 School + + daily_activity_pattern: + mapping: + M : Mandatory + N : Non-Mandatory + H : At-Home + + facility_type: + mapping: + 1: "Interstate" + 3: "Principal Arterial" + 4: "Minor Arterial" + 5: "Major Collector" + 6: "Minor Collector" + 7: "Local Road" + 30: "Ramp" + 998: "Bike/Walk" + + commercial_vehicle_type: + mapping: + car: Car + su: Single-Unit Truck + mu: Multi-Unit Truck + + # run_colors: + # - "#298c8c" # Teal + # - "#a00000" # Red + # - "#b8b8b8" # Light gray + # - "#384860" # Dark blue-gray + # - "#ff7f0e" # Orange + # - "#1f77b4" + # - "#ff7f0e" + # - "#2ca02c" + # - "#d62728" + # - "#9467bd" + # - "#8c564b" + # - "#e377c2" + # - "#7f7f7f" diff --git a/simor_configs/metro_configs/metro_skimjoin_config.yaml b/simor_configs/metro_configs/metro_skimjoin_config.yaml new file mode 100644 index 0000000..ca4b095 --- /dev/null +++ b/simor_configs/metro_configs/metro_skimjoin_config.yaml @@ -0,0 +1,494 @@ +# project: +# skim_files: +# - 'C:\Users\wesley.darling\OneDrive - Resource Systems Group, Inc\odot_joint_estimation_data\metro_data\*.omx' +# - C:\Users\wesley.darling\OneDrive - Resource Systems Group, Inc\odot_joint_estimation_data\metro_data\maz_stop_walk.csv +# - C:\Users\wesley.darling\OneDrive - Resource Systems Group, Inc\odot_joint_estimation_data\metro_data\maz_maz_walk.csv +# - C:\Users\wesley.darling\OneDrive - Resource Systems Group, Inc\odot_joint_estimation_data\metro_data\bike_maz_logsums_commute.csv +# - C:\Users\wesley.darling\OneDrive - Resource Systems Group, Inc\odot_joint_estimation_data\metro_data\bike_maz_logsums_noncommute.csv + +# network_los_file: C:\Users\wesley.darling\project_data\odot_skims\network_los.yaml + + # project.trips_table / project.tours_table / project.output_dir are used by the + # standalone skimjoin CLI only. Integrated runtime skimjoin reads prepared + # in-memory trips/tours from the processor pipeline instead. + # trips_table: C:\Users\wesley.darling\Downloads\viz\viz\output\trip_linked.csv + # trips_table: C:\Users\wesley.darling\projects\activitysim_visualizer\artifacts\prepared_cache\override\trips.parquet + # tours_table: C:\Users\wesley.darling\Downloads\viz\viz\output\tour.csv + # tours_table: C:\Users\wesley.darling\projects\activitysim_visualizer\artifacts\prepared_cache\override\tours.parquet + # output_dir: metro_output + + +activitysim: + trip_mode_column: trip_mode + trip_id_column: trip_id + tour_mode_column: tour_mode + tour_id_column: tour_id + outbound_column: outbound + + +defaults: + origin: OTAZ + destination: DTAZ + output_prefix: skim_ + sentinel_values: + - 9999 + - 999999 + +zone_mapping: + lookup_name: taz + file_lookup_names: + fares.omx: zone_number + 'bike_taz_logsums_*.omx': TAZ + missing_zone_policy: error + +dimensions: + PERIOD: + source_columns: + trip_source_column: trip_period + outbound_tour_source_column: start_period + inbound_tour_source_column: first_inbound_trip_period + VOT: + source_columns: + trip_source_column: vot_bin + outbound_tour_source_column: vot_bin + inbound_tour_source_column: vot_bin + values: + L: L + M: M + H: H + BIKE_PURPOSE: + source_columns: + trip_source_column: tour_purpose + outbound_tour_source_column: tour_purpose + inbound_tour_source_column: tour_purpose + values: + work: commute + school: commute + univ: commute + atwork: noncommute + business: noncommute + eat: noncommute + eatout: noncommute + escort: noncommute + loop: noncommute + maint: noncommute + othdiscr: noncommute + othmaint: noncommute + shopping: noncommute + social: noncommute + +ignore_modes: + - ESCOOTER + - MISSING + - OTHER + +modes: + SOV: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + HOV2: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + HOV3: + output_prefix: skim_auto_ + time: "SR3_{VOT}_TIME__{PERIOD}" + cost: "SR3_{VOT}_COST__{PERIOD}" + distance: "SR3_{VOT}_DIST__{PERIOD}" + KNR_TRANSIT: + output_prefix: skim_ + segment_on: outbound + segments: + true: + auto_time: "KTW_ACC__{PERIOD}" + auto_distance: "KTW_TDD__{PERIOD}" + transit_auxiliary_walk_time: "KTW_AUX__{PERIOD}" + transit_brt_ivtt: "KTW_BRT__{PERIOD}" + transit_bus_ivtt: "KTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "KTW_CRT__{PERIOD}" + walk_time: "KTW_EGR__{PERIOD}" + transit_first_wait_time: "KTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "KTW_LRT__{PERIOD}" + transit_tiv: "KTW_TIV__{PERIOD}" + transit_num_transfers: "KTW_XFR__{PERIOD}" + transit_transfer_wait_time: "KTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + false: + walk_time: "WTK_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTK_AUX__{PERIOD}" + transit_brt_ivtt: "WTK_BRT__{PERIOD}" + transit_bus_ivtt: "WTK_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTK_CRT__{PERIOD}" + auto_time: "WTK_EGR__{PERIOD}" + auto_distance: "WTK_TDD__{PERIOD}" + transit_first_wait_time: "WTK_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTK_LRT__{PERIOD}" + transit_tiv: "WTK_TIV__{PERIOD}" + transit_num_transfers: "WTK_XFR__{PERIOD}" + transit_transfer_wait_time: "WTK_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + PNR_TRANSIT: + output_prefix: skim_ + segment_on: outbound + segments: + true: + auto_time: + matrix: "SOV_{VOT}_TIME__{PERIOD}" + origin: OTAZ + destination: pnr_taz + auto_distance: + matrix: "SOV_{VOT}_DIST__{PERIOD}" + origin: OTAZ + destination: pnr_taz + auto_cost: + matrix: "SOV_{VOT}_COST__{PERIOD}" + origin: OTAZ + destination: pnr_taz + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_auxiliary_walk_time: + matrix: "WTW_AUX__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_brt_ivtt: + matrix: "WTW_BRT__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_bus_ivtt: + matrix: "WTW_BUS__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_commuter_rail_ivtt: + matrix: "WTW_CRT__{PERIOD}" + origin: pnr_taz + destination: DTAZ + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_first_wait_time: + matrix: "WTW_FWT__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_light_rail_ivtt: + matrix: "WTW_LRT__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_tiv: + matrix: "WTW_TIV__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_num_transfers: + matrix: "WTW_XFR__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_transfer_wait_time: + matrix: "WTW_XWT__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_fare: + matrix: "fare__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: pnr_zone_id + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: pnr_zone_id + matrix: maz_stop_walk__walk_dist_premium_transit + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + false: + auto_time: + matrix: "SOV_{VOT}_TIME__{PERIOD}" + origin: pnr_taz + destination: DTAZ + auto_distance: + matrix: "SOV_{VOT}_DIST__{PERIOD}" + origin: pnr_taz + destination: DTAZ + auto_cost: + matrix: "SOV_{VOT}_COST__{PERIOD}" + origin: pnr_taz + destination: DTAZ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_auxiliary_walk_time: + matrix: "WTW_AUX__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_brt_ivtt: + matrix: "WTW_BRT__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_bus_ivtt: + matrix: "WTW_BUS__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_commuter_rail_ivtt: + matrix: "WTW_CRT__{PERIOD}" + origin: OTAZ + destination: pnr_taz + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_first_wait_time: + matrix: "WTW_FWT__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_light_rail_ivtt: + matrix: "WTW_LRT__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_tiv: + matrix: "WTW_TIV__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_num_transfers: + matrix: "WTW_XFR__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_transfer_wait_time: + matrix: "WTW_XWT__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_fare: + matrix: "fare__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: pnr_zone_id + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: pnr_zone_id + matrix: maz_stop_walk__walk_dist_premium_transit + + SCHOOLBUS: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_ACC__{PERIOD}" + origin: OTAZ + destination: DTAZ + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_EGR__{PERIOD}" + origin: OTAZ + destination: DTAZ + TAXI: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + TNC_SHARED: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + TNC_SINGLE: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + WALK: + output_prefix: skim_walk_ + distance: + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__DISTWALK + fallbacks: + - matrix: WLK_DIST + origin: OTAZ + destination: DTAZ + maz_walk_actual: + output: skim_walk_maz_actual + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__actual + WALK_TRANSIT: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + BIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + logsum: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__logsum_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::logsum_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + EBIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + BIKE_TRANSIT: + output_prefix: skim_ + bike_o_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_d_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_o_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + bike_d_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + + diff --git a/simor_configs/metro_configs/metro_skimjoin_config_alternate_id_col.yaml b/simor_configs/metro_configs/metro_skimjoin_config_alternate_id_col.yaml new file mode 100644 index 0000000..3c0dc99 --- /dev/null +++ b/simor_configs/metro_configs/metro_skimjoin_config_alternate_id_col.yaml @@ -0,0 +1,494 @@ +project: + skim_files: + - 'U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/*.omx' + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/maz_stop_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/maz_maz_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/bike_maz_logsums_commute.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/metro/data_full/bike_maz_logsums_noncommute.csv + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + # project.trips_table / project.tours_table / project.output_dir are used by the + # standalone skimjoin CLI only. Integrated runtime skimjoin reads prepared + # in-memory trips/tours from the processor pipeline instead. + # trips_table: C:\Users\wesley.darling\Downloads\viz\viz\output\trip_linked.csv + # trips_table: C:\Users\wesley.darling\projects\activitysim_visualizer\artifacts\prepared_cache\override\trips.parquet + # tours_table: C:\Users\wesley.darling\Downloads\viz\viz\output\tour.csv + # tours_table: C:\Users\wesley.darling\projects\activitysim_visualizer\artifacts\prepared_cache\override\tours.parquet + # output_dir: metro_output + + +activitysim: + # activitysim now only defines structural trip/tour fields used by skimjoin. + trip_mode_column: trip_mode + trip_id_column: linked_trip_id + tour_mode_column: tour_mode + tour_id_column: tour_id + outbound_column: outbound + +# Integrated runtime currently supports OMX skim inputs only. + +defaults: + origin: OTAZ + destination: DTAZ + output_prefix: skim_ + sentinel_values: + - 9999 + - 999999 + +zone_mapping: + lookup_name: taz + file_lookup_names: + fares.omx: zone_number + 'bike_taz_logsums_*.omx': TAZ + missing_zone_policy: error + +dimensions: + PERIOD: + source_columns: + trip_source_column: trip_period + outbound_tour_source_column: start_period + inbound_tour_source_column: first_inbound_trip_period + VOT: + source_columns: + trip_source_column: vot_bin + outbound_tour_source_column: vot_bin + inbound_tour_source_column: vot_bin + values: + L: L + M: M + H: H + BIKE_PURPOSE: + source_columns: + trip_source_column: tour_purpose + outbound_tour_source_column: tour_purpose + inbound_tour_source_column: tour_purpose + values: + work: commute + school: commute + univ: commute + atwork: noncommute + business: noncommute + eat: noncommute + eatout: noncommute + escort: noncommute + loop: noncommute + maint: noncommute + othdiscr: noncommute + othmaint: noncommute + shopping: noncommute + social: noncommute + +ignore_modes: + - ESCOOTER + - MISSING + - OTHER + +modes: + SOV: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + HOV2: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + HOV3: + output_prefix: skim_auto_ + time: "SR3_{VOT}_TIME__{PERIOD}" + cost: "SR3_{VOT}_COST__{PERIOD}" + distance: "SR3_{VOT}_DIST__{PERIOD}" + KNR_TRANSIT: + output_prefix: skim_ + segment_on: outbound + segments: + true: + auto_time: "KTW_ACC__{PERIOD}" + auto_distance: "KTW_TDD__{PERIOD}" + transit_auxiliary_walk_time: "KTW_AUX__{PERIOD}" + transit_brt_ivtt: "KTW_BRT__{PERIOD}" + transit_bus_ivtt: "KTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "KTW_CRT__{PERIOD}" + walk_time: "KTW_EGR__{PERIOD}" + transit_first_wait_time: "KTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "KTW_LRT__{PERIOD}" + transit_tiv: "KTW_TIV__{PERIOD}" + transit_num_transfers: "KTW_XFR__{PERIOD}" + transit_transfer_wait_time: "KTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + false: + walk_time: "WTK_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTK_AUX__{PERIOD}" + transit_brt_ivtt: "WTK_BRT__{PERIOD}" + transit_bus_ivtt: "WTK_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTK_CRT__{PERIOD}" + auto_time: "WTK_EGR__{PERIOD}" + auto_distance: "WTK_TDD__{PERIOD}" + transit_first_wait_time: "WTK_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTK_LRT__{PERIOD}" + transit_tiv: "WTK_TIV__{PERIOD}" + transit_num_transfers: "WTK_XFR__{PERIOD}" + transit_transfer_wait_time: "WTK_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + PNR_TRANSIT: + output_prefix: skim_ + segment_on: outbound + segments: + true: + auto_time: + matrix: "SOV_{VOT}_TIME__{PERIOD}" + origin: OTAZ + destination: pnr_taz + auto_distance: + matrix: "SOV_{VOT}_DIST__{PERIOD}" + origin: OTAZ + destination: pnr_taz + auto_cost: + matrix: "SOV_{VOT}_COST__{PERIOD}" + origin: OTAZ + destination: pnr_taz + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_auxiliary_walk_time: + matrix: "WTW_AUX__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_brt_ivtt: + matrix: "WTW_BRT__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_bus_ivtt: + matrix: "WTW_BUS__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_commuter_rail_ivtt: + matrix: "WTW_CRT__{PERIOD}" + origin: pnr_taz + destination: DTAZ + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_first_wait_time: + matrix: "WTW_FWT__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_light_rail_ivtt: + matrix: "WTW_LRT__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_tiv: + matrix: "WTW_TIV__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_num_transfers: + matrix: "WTW_XFR__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_transfer_wait_time: + matrix: "WTW_XWT__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_fare: + matrix: "fare__{PERIOD}" + origin: pnr_taz + destination: DTAZ + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: pnr_zone_id + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: pnr_zone_id + matrix: maz_stop_walk__walk_dist_premium_transit + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + false: + auto_time: + matrix: "SOV_{VOT}_TIME__{PERIOD}" + origin: pnr_taz + destination: DTAZ + auto_distance: + matrix: "SOV_{VOT}_DIST__{PERIOD}" + origin: pnr_taz + destination: DTAZ + auto_cost: + matrix: "SOV_{VOT}_COST__{PERIOD}" + origin: pnr_taz + destination: DTAZ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_auxiliary_walk_time: + matrix: "WTW_AUX__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_brt_ivtt: + matrix: "WTW_BRT__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_bus_ivtt: + matrix: "WTW_BUS__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_commuter_rail_ivtt: + matrix: "WTW_CRT__{PERIOD}" + origin: OTAZ + destination: pnr_taz + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_first_wait_time: + matrix: "WTW_FWT__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_light_rail_ivtt: + matrix: "WTW_LRT__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_tiv: + matrix: "WTW_TIV__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_num_transfers: + matrix: "WTW_XFR__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_transfer_wait_time: + matrix: "WTW_XWT__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_fare: + matrix: "fare__{PERIOD}" + origin: OTAZ + destination: pnr_taz + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: pnr_zone_id + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: pnr_zone_id + matrix: maz_stop_walk__walk_dist_premium_transit + + SCHOOLBUS: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_ACC__{PERIOD}" + origin: OTAZ + destination: DTAZ + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_EGR__{PERIOD}" + origin: OTAZ + destination: DTAZ + TAXI: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + TNC_SHARED: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + TNC_SINGLE: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + WALK: + output_prefix: skim_walk_ + distance: + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__DISTWALK + fallbacks: + - matrix: WLK_DIST + origin: OTAZ + destination: DTAZ + maz_walk_actual: + output: skim_walk_maz_actual + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__actual + WALK_TRANSIT: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + BIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + logsum: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__logsum_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::logsum_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + EBIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + BIKE_TRANSIT: + output_prefix: skim_ + bike_o_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_d_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_o_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + bike_d_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + + diff --git a/simor_configs/skats_configs/skats_config.yaml b/simor_configs/skats_configs/skats_config.yaml new file mode 100644 index 0000000..08446e5 --- /dev/null +++ b/simor_configs/skats_configs/skats_config.yaml @@ -0,0 +1,723 @@ +# ActivitySim Visualizer Configuration +# Adapt this file for each model deployment. +# Only the sections you need are required — see comments for which are optional. + +name: "SKATS Settings" +root: ../../simor_project_outputs/skats_outputs # relative to the config's location +log_level: INFO + +pipeline: + steps: + - prepare + - skimjoin + # - segment + - summarize # will automatically overwrite summaries with a stale cache + - dashboard + dashboard_mode: export # live | export | host + refresh: [] # list stages here only when a forced rebuild is required + +# --------------------------------------------------------------------------- +# ActivitySim output file names +# Use stems (no extension) for automatic format detection: the reader will try +# .parquet first, then .csv. You may also specify an explicit extension to +# force a particular format (e.g. final_households.csv or final_tours.parquet). +# --------------------------------------------------------------------------- +files: + households: household + persons: person + day: day + tours: tour + trips: trip_linked + vehicles: vehicles + joint_tour_participants: joint_tour_participants + land_use: land_use + + + +# Optional shared fallback files for optional inputs that may be missing in some +# run folders. These must be explicit .csv or .parquet paths. +fallback_files: + land_use: U:/projects/clients/Oregon_HTS/SimOR/resident/outputs/estimation_skats/final_land_use.csv + +# --------------------------------------------------------------------------- +# Runs to compare +# Each entry specifies a run directory, a display label, and optional overrides. +# +# Weight columns (all optional — leave as null or omit to use default rules): +# hh_weight_col: explicit household weight column in final_households +# person_weight_col: explicit person weight column in final_persons +# trip_weight_col: explicit trip weight column in final_trips +# (tour weight = average of its trips' weights when set) +# day_weight_col: explicit day-table weight column; set to null to inherit +# the corresponding person weight +# +# If none of the above are set and sample_rate is not in columns, all weights = 1. +# --------------------------------------------------------------------------- +runs: + + + - dir: U:/projects/clients/Oregon_HTS/hts_pyprocess/template/oregon/or_statewide_out + label: Raw Statewide Survey + hh_weight_col: hh_weight + person_weight_col: person_weight + file_map: + households: toc_household + persons: toc_person + day: toc_day + tours: toc_tour + trips: toc_trip_linked + joint_tour_participants: toc_joint_tour_participants + skimjoin: + enabled: false + + - dir: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/survey_data + label: "Regional: Filtered & Attributed" + hh_weight_col: hh_weight + person_weight_col: person_weight + day_weight_col: null + skimjoin: + config_path: skats_skimjoin_config_alternate_id_col.yaml + + - dir: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full + label: Estimation Mode Inputs + hh_weight_col: hh_weight + person_weight_col: person_weight + file_map: + households: override_households + persons: override_persons + tours: override_tours + trips: override_trips + joint_tour_participants: override_joint_tour_participants + + - dir: U:/projects/clients/Oregon_HTS/SimOR/resident/outputs/estimation_skats/ + label: Estimation Mode Outputs + hh_weight_col: hh_weight + person_weight_col: person_weight + file_map: + households: final_households + persons: final_persons + tours: final_tours + trips: final_trips + vehicles: final_vehicles + joint_tour_participants: final_joint_tour_participants + land_use: final_land_use + # # skimjoin: + # config_path: will_skimjoin_config.yaml + # skim_files: + # - C:\Users\wesley.darling\project_data\odot_skims\*.omx + # - C:\Users\wesley.darling\project_data\odot_skims\maz_stop_walk.csv + # - C:\Users\wesley.darling\project_data\odot_skims\maz_maz_walk.csv + # network_los_file: C:\Users\wesley.darling\project_data\odot_skims\network_los.yaml + + +# --------------------------------------------------------------------------- +# Zone system +# Set use_maz: false for TAZ-only models (zone IDs in outputs are already TAZs). +# For 2-zone (MAZ+TAZ) models, specify which columns in final_land_use +# hold the MAZ and TAZ IDs. +# --------------------------------------------------------------------------- +zones: + use_maz: true # false = TAZ-only model, no MAZ→TAZ conversion needed + maz_col: [MAZ, zone_id] # column in final_land_use containing MAZ IDs + taz_col: TAZ # column in final_land_use containing TAZ IDs + + +# --------------------------------------------------------------------------- +# Column names in the ActivitySim output files +# Change only if your outputs use non-standard column names. +# --------------------------------------------------------------------------- +columns: + household_id: [household_id, hh_id] + trip_id: [trip_id, linked_trip_id] + trip_mode: [trip_mode, linked_trip_mode] + trip_num: [trip_num, linked_trip_num] + ptype: ptype # person type column in persons file + hhsize: hhsize # HH size column in households file + auto_ownership: auto_ownership # vehicle count column in households file + num_workers: num_workers # workers per HH in households file + num_adults: num_adults # adults per HH in households file + # income_segment: [income_segment, income_broad] # OPTIONAL household income segment/level column for trip enrichment + # Prefer a readable purpose column per run. Some survey-style outputs store + # numeric codes in primary_purpose/tour_purpose and labels in tour_type. + tour_purpose: [primary_purpose, tour_type, purpose] + # sample_rate: sample_rate # OPTIONAL override; if omitted, code auto-detects + # a households column literally named 'sample_rate'. + # finalweight = 1/sample_rate (unless explicit run weight columns are provided) + # school_esc_outbound: [school_esc_outbound, out_escort_type] + # school_esc_inbound: [school_esc_inbound, inb_escort_type] + +prepare: + category_mappings: + raw_statewide_survey: + persons: + has_license: + source: can_drive + output_type: boolean + preserve_unmapped: false + mapping: + 1: true + 3: false + 995: false + transit_pass_ownership: + source: transit_pass + output_type: boolean + preserve_unmapped: false + mapping: + 1: false + 3: false + 4: false + 7: true + 8: true + 9: true + 995: null + telecommute_frequency: + source: telework_freq + preserve_unmapped: false + mapping: + 1: 4_days_week + 2: 4_days_week + 3: 4_days_week + 4: 3_days_week + 5: 2_days_week + 6: 1_day_week + 7: No_Telecommute + 8: No_Telecommute + 995: null + 996: null + work_from_home: + source: job_type + output_type: boolean + preserve_unmapped: false + mapping: + 1: false + 2: false + 3: true + 4: false + 5: false + 995: null + day: + cdap_activity: + source: daily_activity_pattern + preserve_unmapped: false + mapping: + 1: M + 2: N + 3: H + tours: + tour_mode: + mapping: &raw_statewide_tour_modes + 1: SCHOOLBUS + 2: WALK + 3: WALK_TRANSIT + 4: PNR_TRANSIT + 5: KNR_TRANSIT + 6: BIKE_TRANSIT + 7: BIKE + 8: EBIKE + 9: ESCOOTER + 10: TAXI + 11: TNC_SINGLE + 12: TNC_SHARED + 13: SOV + 14: HOV2 + 15: HOV3 + 16: OTHER + 995: MISSING + tour_purpose: + mapping: + -1: othdiscr + 1: loop + 2: work + 3: othmaint + 4: school + 5: othmaint + 6: escort + 7: shopping + 8: eatout + 9: social + 10: othmaint + 12: othdiscr + 13: othdiscr + 14: loop + 995: null + tour_category: + mapping: + 1: mandatory + 2: non_mandatory + 3: atwork + 4: joint + trips: + trip_mode: + source: linked_trip_mode + mapping: *raw_statewide_tour_modes + trip_purpose: + source: d_purpose_category + mapping: + -1: othdiscr + 1: home + 2: work + 3: othmaint + 4: school + 5: othmaint + 6: escort + 7: shopping + 8: eatout + 9: social + 10: othmaint + 12: othdiscr + 13: othdiscr + 14: home + 995: null + regional_filtered_attributed: + day: + cdap_activity: + source: daily_activity_pattern + preserve_unmapped: false + mapping: + 1: M + 2: N + 3: H + output: + file_format: parquet + validation: + relationship_checks: warn + distance_skim: + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/autoSkims__MD.omx # path relative to run directory, or absolute + matrix: SOV_H_DIST__MD + non_motorized_distance_skim: + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/maz_maz_walk.csv + matrix: null + time_periods: + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + trip_period_number_column: depart + tour_start_period_number_column: start + tour_end_period_number_column: end + auto_sufficiency_basis: licensed_drivers # licensed_drivers | workers | adults + vot_bins: + source_column: income_segment + output_column: vot_bin + fallback_value: M + mappings: + raw_statewide_survey: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 995: M + 999: M + survey: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 999: M + override: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 999: M + estimation: + 1: L + 2: L + 3: M + 4: M + 5: H + 6: H + 999: M + +skimjoin: + # Create optional long-form trip/tour tables with skim values for alternate modes. + create_hypothetical_skim_tables: true + defaults: + config_path: skats_skimjoin_config.yaml + skim_files: + - 'U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/*.omx' + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/maz_stop_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/maz_maz_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/bike_maz_logsums_commute.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/bike_maz_logsums_noncommute.csv + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + +segment: + dashboard: + segmentation_type: person_sex + visibility: segments_only # full_only | segments_only | full_and_segments + definitions: + signup_platform: + include_full: true + persist_segmented_prepared_tables: false + allow_overlapping: false + on_empty_segment: warn + source: + type: prepared_column + source_table: hh + column: signup_platform + segments: + - id: rmove + label: RMove + values: ["rmove"] + - id: browser + label: Browser + values: ["browser"] + - id: call + label: Call + values: ["call"] + + person_sex: + include_full: false + persist_segmented_prepared_tables: false + allow_overlapping: false + on_empty_segment: warn + source: + type: prepared_column + source_table: per + column: SEX + segments: + - id: male + label: Male + values: [1] + - id: female + label: Female + values: [2] + +summarize: + weighting_modes: [unweighted, weighted] + pnr_tour_modes: + - PNR_TRANSIT + # OPTIONAL summary-time grouping for outputs with a tour_purpose dimension + # group_joint_tour_purposes: true + # group_atwork_tour_purposes: true + # group_school_tour_purposes: true + # Controls additional mapped geography aggregations only. Summaries may still + # emit all_geographies totals, and native prepared home geographies such as + # home_taz, home_county, and home_mpo can appear when those columns exist. + geography: + enabled: true + # Configured mappings create columns such as home_geo__school_district, + # work_geo__county, or land_use_geo__district. + aggregations: + county: + source_zone_system: maz + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/land_use.csv + zone_id_col: MAZ + geography_col: COUNTY + city_code: + source_zone_system: maz + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/land_use.csv + zone_id_col: MAZ + geography_col: CITY + urban_growth_boundary: + source_zone_system: maz + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/land_use.csv + zone_id_col: MAZ + geography_col: UGB + regional_district: + source_zone_system: maz + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/land_use.csv + zone_id_col: MAZ + geography_col: DISTRICT06 + school_district_k8: + source_zone_system: maz + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/land_use.csv + zone_id_col: MAZ + geography_col: DIST_KTO8 + school_district_9_12: + source_zone_system: maz + file: U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/land_use.csv + zone_id_col: MAZ + geography_col: DIST_9TO12 + +dashboard: + title: "SKATS Estimation Visualizer" + logo: ../assets/skats_logo_white_bg.png + enable_maz_geographies: false + live: + pages: + - overview + - long_term_choices + - daily_travel + - joint_travel + - tour_summaries + - trip_summaries + - skim_summaries + # - validation: + # - traffic + # - transit + # - vmt + # - regional_validation + export: + output_path: ../../simor_project_outputs/exports/skats_dashboard.html + # summary series are baked into one exported HTML file. + # `dashboard.export.pages` modifies matching live pages; it is not an + # inclusion list. Omit selector overrides to export all widget states. + # Export segmentation defaults to segmentation.dashboard when omitted. + # Use full_only, segments_only, or full_and_segments to choose which + dashboard: + weighting: [unweighted] + # segmentation_type: signup_platform + # segmentation_visibility: segments_only + pages: + long_term_choices: + mandatory_location_choice: + geography_level: + - All Geography Types + - County + - MPO + # omit TAZ + # parts: + # commuting_flows: + # enabled: false + + # shadow_pricing: + # enabled: true + # geography_level: [all] + # student_type: [all] + # parts: + # workplace_table: + # enabled: false + # school_table: + # enabled: false + # tour_summaries: + # internal_external_tours: + # enabled: false + validation: + vmt: + personal_auto_vmt_breakdown: all + personal_auto_vmt_geography_type: all + personal_auto_vmt_geography: default + personal_auto_vmt_time_period: default + personal_auto_vmt_mode: default + personal_auto_vmt_income_segment: default + personal_auto_vmt_household_size: default + non_motorized_vmt_breakdown: all + non_motorized_vmt_geography_type: all + non_motorized_vmt_geography: default + non_motorized_vmt_time_period: default + non_motorized_vmt_mode: default + non_motorized_vmt_income_segment: default + non_motorized_vmt_household_size: default + + external_travel_metric: all + external_travel_breakdown: all + external_travel_trip_purpose: default + external_travel_time_period: default + + demo_commercial_metric: all + demo_commercial_breakdown: all + demo_commercial_vehicle_type: default + demo_commercial_time_period: default + host: + account: my-connect-cloud-account + # app_id: 12345 + # title: Estimation Mode Comparison Visualizer + # First hosted publish may open a browser for Posit Connect Cloud login. + # Later runs usually reuse saved local rsconnect metadata. + verify: true + + +# Summary-affecting regrouping/normalization belongs under summarize.category_normalization. +# Cosmetic labels and ordering belong under display.labels. +display: + bar_hover_mode: all # closest | all + density_hover_mode: all # closest | all + labels: + person_type: + mapping: + all_person_types: All Person Types + 1: Full-time worker + 2: Part-time worker + 3: University student + 4: Non-worker adult + 5: Retired + 6: Student + 7: Preschool + + transit_subsidy: + mapping: + 0: No subsidy + 1: Discounted + 2: Free + + license_holding_status: + mapping: + has_license: Has License + no_license: No License + + transit_pass_ownership_status: + mapping: + has_transit_pass: Has Transit Pass + no_transit_pass: No Transit Pass + + telecommute_frequency: + mapping: + 1_day_week: 1 Day per Week + 2_days_week: 2 Days per Week + 3_days_week: 3 Days per Week + 2_3_days_week: 2-3 Days per Week + 4_days_week: 4+ Days per Week + No_Telecommute: No Telecommute + + mode: + mapping: + SOV: Drive Alone + HOV2: Shared Ride 2 + HOV3: Shared Ride 3+ + WALK: Walk + BIKE: Bike + EBIKE: E-Bike + ESCOOTER: E-Scooter + WALK_TRANSIT: Walk-Transit + BIKE_TRANSIT: Bike-Transit + PNR_TRANSIT: PNR-Transit + KNR_TRANSIT: KNR-Transit + TAXI: Taxi + TNC_SINGLE: TNC-Single + TNC_SHARED: TNC-Pool + SCHOOLBUS: School Bus + + + # escort: + # mapping: + # not_escorted: No Escort + # pure_escort: Pure Escort + # ride_share: Ride Share + # 0: No Escort + # 1: Pure Escort + # 2: Ride Share + # "": No Escort + + tour_purpose: + mapping: + all_tour_purposes: All Tour Purposes + work: Work + school: School + joint: Joint + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + atwork: At-Work + loop: Loop + + trip_purpose: + mapping: + home: Home + work: Work + school: School + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + loop: Loop + + stop_purpose: + mapping: + work: Work + school: School + escort: Escort + shopping: Shopping + othmaint: Other Maintenance + eatout: Eat Out + social: Social + othdiscr: Other Discretionary + + tour_composition: + mapping: + adults: Adults + mixed: Mixed + children: Children + + tour_category: + mapping: + mandatory: Mandatory + joint: Joint + non_mandatory: Non-Mandatory + atwork: At-Work + + atwork_subtour_frequency_category: + mapping: + no_subtours: None + eat: 1 Eating Out + business1: 1 Business + maint: 1 Maintenance + business2: 2 Business + eat_business: 1 Eating Out + 1 Business + + geography: + mapping: + all_geographies: All Geographies + county: County + home_county: County + home_mpo: MPO + home_taz: TAZ + district: District + taz: TAZ + maz: MAZ + school_district: School District + + mandatory_tour_frequency: + # mapping: + # work1: Work + # school1: School + # work2: 2 Work + # school2: 2 School + # work_and_school: 1 Work + 1 School + mapping: + 1: Work + 3: School + 2: 2 Work + 4: 2 School + 5: 1 Work + 1 School + + daily_activity_pattern: + mapping: + M : Mandatory + N : Non-Mandatory + H : At-Home + + facility_type: + mapping: + 1: "Interstate" + 3: "Principal Arterial" + 4: "Minor Arterial" + 5: "Major Collector" + 6: "Minor Collector" + 7: "Local Road" + 30: "Ramp" + 998: "Bike/Walk" + + commercial_vehicle_type: + mapping: + car: Car + su: Single-Unit Truck + mu: Multi-Unit Truck + + run_colors: + - "#264653" + - "#2A9D8F" + - "#457B9D" + - "#6A4C93" + - "#3A86FF" # Blue + - "#06D6A0" # Mint + - "#FF006E" # Pink + - "#FFBE0B" # Yellow + - "#8338EC" # Purple + - "#FB5607" # Orange + # - "#ff7f0e" + # - "#2ca02c" + # - "#d62728" + # - "#9467bd" + # - "#8c564b" + # - "#e377c2" + # - "#7f7f7f" diff --git a/simor_configs/skats_configs/skats_skimjoin_config.yaml b/simor_configs/skats_configs/skats_skimjoin_config.yaml new file mode 100644 index 0000000..1edb68b --- /dev/null +++ b/simor_configs/skats_configs/skats_skimjoin_config.yaml @@ -0,0 +1,267 @@ +project: + skim_files: + - 'U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/*.omx' + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/maz_stop_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/maz_maz_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/bike_maz_logsums_commute.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/bike_maz_logsums_noncommute.csv + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + # project.trips_table / project.tours_table / project.output_dir are used by the + # standalone skimjoin CLI only. Integrated runtime skimjoin reads prepared + # in-memory trips/tours from the processor pipeline instead. + # trips_table: C:\Users\wesley.darling\Downloads\viz\viz\output\trip_linked.csv + # trips_table: C:\Users\wesley.darling\projects\activitysim_visualizer\artifacts\prepared_cache\override\trips.parquet + # tours_table: C:\Users\wesley.darling\Downloads\viz\viz\output\tour.csv + # tours_table: C:\Users\wesley.darling\projects\activitysim_visualizer\artifacts\prepared_cache\override\tours.parquet + # output_dir: skats_output + + +activitysim: + # activitysim now only defines structural trip/tour fields used by skimjoin. + trip_mode_column: trip_mode + trip_id_column: trip_id + tour_mode_column: tour_mode + tour_id_column: tour_id + outbound_column: outbound + +defaults: + origin: OTAZ + destination: DTAZ + output_prefix: skim_ + sentinel_values: + - 9999 + - 999999 + +zone_mapping: + lookup_name: taz + file_lookup_names: + fares.omx: taz + missing_zone_policy: error + +dimensions: + PERIOD: + source_columns: + trip_source_column: trip_period + outbound_tour_source_column: start_period + inbound_tour_source_column: first_inbound_trip_period + VOT: + source_columns: + trip_source_column: vot_bin + outbound_tour_source_column: vot_bin + inbound_tour_source_column: vot_bin + values: + L: L + M: M + H: H + BIKE_PURPOSE: + source_columns: + trip_source_column: tour_purpose + outbound_tour_source_column: tour_purpose + inbound_tour_source_column: tour_purpose + values: + work: commute + school: commute + univ: commute + atwork: noncommute + business: noncommute + eat: noncommute + eatout: noncommute + escort: noncommute + loop: noncommute + maint: noncommute + othdiscr: noncommute + othmaint: noncommute + shopping: noncommute + social: noncommute + +ignore_modes: + - ESCOOTER + # SKATS has WTW skims only and no prepared drive-transit parking-zone field. + - KNR_TRANSIT + - MISSING + - OTHER + - PNR_TRANSIT + +modes: + SOV: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + HOV2: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + HOV3: + output_prefix: skim_auto_ + time: "SR3_{VOT}_TIME__{PERIOD}" + cost: "SR3_{VOT}_COST__{PERIOD}" + distance: "SR3_{VOT}_DIST__{PERIOD}" + SCHOOLBUS: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_ACC__{PERIOD}" + origin: OTAZ + destination: DTAZ + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_EGR__{PERIOD}" + origin: OTAZ + destination: DTAZ + TAXI: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + TNC_SHARED: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + TNC_SINGLE: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + WALK: + output_prefix: skim_walk_ + distance: + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__DISTWALK + fallbacks: + - matrix: WLK_DIST + origin: OTAZ + destination: DTAZ + maz_walk_actual: + output: skim_walk_maz_actual + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__actual + WALK_TRANSIT: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + BIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + logsum: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__logsum_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::logsum_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + EBIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + BIKE_TRANSIT: + output_prefix: skim_ + bike_o_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_d_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_o_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + bike_d_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + + diff --git a/simor_configs/skats_configs/skats_skimjoin_config_alternate_id_col.yaml b/simor_configs/skats_configs/skats_skimjoin_config_alternate_id_col.yaml new file mode 100644 index 0000000..37e97e5 --- /dev/null +++ b/simor_configs/skats_configs/skats_skimjoin_config_alternate_id_col.yaml @@ -0,0 +1,267 @@ +project: + skim_files: + - 'U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/*.omx' + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/maz_stop_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/maz_maz_walk.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/bike_maz_logsums_commute.csv + - U:/projects/clients/Oregon_HTS/SimOR/resident/model_data/skats/data_full/bike_maz_logsums_noncommute.csv + network_los_file: U:/projects/clients/Oregon_HTS/SimOR/resident/configs/network_los.yaml + # project.trips_table / project.tours_table / project.output_dir are used by the + # standalone skimjoin CLI only. Integrated runtime skimjoin reads prepared + # in-memory trips/tours from the processor pipeline instead. + # trips_table: C:\Users\wesley.darling\Downloads\viz\viz\output\trip_linked.csv + # trips_table: C:\Users\wesley.darling\projects\activitysim_visualizer\artifacts\prepared_cache\override\trips.parquet + # tours_table: C:\Users\wesley.darling\Downloads\viz\viz\output\tour.csv + # tours_table: C:\Users\wesley.darling\projects\activitysim_visualizer\artifacts\prepared_cache\override\tours.parquet + # output_dir: skats_output + + +activitysim: + # activitysim now only defines structural trip/tour fields used by skimjoin. + trip_mode_column: trip_mode + trip_id_column: linked_trip_id + tour_mode_column: tour_mode + tour_id_column: tour_id + outbound_column: outbound + +defaults: + origin: OTAZ + destination: DTAZ + output_prefix: skim_ + sentinel_values: + - 9999 + - 999999 + +zone_mapping: + lookup_name: taz + file_lookup_names: + fares.omx: taz + missing_zone_policy: error + +dimensions: + PERIOD: + source_columns: + trip_source_column: trip_period + outbound_tour_source_column: start_period + inbound_tour_source_column: first_inbound_trip_period + VOT: + source_columns: + trip_source_column: vot_bin + outbound_tour_source_column: vot_bin + inbound_tour_source_column: vot_bin + values: + L: L + M: M + H: H + BIKE_PURPOSE: + source_columns: + trip_source_column: tour_purpose + outbound_tour_source_column: tour_purpose + inbound_tour_source_column: tour_purpose + values: + work: commute + school: commute + univ: commute + atwork: noncommute + business: noncommute + eat: noncommute + eatout: noncommute + escort: noncommute + loop: noncommute + maint: noncommute + othdiscr: noncommute + othmaint: noncommute + shopping: noncommute + social: noncommute + +ignore_modes: + - ESCOOTER + # SKATS has WTW skims only and no prepared drive-transit parking-zone field. + - KNR_TRANSIT + - MISSING + - OTHER + - PNR_TRANSIT + +modes: + SOV: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + HOV2: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + HOV3: + output_prefix: skim_auto_ + time: "SR3_{VOT}_TIME__{PERIOD}" + cost: "SR3_{VOT}_COST__{PERIOD}" + distance: "SR3_{VOT}_DIST__{PERIOD}" + SCHOOLBUS: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_ACC__{PERIOD}" + origin: OTAZ + destination: DTAZ + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + fallbacks: + - matrix: "WTW_EGR__{PERIOD}" + origin: OTAZ + destination: DTAZ + TAXI: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + TNC_SHARED: + output_prefix: skim_auto_ + time: "SR2_{VOT}_TIME__{PERIOD}" + cost: "SR2_{VOT}_COST__{PERIOD}" + distance: "SR2_{VOT}_DIST__{PERIOD}" + TNC_SINGLE: + output_prefix: skim_auto_ + time: "SOV_{VOT}_TIME__{PERIOD}" + cost: "SOV_{VOT}_COST__{PERIOD}" + distance: "SOV_{VOT}_DIST__{PERIOD}" + WALK: + output_prefix: skim_walk_ + distance: + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__DISTWALK + fallbacks: + - matrix: WLK_DIST + origin: OTAZ + destination: DTAZ + maz_walk_actual: + output: skim_walk_maz_actual + origin: o_maz + destination: d_maz + matrix: maz_maz_walk__actual + WALK_TRANSIT: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_ACC__{PERIOD}" + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: "WTW_EGR__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + transit_o_maz_stop_walk_bus: + output: skim_transit_o_maz_stop_walk_bus + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_d_maz_stop_walk_bus: + output: skim_transit_d_maz_stop_walk_bus + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + transit_o_maz_stop_walk_premium: + output: skim_transit_o_maz_stop_walk_premium + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_d_maz_stop_walk_premium: + output: skim_transit_d_maz_stop_walk_premium + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + BIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + logsum: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__logsum_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::logsum_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + EBIKE: + output_prefix: skim_bike_ + distance: + origin: o_maz + destination: d_maz + matrix: "bike_maz_logsums_{BIKE_PURPOSE}__distance_{BIKE_PURPOSE}" + fallbacks: + - matrix: "bike_taz_logsums_{BIKE_PURPOSE}.omx::distance_{BIKE_PURPOSE}" + origin: OTAZ + destination: DTAZ + BIKE_TRANSIT: + output_prefix: skim_ + bike_o_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_d_maz_stop_distance_bus: + output: skim_bike_transit_distance_bus + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_local_bus + bike_o_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: o_maz + matrix: maz_stop_walk__walk_dist_premium_transit + bike_d_maz_stop_distance_premium: + output: skim_bike_transit_distance_premium + combine: sum + lookup: key + key_column: d_maz + matrix: maz_stop_walk__walk_dist_premium_transit + transit_auxiliary_walk_time: "WTW_AUX__{PERIOD}" + transit_brt_ivtt: "WTW_BRT__{PERIOD}" + transit_bus_ivtt: "WTW_BUS__{PERIOD}" + transit_commuter_rail_ivtt: "WTW_CRT__{PERIOD}" + transit_first_wait_time: "WTW_FWT__{PERIOD}" + transit_light_rail_ivtt: "WTW_LRT__{PERIOD}" + transit_tiv: "WTW_TIV__{PERIOD}" + transit_num_transfers: "WTW_XFR__{PERIOD}" + transit_transfer_wait_time: "WTW_XWT__{PERIOD}" + transit_fare: "fare__{PERIOD}" + + diff --git a/tests/test_calculation_notes.py b/tests/test_calculation_notes.py index 7e3c06c..224334a 100644 --- a/tests/test_calculation_notes.py +++ b/tests/test_calculation_notes.py @@ -74,6 +74,7 @@ def test_calculation_note_renderer_escapes_configured_text() -> None: method_text="Join by their shared key.", sources=("source_one",), source_filters=("Only A & B.",), + column_definitions=("Column : A & B.",), ) rendered = render_calculation_note_html(note) @@ -87,6 +88,8 @@ def test_calculation_note_renderer_escapes_configured_text() -> None: assert "Join <records> by their shared key." in rendered assert "Generic grouping text" not in rendered assert "Only A & B." in rendered + assert "Table columns:" in rendered + assert "Column <A>: A & B." in rendered assert "source_one" in rendered assert "Summary Tables Used:" in rendered assert "Prepared summaries used:" not in rendered @@ -94,6 +97,36 @@ def test_calculation_note_renderer_escapes_configured_text() -> None: assert rendered.endswith("") +def test_skim_summary_notes_render_table_column_definitions() -> None: + trip_note = render_calculation_note_html( + get_calculation_note("trip_skims.summary_table") + ) + tour_note = render_calculation_note_html( + get_calculation_note("tour_skims.summary_table") + ) + + for rendered in (trip_note, tour_note): + assert "Table columns:" in rendered + assert "Zero is a valid value." in rendered + assert "Mean Non-Zero:" in rendered + assert "including zeros" in rendered + assert "Zero Share:" in rendered + assert "nonmissing numeric skim value" in rendered + assert "summed" in rendered + assert "Missing Share:" in rendered + + +def test_validation_notes_expose_comparison_and_error_formulas() -> None: + regional = get_calculation_note("regional_validation.flows") + facility = get_calculation_note("traffic.facility_summary") + + assert "difference = modeled - observed" in regional.formula + assert ( + "RMSPE = sqrt(mean(((observed_i - modeled_i) / observed_i)²)) * 100" + in facility.formula + ) + + def test_calculation_note_is_collapsed_html_pane_exported_without_conversion() -> None: pane = calculation_note("traffic.link_volume") diff --git a/tests/test_config_refactor_phase1.py b/tests/test_config_refactor_phase1.py index 79778c8..38827b6 100644 --- a/tests/test_config_refactor_phase1.py +++ b/tests/test_config_refactor_phase1.py @@ -128,7 +128,7 @@ def test_new_config_layout_normalizes_to_existing_runtime_fields(tmp_path: Path) " - summarize", " - dashboard", " dashboard_mode: export", - " overwrite: true", + " refresh: [summarize]", "dashboard:", ' title: "Refactor Dashboard"', " live:", @@ -192,7 +192,7 @@ def test_new_config_layout_normalizes_to_existing_runtime_fields(tmp_path: Path) "dashboard", ) assert config.pipeline.dashboard_mode == "export" - assert config.pipeline.overwrite is True + assert config.pipeline.refresh == ("summarize",) assert config.dashboard_title == "Refactor Dashboard" assert [entry.page_id for entry in config.dashboard_pages or []] == [ "overview", @@ -284,6 +284,35 @@ def test_dashboard_host_placeholder_rejects_unknown_fields(tmp_path: Path) -> No ) +def test_dashboard_logo_resolves_relative_to_config_file(tmp_path: Path) -> None: + logo_path = tmp_path / "assets" / "logo.png" + logo_path.parent.mkdir() + logo_path.write_bytes(b"logo") + + config = _write_config( + tmp_path, + [ + "dashboard:", + " logo: assets/logo.png", + "runs: []", + ], + ) + + assert config.dashboard_logo == str(logo_path.resolve()) + + +def test_dashboard_logo_rejects_missing_file(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="dashboard.logo file does not exist"): + _write_config( + tmp_path, + [ + "dashboard:", + " logo: missing.png", + "runs: []", + ], + ) + + @pytest.mark.parametrize( ("lines", "message"), [ @@ -293,7 +322,9 @@ def test_dashboard_host_placeholder_rejects_unknown_fields(tmp_path: Path) -> No (["pipeline:", " steps: [segment, dashboard]"], "without 'summarize'"), (["pipeline:", " steps: [dashboard, summarize]"], "place 'dashboard' last"), (["pipeline:", " dashboard_mode: deploy"], "dashboard_mode"), - (["pipeline:", " overwrite: maybe"], "pipeline.overwrite"), + (["pipeline:", " refresh: maybe"], "pipeline.refresh"), + (["pipeline:", " refresh: [dashboard]"], "pipeline.refresh"), + (["pipeline:", " overwrite: false"], "pipeline.overwrite"), ], ) def test_pipeline_validation_rejects_invalid_configurations( @@ -303,3 +334,21 @@ def test_pipeline_validation_rejects_invalid_configurations( ) -> None: with pytest.raises(ValueError, match=message): _write_config(tmp_path, [*lines, "runs: []"]) + + +def test_pipeline_refresh_all_expands_only_enabled_materialized_steps( + tmp_path: Path, +) -> None: + config = _write_config( + tmp_path, + [ + "pipeline:", + " steps: [prepare, skimjoin, summarize, dashboard]", + " refresh: all", + "skimjoin:", + " defaults:", + "runs: []", + ], + ) + + assert config.pipeline.refresh == ("prepare", "skimjoin", "summarize") diff --git a/tests/test_dashboard_helpers_phase1.py b/tests/test_dashboard_helpers_phase1.py index fa9da15..0d84955 100644 --- a/tests/test_dashboard_helpers_phase1.py +++ b/tests/test_dashboard_helpers_phase1.py @@ -16,6 +16,7 @@ ) from dashboard.data_access import RunTables from dashboard.helpers.category_helpers import ( + cap_numeric_category_frame, column_value_intersection, common_column_options, complete_category_counts, @@ -55,12 +56,33 @@ timebin_label, ) from dashboard.page_base import DashboardPage +from dashboard.pages.skim_summaries._shared import component_display_name from dashboard.pages.trip_summaries.parking_location import parking_scatter_data from dashboard.state import DashboardState from processor.models import RunData from test_export_html import _full_summary_run, _write_config +@pytest.mark.parametrize( + ("component", "expected"), + [ + ( + "skim_bike_transit_distance_bus", + "Total Bike Distance - Local Bus (mi) (Estimated from Walk Skims)", + ), + ( + "skim_bike_transit_distance_premium", + "Total Bike Distance - Premium Transit (mi) (Estimated from Walk Skims)", + ), + ], +) +def test_component_display_name_labels_estimated_bike_transit_distances( + component: str, + expected: str, +) -> None: + assert component_display_name(component) == expected + + def test_run_table_view_filters_transforms_and_joins_by_run_label() -> None: counts = RunTables.from_runs( [ @@ -174,7 +196,7 @@ def test_category_helpers_support_intersection_normalization_and_numeric_sort( value_cols=("count",), ) sorted_bins = ( - pl.DataFrame({"bin": ["40+", "2", "10"]}) + pl.DataFrame({"bin": ["40+", "2", ">0-<1", "10"]}) .with_columns(numeric_like_sort_expr("bin").alias("_sort")) .sort("_sort") .drop("_sort")["bin"] @@ -186,7 +208,25 @@ def test_category_helpers_support_intersection_normalization_and_numeric_sort( assert mapping["10"] == "10" assert normalized[0][1]["category"].to_list() == ["Unspecified", "Worker"] assert completed[0][1]["count"].to_list() == [0, 2, 1] - assert sorted_bins == ["2", "10", "40+"] + assert sorted_bins == [">0-<1", "2", "10", "40+"] + + capped_distance = cap_numeric_category_frame( + pl.DataFrame( + { + "bin": ["0", ">0-<1", "39", "40", "51+"], + "count": [1, 2, 3, 4, 5], + } + ), + category="bin", + cap_value=40, + value_cols=("count",), + ) + assert capped_distance.to_dicts() == [ + {"bin": "0", "count": 1}, + {"bin": ">0-<1", "count": 2}, + {"bin": "39", "count": 3}, + {"bin": "40+", "count": 9}, + ] def test_geography_helpers_normalize_and_build_options(tmp_path: Path) -> None: @@ -474,6 +514,22 @@ def test_data_table_drops_index_columns_and_hides_pandas_index() -> None: assert tabulator.titles == {"metric": "Metric", "value": "Value"} +def test_data_table_shortens_long_run_tabs_and_column_titles() -> None: + run_label = "Regional Transportation Scenario Baseline 2050 North" + column = "regional_transportation_scenario_comparison_measure" + table = data_table([(run_label, pl.DataFrame({column: [1.0]}))]) + + assert table._names == ["Regional Transportation…North"] + assert table._run_label_full_titles == (run_label,) + content = table.objects[0] + assert run_label in content.objects[0].object + tabulator = content.objects[1] + assert len(tabulator.titles[column]) <= 30 + assert tabulator.header_tooltips == { + column: "Regional Transportation Scenario Comparison Measure" + } + + def test_column_titles_for_display_humanizes_machine_column_names() -> None: titles = column_titles( [ diff --git a/tests/test_dashboard_live.py b/tests/test_dashboard_live.py index 583d098..9f12642 100644 --- a/tests/test_dashboard_live.py +++ b/tests/test_dashboard_live.py @@ -469,7 +469,8 @@ def test_external_traffic_helpers_filter_period_and_facility_type( "Total Modeled Count": 440.0, "% Difference": "10.00%", "RMSE": 22.360679774997898, - "R^2": 0.875, + "RMSPE": "10.00%", + "R²": 0.875, } ] @@ -490,7 +491,7 @@ def test_demo_count_fit_line_helper_builds_plot_data() -> None: "observed_min": [10.0, 20.0], "observed_max": [30.0, 40.0], "equation_label": ["y = 2.00x + 5.00", "y = 3.00x + 7.00"], - "r_squared_label": ["R^2 = 1.00", "R^2 = 0.90"], + "r_squared_label": ["R² = 1.00", "R² = 0.90"], } ), ) @@ -499,11 +500,15 @@ def test_demo_count_fit_line_helper_builds_plot_data() -> None: facility_type="4", ) - assert fit_lines[0][1].select("observed_volume", "modeled_volume").to_dicts() == [ - {"observed_volume": 20.0, "modeled_volume": 67.0}, - {"observed_volume": 40.0, "modeled_volume": 127.0}, - ] - assert "y = 3.00x + 7.00" in fit_lines[0][1]["annotation"][0] + fit_frame = fit_lines[0][1] + assert fit_frame.height == 101 + assert fit_frame.select("observed_volume", "modeled_volume").row( + 0, named=True + ) == {"observed_volume": 20.0, "modeled_volume": 67.0} + assert fit_frame.select("observed_volume", "modeled_volume").row( + -1, named=True + ) == {"observed_volume": 40.0, "modeled_volume": 127.0} + assert "y = 3.00x + 7.00" in fit_frame["annotation"][0] def test_external_vmt_helper_reshapes_wide_tod_table() -> None: @@ -2008,11 +2013,11 @@ def test_regional_validation_page_compares_county_flows_to_commuting_flows( run_key="base", summaries_by_mode={ "weighted": { - "county_flows_joja_validation_summary": observed, + "county_commuting_flows_validation_summary": observed, "commuting_flows": modeled, }, "unweighted": { - "county_flows_joja_validation_summary": observed, + "county_commuting_flows_validation_summary": observed, "commuting_flows": modeled, }, }, @@ -2027,18 +2032,18 @@ def test_regional_validation_page_compares_county_flows_to_commuting_flows( assert list(page.flow_matrix_sel.options) == ["County flows"] assert list(page.comparison_metric_sel.options) == [ + "Modeled", "Observed", "Difference", - "Percent Difference", - "Absolute Percent Difference", - "Modeled", + "% Difference", + "Absolute % Difference", ] chart = page.render_flow_section() tabs = chart.objects[0] plot = tabs.objects[0][0] - assert plot.object.layout.title.text == "Observed County flows" - assert plot.object.data[0].z == ([10.0, 5.0], [3.0, 20.0]) + assert plot.object.layout.title.text == "Modeled County flows" + assert plot.object.data[0].z == ([12.0, 4.0], [3.0, 18.0]) page.comparison_metric_sel.value = "Difference" chart = page.render_flow_section() @@ -2154,7 +2159,10 @@ def test_data_requirements_for_pages_tracks_optional_summary_dependencies() -> N assert "commercial_vmt_totals" not in requirements.required_summary_ids assert "commercial_vmt_totals" not in requirements.optional_summary_ids assert "auto_vmt_validation_summary" not in requirements.optional_summary_ids - assert "county_flows_validation_summary" in requirements.optional_summary_ids + assert ( + "district_commuting_flows_validation_summary" + in requirements.optional_summary_ids + ) assert "commuting_flows" in requirements.optional_summary_ids assert "auto_vmt_validation_summary" not in requirements.summary_ids_for_pruning @@ -2181,9 +2189,12 @@ def test_resolve_page_definitions_rejects_duplicate_configured_page_ids( def test_build_dashboard_uses_expected_default_page_order(tmp_path: Path) -> None: - config = _write_config(tmp_path) + logo_path = tmp_path / "logo.png" + logo_path.write_bytes(b"logo") + config = _write_config(tmp_path, dashboard_logo=logo_path.name) template = build_dashboard([], config, summary_runs=[_full_summary_run()]) + assert template.logo == config.dashboard_logo assert [ page.name for page in template._dashboard_pages ] == EXPECTED_DEFAULT_PAGE_TITLES diff --git a/tests/test_export_html.py b/tests/test_export_html.py index 6dee1b6..511c4e9 100644 --- a/tests/test_export_html.py +++ b/tests/test_export_html.py @@ -28,6 +28,7 @@ def _write_config( tmp_path: Path, *, dashboard_pages: list[object] | None | object = ..., + dashboard_logo: str | None = None, weighting_modes: list[str] | None = None, modes_lines: list[str] | None = None, geography_lines: list[str] | None = None, @@ -54,6 +55,8 @@ def _write_config( ' title: "Test Dashboard"', ] ) + if dashboard_logo is not None: + lines.append(f" logo: {json.dumps(dashboard_logo)}") if dashboard_pages is ...: dashboard_pages = [page_id for page_id, _ in EXPECTED_DEFAULT_PAGES] if dashboard_pages is not None: @@ -566,6 +569,7 @@ def _skim_summary_run(): "trip_mode": ["DRIVE", "WALK", "DRIVE"], "n_valid": [10.0, 8.0, 12.0], "mean": [15.0, 20.0, 7.0], + "mean_nonzero": [15.0, 20.0, 7.0], "std": [1.5, 2.0, 0.7], "min": [10.0, 15.0, 5.0], "max": [20.0, 30.0, 9.0], @@ -581,6 +585,7 @@ def _skim_summary_run(): "tour_mode": ["DRIVE", "WALK", "DRIVE"], "n_valid": [6.0, 5.0, 7.0], "mean": [25.0, 18.0, 11.0], + "mean_nonzero": [25.0, 18.0, 11.0], "std": [2.5, 1.8, 1.1], "min": [21.0, 14.0, 9.0], "max": [30.0, 22.0, 13.0], @@ -930,6 +935,7 @@ def test_config_defaults_when_optional_sections_are_absent( ) assert config.weighting_modes == ["weighted", "unweighted"] assert config.dashboard_title == "ActivitySim Visualizer" + assert config.dashboard_logo is None assert config.dashboard_pages is None assert config.run_colors == [ "#1f77b4", @@ -944,6 +950,35 @@ def test_config_defaults_when_optional_sections_are_absent( assert config.export_html.dashboard.weighting == ["weighted", "unweighted"] assert config.export_html.dashboard.values == ["percent", "count"] assert config.export_html.pages == {} + + +def test_build_export_html_document_embeds_configured_logo(tmp_path: Path) -> None: + logo_path = tmp_path / "assets" / "logo.png" + logo_path.parent.mkdir() + logo_path.write_bytes(b"logo") + config = _write_config( + tmp_path, + dashboard_pages=["overview"], + dashboard_logo="assets/logo.png", + weighting_modes=["weighted"], + export_html_lines=[ + "dashboard:", + " weighting: [weighted]", + " values: [percent]", + ], + ) + + document = build_export_html_document( + [], + config, + summary_runs=[_full_summary_run()], + ) + payload = _extract_payload(document) + + assert payload["logo"] == "data:image/png;base64,bG9nbw==" + assert "assets/logo.png" not in document + + def test_export_html_config_rejects_invalid_or_empty_values(tmp_path: Path) -> None: with pytest.raises( ValueError, match="Unsupported dashboard.export.dashboard.weighting" @@ -1129,6 +1164,7 @@ def test_build_export_html_document_serializes_dashboard_states_and_pages( payload = _extract_payload(html) assert payload["schema_version"] == EXPORT_SCHEMA_VERSION + assert payload["logo"] is None assert payload["runs_loaded"] == [{"label": "Base", "color": "#1f77b4"}] assert payload["chrome"] == { "layout": "left_rail", @@ -1670,7 +1706,7 @@ def test_build_export_html_document_keeps_grouped_tour_mode_chart_when_mode_grou assert any( node.get("kind") == "plotly" and node.get("figure", {}).get("layout", {}).get("title", {}).get("text") - == "Tour Mode - Zero Auto" + == "Tour Mode - Zero Auto Households" for node in variant_nodes ) vehicle_nodes = _walk_nodes(region_nodes["tour_mode_vehicles"]["default_content"]) diff --git a/tests/test_export_html_smoke.py b/tests/test_export_html_smoke.py index 33f7790..c5ef5a6 100644 --- a/tests/test_export_html_smoke.py +++ b/tests/test_export_html_smoke.py @@ -130,6 +130,10 @@ def test_export_runtime_assets_are_loaded_from_source_files() -> None: assert ".export-shell" in css assert ".export-error-panel" in css assert ".export-table-sort" in css + assert ".export-layout.rail-collapsed" in css + assert ".export-layout.rail-collapsed .export-rail" in css + assert ".export-logo" in css + assert '.widget-shell input[type="number"]:disabled' in css assert "function validatePayloadSchema(candidate)" in runtime_js assert "function renderPlot(node, context)" in runtime_js assert "function renderTable(node)" in runtime_js @@ -138,6 +142,7 @@ def test_export_runtime_assets_are_loaded_from_source_files() -> None: assert "function getLeafPageId(currentPayload, currentState)" in runtime_js assert "function createRuntimeContext(config)" in runtime_js assert "function createRuntimeActions(context)" in runtime_js + assert 'className: "export-logo"' in runtime_js assert "Plotly.react" in runtime_js assert "__EXPORT_SCHEMA_VERSION__" not in runtime_js diff --git a/tests/test_export_payload.py b/tests/test_export_payload.py index 6628c65..4c19916 100644 --- a/tests/test_export_payload.py +++ b/tests/test_export_payload.py @@ -82,8 +82,11 @@ def test_vmt_export_content_includes_dropdown_availability_note() -> None: @pytest.mark.full_export def test_build_export_payload_has_stable_top_level_contract() -> None: tmp_path = _workspace_tmp_dir("payload_contract") + logo_path = tmp_path / "logo.png" + logo_path.write_bytes(b"logo") config = _write_config( tmp_path, + dashboard_logo=logo_path.name, export_html_lines=[ "dashboard:", " weighting: all", @@ -96,6 +99,7 @@ def test_build_export_payload_has_stable_top_level_contract() -> None: assert list(payload) == [ "schema_version", "title", + "logo", "runs_loaded", "chrome", "dashboard_controls", @@ -106,6 +110,7 @@ def test_build_export_payload_has_stable_top_level_contract() -> None: "client_runtime", ] assert payload["schema_version"] == EXPORT_SCHEMA_VERSION + assert payload["logo"] == "data:image/png;base64,bG9nbw==" assert payload["client_runtime"] == EXPORT_CLIENT_RUNTIME assert ( payload["page_export_support"]["client_side_runtime"] diff --git a/tests/test_export_runtime_contract.py b/tests/test_export_runtime_contract.py index 4e03d4a..9c89032 100644 --- a/tests/test_export_runtime_contract.py +++ b/tests/test_export_runtime_contract.py @@ -218,6 +218,7 @@ def test_runtime_asset_contains_explicit_context_action_and_region_helpers() -> assert "function createRuntimeContext(config)" in runtime_js assert "function createRuntimeActions(context)" in runtime_js assert "function makeButton(config)" in runtime_js + assert "button.title = String(config.title);" in runtime_js assert "function buildRegionVariantKey(selectorValues)" in runtime_js assert "const PLOT_RESIZE_RETRY_DELAYS_MS = [60, 180, 320];" in runtime_js assert 'displayModeBar: "hover"' in runtime_js @@ -227,6 +228,32 @@ def test_runtime_asset_contains_explicit_context_action_and_region_helpers() -> assert "modeBarButtonsToAdd: [makePlotCsvDownloadButton(figure)]" in runtime_js +def test_runtime_asset_preserves_exported_plot_aspect_ratios() -> None: + runtime_js = load_export_runtime_js() + + assert "plotElement.style.aspectRatio = String(aspectRatio);" in runtime_js + assert "delete layout.height;" in runtime_js + + +def test_runtime_asset_exposes_full_table_and_tab_titles_as_tooltips() -> None: + runtime_js = load_export_runtime_js() + + assert "title: tab.full_title || tab.title" in runtime_js + assert "(node.column_tooltips || {})[column] || column" in runtime_js + + +def test_runtime_asset_contains_collapsible_export_rail() -> None: + runtime_js = load_export_runtime_js() + + assert "railCollapsed: false" in runtime_js + assert 'rail.id = "export-rail"' in runtime_js + assert '"aria-controls": "export-rail"' in runtime_js + assert 'className: "export-layout" + (railCollapsed ? " rail-collapsed" : "")' in runtime_js + assert 'railCollapsed ? "Show sidebar" : "Hide sidebar"' in runtime_js + assert 'layout.classList.toggle("rail-collapsed", context.railCollapsed)' in runtime_js + assert 'context.plotManager.scheduleResize();' in runtime_js + + def test_runtime_asset_contains_plot_csv_export_helpers() -> None: runtime_js = load_export_runtime_js() @@ -239,6 +266,7 @@ def test_runtime_asset_contains_plot_csv_export_helpers() -> None: assert '"y"' in runtime_js assert '"trace_index"' not in runtime_js assert '"customdata"' not in runtime_js + assert "trace.meta.run_name" in runtime_js assert '"-" + valueMode + ".csv"' in runtime_js assert 'return normalized || "plot-data";' in runtime_js diff --git a/tests/test_export_serializer.py b/tests/test_export_serializer.py index 6620064..80fbf85 100644 --- a/tests/test_export_serializer.py +++ b/tests/test_export_serializer.py @@ -68,6 +68,7 @@ def test_serialize_viewable_supports_plotly_and_table_nodes() -> None: assert plot_payload["kind"] == "plotly" assert plot_payload["figure"]["data"][0]["type"] == "bar" + assert "aspect_ratio" not in plot_payload assert table_payload == { "kind": "table", "columns": ["Alpha Value", "beta", "Gamma Value"], @@ -75,6 +76,19 @@ def test_serialize_viewable_supports_plotly_and_table_nodes() -> None: } +def test_serialize_viewable_preserves_numeric_plotly_aspect_ratio() -> None: + pane = pn.pane.Plotly( + go.Figure(data=[go.Scatter(x=[1], y=[1])], layout={"height": 400}), + sizing_mode="scale_width", + aspect_ratio=1.0, + ) + + payload = serialize_viewable(pane, disable_widgets=True) + + assert payload["height"] == 400 + assert payload["aspect_ratio"] == 1.0 + + def test_serialize_viewable_supports_widget_nodes_with_export_metadata() -> None: radio = pn.widgets.RadioButtonGroup( name="Legacy Mode Name", @@ -240,6 +254,31 @@ def test_sanitize_export_payload_removes_nan_and_infinity() -> None: } +def test_serialize_viewable_preserves_shortened_tab_and_column_tooltips() -> None: + tabs = pn.Tabs( + ( + "Regional Transportat…050 North", + pn.widgets.Tabulator( + pd.DataFrame({"long_column": [1]}), + titles={"long_column": "Long Column…Title"}, + header_tooltips={"long_column": "Long Column Full Title"}, + ), + ) + ) + tabs._run_label_full_titles = ( + "Regional Transportation Scenario Baseline 2050 North", + ) + + payload = serialize_viewable(tabs, disable_widgets=True) + + assert payload["tabs"][0]["full_title"] == ( + "Regional Transportation Scenario Baseline 2050 North" + ) + assert payload["tabs"][0]["content"]["column_tooltips"] == { + "Long Column…Title": "Long Column Full Title" + } + + def test_sanitize_export_payload_in_place_retains_existing_containers() -> None: nested = [1.0, np.float64(2.5), float("nan")] payload = { diff --git a/tests/test_figure_builders.py b/tests/test_figure_builders.py index a447a5f..316b869 100644 --- a/tests/test_figure_builders.py +++ b/tests/test_figure_builders.py @@ -7,6 +7,25 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1])) from dashboard.rendering import Plotter, RenderContext +from dashboard.rendering.labels import display_label_map + + +def test_display_run_labels_preserve_distinct_ends_and_deduplicate_collisions() -> None: + labels = [ + "Regional Transportation Scenario Baseline 2050 North", + "Regional Transportation Scenario Baseline 2050 South", + f"{'A' * 20} first {'Z' * 12}", + f"{'A' * 20} second {'Z' * 12}", + ] + + display_labels = display_label_map(labels) + + assert display_labels[labels[0]] == "Regional Transportation…North" + assert display_labels[labels[1]] == "Regional Transportation…South" + assert len(set(display_labels.values())) == len(labels) + assert all(len(label) <= 30 for label in display_labels.values()) + assert display_labels[labels[2]].endswith("[1]") + assert display_labels[labels[3]].endswith("[2]") def test_figure_first_bar_omits_undeclared_hover_columns() -> None: @@ -77,6 +96,25 @@ def test_bar_and_density_chart_hover_formatting_matches_units() -> None: ) +def test_long_run_names_are_short_in_legends_and_full_in_hovers() -> None: + label = "Regional Transportation Scenario Baseline 2050 North" + data = [(label, pl.DataFrame({"period": [1, 2], "value": [10.0, 20.0]}))] + context = RenderContext(run_labels=(label,)) + + bar = Plotter(context).figure.bar(data, x="period", y="value") + line = Plotter(context).figure.line(data, x="period", y="value") + density = Plotter(context).figure.density(data, x="period", y="value") + scatter = Plotter(context).figure.scatter(data, x="period", y="value") + + for figure in (bar, line, density, scatter): + trace = figure.data[0] + assert trace.name == "Regional Transportation…North" + assert trace.meta == {"run_name": label} + hover = trace.hovertemplate + "".join(map(str, trace.customdata or [])) + assert "Regional Transportation Scenario
    Baseline 2050 North" in hover + assert scatter.data[0].legendgroup == label + + def test_bar_chart_uses_configured_all_series_hover_mode() -> None: data = [ ("Base", pl.DataFrame({"mode": ["Walk", "Bike"], "trip_count": [5.0, 1.0]})), @@ -109,13 +147,93 @@ def test_scatter_chart_can_add_one_to_one_reference_line() -> None: [("Base", pl.DataFrame({"observed": [10.0, 20.0], "modeled": [12.0, 25.0]}))], x="observed", y="modeled", + fit_overlays=[ + ( + "Base", + pl.DataFrame( + { + "observed": [0.0, 100.0], + "modeled": [-100.0, 200.0], + "annotation": [ + "Base
    y = 3.00x - 100.00
    R² = 0.90
    n = 2" + ] + * 2, + } + ), + ) + ], + x_title="Observed Count (vehicles)", + y_title="Modeled Volume (vehicles)", one_to_one=True, ) + point_trace = chart.object.data[0] reference_line = chart.object.data[-1] + fit_line = chart.object.data[-2] + assert point_trace.hovertemplate == ( + "Base
    Observed Count (vehicles): %{x}
    " + "Modeled Volume (vehicles): %{y}" + ) + assert fit_line.name == "Base fit" + assert "y = 3.00x - 100.00" in fit_line.hovertemplate + assert not chart.object.layout.annotations + assert chart.object.layout.height == 400 + assert chart.object.layout.margin.t == 90 assert reference_line.name == "1:1 line" - assert list(reference_line.x) == [0.0, 25.0] - assert list(reference_line.y) == [0.0, 25.0] + assert list(reference_line.x) == [10.0, 25.0] + assert list(reference_line.y) == [10.0, 25.0] assert reference_line.line.color == "#BDBDBD" assert reference_line.line.dash == "dash" - assert reference_line.showlegend is False + assert reference_line.showlegend is True + assert list(chart.object.layout.xaxis.range) == [10.0, 25.0] + assert list(chart.object.layout.yaxis.range) == [10.0, 25.0] + assert chart.object.layout.xaxis.constrain == "domain" + assert chart.object.layout.yaxis.constrain == "domain" + assert chart.object.layout.yaxis.scaleanchor == "x" + assert chart.object.layout.yaxis.scaleratio == 1.0 + + +def test_scatter_fit_details_are_hover_only_for_multiple_runs() -> None: + labels = [f"Run {index}" for index in range(4)] + scatter_data = [ + ( + label, + pl.DataFrame({"observed": [10.0, 20.0], "modeled": [12.0, 25.0]}), + ) + for label in labels + ] + fit_data = [ + ( + label, + pl.DataFrame( + { + "observed": [0.0, 100.0], + "modeled": [-100.0, 200.0], + "annotation": [ + f"{label}
    y = 3.00x - 100.00
    R² = 0.90
    n = 2" + ] + * 2, + } + ), + ) + for label in labels + ] + + chart = Plotter(RenderContext()).scatter( + scatter_data, + x="observed", + y="modeled", + x_title="Observed Count (vehicles)", + y_title="Modeled Volume (vehicles)", + fit_overlays=fit_data, + one_to_one=True, + panel_aspect_ratio=1.0, + ) + + assert not chart.object.layout.annotations + assert chart.object.layout.height == 400 + assert chart.object.layout.margin.t == 90 + assert all("R² = 0.90" in trace.hovertemplate for trace in chart.object.data[4:8]) + assert list(chart.object.layout.xaxis.range) == [10.0, 25.0] + assert list(chart.object.layout.yaxis.range) == [10.0, 25.0] + assert chart.aspect_ratio == 1.0 diff --git a/tests/test_joint_party_weighting.py b/tests/test_joint_party_weighting.py new file mode 100644 index 0000000..7fb02ef --- /dev/null +++ b/tests/test_joint_party_weighting.py @@ -0,0 +1,312 @@ +from __future__ import annotations + +from types import SimpleNamespace + +import polars as pl +import pytest + +from processor.models import RunData +from processor.summarize.cache_types import strip_weights +from processor.summarize.summaries.daily_travel_activity import ( + tour_rate_per_person, + trip_rate_per_person, +) +from processor.summarize.summaries.demographics import population_totals +from processor.summarize.summaries.trip import trip_mode +from processor.summarize.summaries.trip_distributions import trip_distance + + +def _config() -> SimpleNamespace: + return SimpleNamespace() + + +def _run() -> RunData: + return RunData( + label="Test", + run_dir="C:/runs/test", + skim_file=None, + hh=pl.DataFrame({"finalweight": [5.0]}), + per=pl.DataFrame( + { + "person_id": [1, 2], + "person_type": ["worker", "student"], + "finalweight": [2.0, 3.0], + } + ), + tours=pl.DataFrame( + { + "person_id": [1, 1, 2, 2, 2, 2, 2], + "tour_purpose": [ + "work", + "eat", + "shopping", + "shopping", + "escort", + "social", + "other", + ], + "tour_category": [ + "mandatory", + "atwork", + "joint", + "non_mandatory", + "joint", + "joint", + "joint", + ], + "NUMBER_HH": [1, 1, 4, 995, 995, 0, None], + "finalweight": [2.0, 100.0, 3.0, 5.0, 7.0, 11.0, 13.0], + } + ), + trips=pl.DataFrame( + { + "person_id": [1, 2, 2, 2, 2, 2], + "tour_purpose": [ + "work", + "shopping", + "escort", + "social", + "other", + "shopping", + ], + "trip_purpose": [ + "work", + "shopping", + "escort", + "social", + "other", + "shopping", + ], + "tour_category": [ + "mandatory", + "joint", + "joint", + "joint", + "joint", + "non_mandatory", + ], + "tour_mode": ["SOV", "HOV", "HOV", "HOV", "HOV", "SOV"], + "trip_mode": ["SOV", "HOV", "HOV", "HOV", "HOV", "SOV"], + "num_participants": [995, 4, 995, 0, None, 4], + "od_dist": [1.2] * 6, + "stops": [1, 1, 1, 0, 1, 1], + "finalweight": [2.0, 3.0, 5.0, 7.0, 11.0, 13.0], + } + ), + joint_participants=pl.DataFrame(), + land_use=pl.DataFrame(), + skim_matrix=None, + skim_zone_map=None, + ) + + +def test_daily_rates_expand_only_valid_joint_parties_and_exclude_atwork() -> None: + run = _run() + + tour_rates = tour_rate_per_person(run, _config()).filter( + pl.col("person_type") == "all_person_types" + ) + trip_rates = trip_rate_per_person(run, _config()).filter( + pl.col("person_type") == "all_person_types" + ) + + assert "eat" not in tour_rates["tour_purpose"].to_list() + assert dict(zip(tour_rates["tour_purpose"], tour_rates["tour_rate"])) == ( + pytest.approx( + { + "escort": 7.0 / 5.0, + "other": 13.0 / 5.0, + "shopping": 17.0 / 5.0, + "social": 11.0 / 5.0, + "work": 2.0 / 5.0, + } + ) + ) + assert dict(zip(trip_rates["trip_purpose"], trip_rates["trip_rate"])) == ( + pytest.approx( + { + "escort": 5.0 / 5.0, + "other": 11.0 / 5.0, + "shopping": 25.0 / 5.0, + "social": 7.0 / 5.0, + "work": 2.0 / 5.0, + } + ) + ) + + +def test_daily_rates_use_each_joint_participants_weight() -> None: + run = _run() + run.per = pl.DataFrame( + { + "person_id": [1, 2, 3], + "person_type": ["worker", "student", "child"], + "finalweight": [2.0, 1.0, 3.0], + } + ) + run.tours = pl.DataFrame( + { + "tour_id": [10, 20], + "person_id": [1, 1], + "tour_purpose": ["shopping", "escort"], + "tour_category": ["joint", "joint"], + "NUMBER_HH": [3, 2], + "finalweight": [4.0, 5.0], + } + ) + run.trips = pl.DataFrame( + { + "tour_id": [10, 20], + "person_id": [1, 1], + "trip_purpose": ["shopping", "escort"], + "tour_category": ["joint", "joint"], + "num_participants": [3, 2], + "finalweight": [2.0, 3.0], + } + ) + run.joint_participants = pl.DataFrame( + { + "tour_id": [10, 10, 10], + "person_id": [1, 2, 3], + } + ) + + tour_rates = tour_rate_per_person(run, _config()) + trip_rates = trip_rate_per_person(run, _config()) + unweighted_tour_rates = tour_rate_per_person(strip_weights(run), _config()) + unweighted_trip_rates = trip_rate_per_person(strip_weights(run), _config()) + + shopping_tours = tour_rates.filter(pl.col("tour_purpose") == "shopping") + assert dict(zip(shopping_tours["person_type"], shopping_tours["tour_rate"])) == ( + pytest.approx( + { + "worker": 1.0, + "student": 1.0, + "child": 1.0, + "all_person_types": 1.0, + } + ) + ) + escort_tours = tour_rates.filter(pl.col("tour_purpose") == "escort") + assert dict(zip(escort_tours["person_type"], escort_tours["tour_rate"])) == ( + pytest.approx( + { + "worker": 10.0 / 2.0, + "all_person_types": 10.0 / 6.0, + } + ) + ) + shopping_trips = trip_rates.filter(pl.col("trip_purpose") == "shopping") + assert dict(zip(shopping_trips["person_type"], shopping_trips["trip_rate"])) == ( + pytest.approx( + { + "worker": 1.0, + "student": 1.0, + "child": 1.0, + "all_person_types": 1.0, + } + ) + ) + assert unweighted_tour_rates.filter(pl.col("tour_purpose") == "shopping")[ + "tour_rate" + ].to_list() == [1.0, 1.0, 1.0, 1.0] + assert unweighted_trip_rates.filter(pl.col("trip_purpose") == "shopping")[ + "trip_rate" + ].to_list() == [1.0, 1.0, 1.0, 1.0] + escort_trips = trip_rates.filter(pl.col("trip_purpose") == "escort") + assert dict(zip(escort_trips["person_type"], escort_trips["trip_rate"])) == ( + pytest.approx( + { + "worker": 6.0 / 2.0, + "all_person_types": 6.0 / 6.0, + } + ) + ) + + +def test_daily_rates_do_not_multiply_person_level_joint_rows() -> None: + run = _run() + run.per = pl.DataFrame( + { + "person_id": [1, 2], + "person_type": ["worker", "student"], + "finalweight": [2.0, 1.0], + } + ) + run.tours = pl.DataFrame( + { + "tour_id": [10, 11], + "person_id": [1, 2], + "tour_purpose": ["shopping", "shopping"], + "tour_category": ["joint", "joint"], + "NUMBER_HH": [2, 2], + "finalweight": [2.0, 1.0], + } + ) + run.trips = pl.DataFrame( + { + "tour_id": [10, 11], + "person_id": [1, 2], + "trip_purpose": ["shopping", "shopping"], + "tour_category": ["joint", "joint"], + "num_participants": [2, 2], + "finalweight": [2.0, 1.0], + } + ) + run.joint_participants = pl.DataFrame( + { + "tour_id": [10, 11], + "person_id": [1, 2], + } + ) + + for mode_run in (run, strip_weights(run)): + tour_rate = tour_rate_per_person(mode_run, _config()).filter( + pl.col("person_type") == "all_person_types" + )["tour_rate"].sum() + trip_rate = trip_rate_per_person(mode_run, _config()).filter( + pl.col("person_type") == "all_person_types" + )["trip_rate"].sum() + + assert tour_rate == pytest.approx(1.0) + assert trip_rate == pytest.approx(1.0) + + +def test_population_totals_count_joint_travel_for_each_participant() -> None: + result = population_totals(_run()) + + assert result.to_dicts() == [ + { + "person_count": 5.0, + "household_count": 5.0, + "tour_count": 150.0, + "trip_count": 50.0, + "stop_count": 43.0, + } + ] + + +def test_trip_mode_and_distance_ignore_sentinel_and_nonjoint_party_sizes() -> None: + run = _run() + + modes = trip_mode(run, _config()) + shopping_sov = modes.filter( + (pl.col("tour_purpose") == "shopping") + & (pl.col("tour_mode") == "SOV") + & (pl.col("trip_mode") == "SOV") + ) + shopping_hov = modes.filter( + (pl.col("tour_purpose") == "shopping") + & (pl.col("tour_mode") == "HOV") + & (pl.col("trip_mode") == "HOV") + ) + distances = trip_distance(run, _config()).filter(pl.col("distance_bin") == "1") + + assert shopping_sov["trip_count"].to_list() == [13.0] + assert shopping_hov["trip_count"].to_list() == [12.0] + assert distances.filter(pl.col("tour_purpose") == "shopping")[ + "trip_count" + ].to_list() == [25.0] + assert distances.filter(pl.col("tour_purpose") == "all_tour_purposes")[ + "trip_count" + ].to_list() == [50.0] diff --git a/tests/test_joint_tour_frequency.py b/tests/test_joint_tour_frequency.py new file mode 100644 index 0000000..45905b0 --- /dev/null +++ b/tests/test_joint_tour_frequency.py @@ -0,0 +1,365 @@ +from __future__ import annotations + +from types import SimpleNamespace + +import polars as pl +import pytest + +from processor.models import RunData +from processor.summarize.cache_types import strip_weights +from processor.summarize.summaries.joint_travel import ( + joint_composition, + joint_composition_by_party_size, + joint_party_size, + joint_participation_person_by_hhsize, + joint_tour_freq, + jtf_by_hhsize, +) +from processor.summarize.summaries.summary_helpers import ( + household_tour_weight_expr, +) +from processor.summarize.summaries.tour import tour_category, tour_purpose +from processor.summarize.summaries.tour_profiles import stop_freq, tour_distance, tour_tod + + +def _run_with_tours(tours: pl.DataFrame) -> RunData: + return RunData( + label="Test", + run_dir="C:/runs/test", + skim_file=None, + hh=pl.DataFrame(), + per=pl.DataFrame(), + tours=tours, + trips=pl.DataFrame(), + joint_participants=pl.DataFrame(), + land_use=pl.DataFrame(), + skim_matrix=None, + ) + + +def test_joint_tour_frequency_uses_fixed_legacy_purpose_alternatives() -> None: + households = pl.DataFrame( + { + "household_id": [1, 2, 3, 4, 5, 6, 7], + "finalweight": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0], + } + ) + tours = pl.DataFrame( + { + "household_id": [2, 3, 4, 4, 5, 5, 6, 6, 7, 7], + "tour_category": ["joint"] * 10, + "tour_purpose": [ + "shopping", + "othmaint", + "maintenance", + "othmaint", + "SHOPPING", + "maintenance", + "eatout", + "social", + "othdiscr", + "other_discretionary", + ], + } + ) + run = RunData( + label="Test", + run_dir="C:/runs/test", + skim_file=None, + hh=households, + per=pl.DataFrame(), + tours=tours, + trips=pl.DataFrame(), + joint_participants=pl.DataFrame(), + land_use=pl.DataFrame(), + skim_matrix=None, + ) + + result = joint_tour_freq(run).filter(pl.col("household_count") > 0) + + assert result.to_dicts() == [ + {"jtf_code": 1, "jtf_label": "No Joint Tours", "household_count": 1.0}, + {"jtf_code": 2, "jtf_label": "1 Shopping", "household_count": 2.0}, + {"jtf_code": 3, "jtf_label": "1 Maintenance", "household_count": 3.0}, + { + "jtf_code": 8, + "jtf_label": "1 Shopping / 1 Maintenance", + "household_count": 5.0, + }, + {"jtf_code": 12, "jtf_label": "2 Maintenance", "household_count": 4.0}, + { + "jtf_code": 17, + "jtf_label": "1 Eating Out / 1 Visiting", + "household_count": 6.0, + }, + { + "jtf_code": 21, + "jtf_label": "2 Other Discretionary", + "household_count": 7.0, + }, + ] + + +def test_joint_tour_frequency_does_not_pool_multiday_diaries() -> None: + run = RunData( + label="Multi-day Survey", + run_dir="C:/runs/test", + skim_file=None, + hh=pl.DataFrame( + { + "household_id": [1], + "HHSIZE": [2], + "finalweight": [3.0], + } + ), + per=pl.DataFrame(), + day=pl.DataFrame( + { + "household_id": [1, 1, 1, 1], + "person_id": [10, 20, 10, 20], + "day_num": [1, 1, 2, 2], + "finalweight": [3.0, 3.0, 3.0, 3.0], + } + ), + tours=pl.DataFrame( + { + "household_id": [1, 1], + "tour_category": ["joint", "joint"], + "tour_purpose": ["shopping", "shopping"], + } + ), + trips=pl.DataFrame(), + joint_participants=pl.DataFrame(), + land_use=pl.DataFrame(), + skim_matrix=None, + ) + + assert joint_tour_freq(run).is_empty() + assert jtf_by_hhsize(run).is_empty() + + +def test_joint_tour_frequency_counts_unique_tours_by_household_day() -> None: + run = RunData( + label="Multi-day Survey", + run_dir="C:/runs/test", + skim_file=None, + hh=pl.DataFrame( + {"household_id": [1], "HHSIZE": [2], "finalweight": [3.0]} + ), + per=pl.DataFrame(), + day=pl.DataFrame( + { + "household_id": [1, 1, 1, 1], + "person_id": [10, 20, 10, 20], + "day_num": [1, 1, 2, 2], + "finalweight": [3.0, 3.0, 3.0, 3.0], + } + ), + tours=pl.DataFrame( + { + "household_id": [1, 1, 1, 1], + "day_num": [1, 1, 2, 2], + "tour_id": [11, 12, 21, 22], + "joint_tour_id": [100, 100, 200, 200], + "tour_category": ["joint"] * 4, + "tour_purpose": ["shopping"] * 4, + } + ), + trips=pl.DataFrame(), + joint_participants=pl.DataFrame(), + land_use=pl.DataFrame(), + skim_matrix=None, + ) + + frequency = joint_tour_freq(run).filter(pl.col("household_count") > 0) + participation = jtf_by_hhsize(run) + + assert frequency.to_dicts() == [ + {"jtf_code": 2, "jtf_label": "1 Shopping", "household_count": 6.0} + ] + assert participation.to_dicts() == [ + {"jtf": "1", "household_size": "2", "household_percent": 100.0} + ] + + +def test_person_joint_participation_uses_person_days_for_multiday_data() -> None: + run = RunData( + label="Multi-day Survey", + run_dir="C:/runs/test", + skim_file=None, + hh=pl.DataFrame( + {"household_id": [1], "HHSIZE": [2], "finalweight": [1.0]} + ), + per=pl.DataFrame( + { + "household_id": [1, 1], + "person_id": [10, 20], + "finalweight": [1.0, 1.0], + } + ), + day=pl.DataFrame( + { + "household_id": [1, 1, 1, 1], + "person_id": [10, 20, 10, 20], + "day_num": [1, 1, 2, 2], + "finalweight": [1.0, 1.0, 1.0, 1.0], + } + ), + tours=pl.DataFrame( + { + "household_id": [1], + "day_num": [1], + "tour_id": [11], + "joint_tour_id": [100], + "tour_category": ["joint"], + } + ), + trips=pl.DataFrame(), + joint_participants=pl.DataFrame( + { + "tour_id": [11, 11], + "joint_tour_id": [100, 100], + "person_id": [10, 20], + } + ), + land_use=pl.DataFrame(), + skim_matrix=None, + ) + + assert joint_participation_person_by_hhsize(run, None).to_dicts() == [ + { + "household_size": 2, + "joint_tour_person_count": 2.0, + "total_person_count": 4.0, + } + ] + + +def test_joint_party_summaries_exclude_995_sentinel() -> None: + tours = pl.DataFrame( + { + "tour_category": ["joint", "joint"], + "NUMBER_HH": [2, 995], + "number_of_participants": [2, 995], + "composition": ["mixed", "mixed"], + "finalweight": [2.0, 100.0], + } + ) + run = RunData( + label="Test", + run_dir="C:/runs/test", + skim_file=None, + hh=pl.DataFrame(), + per=pl.DataFrame(), + tours=tours, + trips=pl.DataFrame(), + joint_participants=pl.DataFrame(), + land_use=pl.DataFrame(), + skim_matrix=None, + ) + + assert joint_party_size(run).to_dicts() == [ + {"party_size": 2, "joint_tour_count": 2.0}, + {"party_size": 5, "joint_tour_count": 0.0}, + ] + assert joint_composition_by_party_size(run, None).to_dicts() == [ + {"tour_composition": "mixed", "party_size": 2, "joint_tour_count": 2.0} + ] + + +def test_household_tour_summaries_fractionally_use_every_participant_row() -> None: + raw_tours = pl.DataFrame( + { + "tour_category": ["joint", "joint", "joint"], + "tour_purpose": ["shopping", "shopping", "shopping"], + "tour_id": [11, 12, 13], + "joint_tour_id": [100, 100, 100], + "NUMBER_HH": [1, 1, 1], + "number_of_participants": [3, 3, 3], + "composition": ["mixed", "mixed", "mixed"], + "start_hour": [10, 10, 10], + "end_hour": [12, 12, 12], + "tourdur": [2, 2, 2], + "SKIMDIST": [4.0, 4.0, 4.0], + "num_ob_stops": pl.Series([1, 1, 1], dtype=pl.Int32), + "num_ib_stops": pl.Series([2, 2, 2], dtype=pl.Int32), + "num_tot_stops": pl.Series([3, 3, 3], dtype=pl.Int32), + "finalweight": [2.0, 2.0, 2.0], + } + ) + model_tours = raw_tours.head(1).drop("joint_tour_id").with_columns( + pl.lit(3).alias("NUMBER_HH") + ) + raw_run = _run_with_tours(raw_tours) + model_run = _run_with_tours(model_tours) + + represented_raw = raw_tours.with_columns( + household_tour_weight_expr(raw_tours, output_col="household_tour_weight") + ) + represented_unweighted = strip_weights(raw_run).tours.with_columns( + household_tour_weight_expr( + strip_weights(raw_run).tours, + output_col="household_tour_weight", + ) + ) + assert represented_raw.height == raw_tours.height + assert represented_raw["household_tour_weight"].to_list() == pytest.approx( + [2.0 / 3.0] * 3 + ) + assert represented_unweighted["household_tour_weight"].to_list() == ( + pytest.approx([1.0 / 3.0] * 3) + ) + + for run, expected in ( + (raw_run, 2.0), + (model_run, 2.0), + (strip_weights(raw_run), 1.0), + (strip_weights(model_run), 1.0), + ): + assert joint_party_size(run).filter(pl.col("party_size") == 3)[ + "joint_tour_count" + ].item() == pytest.approx(expected) + assert 1 not in joint_party_size(run)["party_size"].to_list() + assert joint_composition(run).filter(pl.col("tour_composition") == "mixed")[ + "joint_tour_count" + ].item() == pytest.approx(expected) + assert joint_composition_by_party_size(run, None).filter( + (pl.col("tour_composition") == "mixed") & (pl.col("party_size") == 3) + )["joint_tour_count"].item() == pytest.approx(expected) + assert tour_tod(run, SimpleNamespace()).filter( + (pl.col("tour_purpose") == "shopping") & (pl.col("time_bin") == 10) + )["departure_tour_count"].item() == pytest.approx(expected) + assert tour_distance(run, SimpleNamespace()).filter( + (pl.col("tour_purpose") == "shopping") + & (pl.col("distance_bin") == "4") + )["tour_count"].item() == pytest.approx(expected) + assert stop_freq(run, SimpleNamespace()).filter( + (pl.col("tour_purpose") == "shopping") + & (pl.col("outbound_stop_count") == 1) + & (pl.col("inbound_stop_count") == 2) + )["tour_count"].item() == pytest.approx(expected) + assert tour_category(run, SimpleNamespace()).filter( + pl.col("tour_category") == "joint" + )["tour_count"].item() == pytest.approx(expected) + assert tour_purpose(run, SimpleNamespace()).filter( + pl.col("tour_purpose") == "shopping" + )["tour_count"].item() == pytest.approx(expected) + + +def test_household_tour_weight_does_not_use_sentinel_party_size() -> None: + tours = pl.DataFrame( + { + "tour_category": ["joint", "joint"], + "joint_tour_id": [200, 200], + "NUMBER_HH": [995, 995], + "number_of_participants": [995, 995], + "finalweight": [4.0, 4.0], + } + ) + + represented = tours.with_columns( + household_tour_weight_expr(tours, output_col="household_tour_weight") + ) + + assert represented.height == tours.height + assert represented["household_tour_weight"].to_list() == [4.0, 4.0] diff --git a/tests/test_page_registry_contract.py b/tests/test_page_registry_contract.py index 838d68c..43c3186 100644 --- a/tests/test_page_registry_contract.py +++ b/tests/test_page_registry_contract.py @@ -3,20 +3,62 @@ from pathlib import Path import sys +import panel as pn + sys.path.insert(0, str(Path(__file__).resolve().parents[1])) sys.path.insert(0, str(Path(__file__).resolve().parent)) from dashboard import DashboardState +from dashboard.data_access import DashboardPreparedRunProvider from dashboard.export.protocols import validate_export_page from dashboard.export.traversal import resolve_page_parts from dashboard.page_registry import ( + all_page_definitions, build_registered_live_pages, build_registered_export_pages, + page_definition_by_id, +) +from dashboard.pages.daily_travel._escorted_tours.contracts import ( + CORE_SUMMARY_IDS, + OPTIONAL_SUMMARY_IDS, ) +from processor.summarize.cache_types import create_summary_run from _dashboard_expectations import EXPECTED_DEFAULT_LEAF_PAGE_IDS from test_export_html import _full_summary_run, _write_config +def _walk(viewable): + yield viewable + for child in getattr(viewable, "objects", []): + yield from _walk(child) + + +def _assert_sections_render_outcomes(page, *, allow_no_runs: bool = False) -> None: + for section in page.registered_sections: + context = f"{page.page_id()}.{section.section_id}" + nodes = list(_walk(section.container)) + cards = [node for node in nodes if isinstance(node, pn.Card)] + plots = [node for node in nodes if isinstance(node, pn.pane.Plotly)] + tables = [node for node in nodes if isinstance(node, pn.widgets.Tabulator)] + no_run_messages = [ + node + for node in nodes + if isinstance(node, pn.pane.Markdown) + and str(node.object) == "No runs loaded." + ] + assert cards or plots or tables or (allow_no_runs and no_run_messages), context + assert all(plot.object.data for plot in plots), context + assert all(tabs.objects for tabs in nodes if isinstance(tabs, pn.Tabs)), context + + +def _state_for_run(run, config) -> DashboardState: + return DashboardState( + summary_runs=[run], + weighting_modes=config.weighting_modes, + prepared_run_provider=DashboardPreparedRunProvider.unavailable(), + ) + + def test_all_registered_export_pages_satisfy_export_protocol(tmp_path: Path) -> None: config = _write_config(tmp_path) state = DashboardState( @@ -113,3 +155,78 @@ def test_representative_export_pages_keep_expected_runtime_sections( (part_def.part_id, tuple(part_def.selector_ids)) for part_def, _ in resolved_parts ] == expected + + +def test_escorted_tours_declares_independent_addons_as_optional() -> None: + definition = page_definition_by_id("escorted_tours") + assert definition is not None + assert definition.required_summary_ids == CORE_SUMMARY_IDS + assert definition.optional_summary_ids == OPTIONAL_SUMMARY_IDS + + +def test_all_registered_page_sections_explain_no_run_sessions(tmp_path: Path) -> None: + config = _write_config(tmp_path) + + for definition in all_page_definitions(): + state = DashboardState( + summary_runs=[], + weighting_modes=config.weighting_modes, + ) + page = definition.page_cls(state, config) + page.refresh(force=True) + _assert_sections_render_outcomes(page, allow_no_runs=True) + + +def test_all_registered_page_selector_states_render_valid_outcomes( + tmp_path: Path, +) -> None: + config = _write_config(tmp_path) + summary_run = _full_summary_run() + + for definition in all_page_definitions(): + page = definition.page_cls(_state_for_run(summary_run, config), config) + page.refresh(force=True) + _assert_sections_render_outcomes(page) + + for selector in page.registered_selectors: + widget = selector.widget + values = ( + [False, True] + if isinstance(widget, pn.widgets.Checkbox) + else list(getattr(widget, "options", []) or []) + ) + for value in values: + widget.value = value + page.refresh(force=False) + _assert_sections_render_outcomes(page) + + +def test_all_registered_pages_handle_each_missing_declared_summary( + tmp_path: Path, +) -> None: + config = _write_config(tmp_path) + full_run = _full_summary_run() + + for definition in all_page_definitions(): + summary_ids = ( + *definition.required_summary_ids, + *definition.optional_summary_ids, + ) + for missing_summary_id in summary_ids: + summaries_by_mode = { + mode: { + summary_id: table + for summary_id, table in tables.items() + if summary_id != missing_summary_id + } + for mode, tables in full_run.summaries_by_mode.items() + } + summary_run = create_summary_run( + label="Base", + run_key="base", + summaries_by_mode=summaries_by_mode, + source_run_dir="C:/runs/base", + ) + page = definition.page_cls(_state_for_run(summary_run, config), config) + page.refresh(force=True) + _assert_sections_render_outcomes(page) diff --git a/tests/test_person_travel_weighting.py b/tests/test_person_travel_weighting.py new file mode 100644 index 0000000..b72668f --- /dev/null +++ b/tests/test_person_travel_weighting.py @@ -0,0 +1,166 @@ +from __future__ import annotations + +from types import SimpleNamespace + +import polars as pl +import pytest +from polars.testing import assert_frame_equal + +from processor.models import RunData +from processor.summarize.cache_types import strip_weights +from processor.summarize.summaries.demographics import population_totals +from processor.summarize.summaries.tour_profiles import tour_mode +from processor.summarize.summaries.trip import ( + stop_purpose_by_tour_purpose, + trip_mode, + trip_purpose, +) +from processor.summarize.summaries.trip_distributions import ( + stop_ood_distance, + trip_distance, + trip_stop_tod, +) + + +def _config() -> SimpleNamespace: + return SimpleNamespace() + + +def _person_travel_run(*, participant_rows: bool) -> RunData: + person_ids = [1, 2, 3] + person_weights = [2.0, 3.0, 4.0] + if participant_rows: + tour_ids = [101, 102, 103] + tours = pl.DataFrame( + { + "tour_id": tour_ids, + "person_id": person_ids, + "tour_purpose": ["shopping"] * 3, + "tour_category": ["joint"] * 3, + "tour_mode": ["HOV3"] * 3, + "NUMBER_HH": [3] * 3, + "AUTOSUFF": [0] * 3, + "finalweight": [97.0] * 3, + } + ) + trips = pl.DataFrame( + { + "tour_id": [tour_id for tour_id in tour_ids for _ in range(2)], + "person_id": [person_id for person_id in person_ids for _ in range(2)], + "tour_purpose": ["shopping"] * 6, + "trip_purpose": ["shopping", "eatout"] * 3, + "tour_category": ["joint"] * 6, + "tour_mode": ["HOV3"] * 6, + "trip_mode": ["HOV3"] * 6, + "num_participants": [3] * 6, + "od_dist": [1.2, 2.2] * 3, + "out_dir_dist": [0.0, 3.2] * 3, + "depart": [5, 6] * 3, + "stops": [0, 1] * 3, + "finalweight": [89.0] * 6, + } + ) + joint_participants = pl.DataFrame( + {"tour_id": tour_ids, "person_id": person_ids} + ) + else: + tours = pl.DataFrame( + { + "tour_id": [10], + "person_id": [1], + "tour_purpose": ["shopping"], + "tour_category": ["joint"], + "tour_mode": ["HOV3"], + "NUMBER_HH": [3], + "AUTOSUFF": [0], + "finalweight": [97.0], + } + ) + trips = pl.DataFrame( + { + "tour_id": [10, 10], + "person_id": [1, 1], + "tour_purpose": ["shopping", "shopping"], + "trip_purpose": ["shopping", "eatout"], + "tour_category": ["joint", "joint"], + "tour_mode": ["HOV3", "HOV3"], + "trip_mode": ["HOV3", "HOV3"], + "num_participants": [3, 3], + "od_dist": [1.2, 2.2], + "out_dir_dist": [0.0, 3.2], + "depart": [5, 6], + "stops": [0, 1], + "finalweight": [89.0, 89.0], + } + ) + joint_participants = pl.DataFrame( + {"tour_id": [10, 10, 10], "person_id": person_ids} + ) + + return RunData( + label="Raw participants" if participant_rows else "Model household", + run_dir="C:/runs/test", + skim_file=None, + hh=pl.DataFrame({"finalweight": [4.0]}), + per=pl.DataFrame( + {"person_id": person_ids, "finalweight": person_weights} + ), + tours=tours, + trips=trips, + joint_participants=joint_participants, + land_use=pl.DataFrame(), + skim_matrix=None, + ) + + +@pytest.mark.parametrize( + ("unweighted", "expected_tour_count", "expected_trip_count"), + [(False, 9.0, 18.0), (True, 3.0, 6.0)], +) +def test_person_travel_summaries_match_model_and_raw_participant_rows( + unweighted: bool, + expected_tour_count: float, + expected_trip_count: float, +) -> None: + model = _person_travel_run(participant_rows=False) + raw = _person_travel_run(participant_rows=True) + if unweighted: + model = strip_weights(model) + raw = strip_weights(raw) + + builders = ( + population_totals, + tour_mode, + trip_purpose, + stop_purpose_by_tour_purpose, + trip_mode, + trip_stop_tod, + trip_distance, + stop_ood_distance, + ) + for builder in builders: + assert_frame_equal( + builder(model, _config()), + builder(raw, _config()), + check_row_order=False, + ) + + totals = population_totals(model) + assert totals["tour_count"][0] == pytest.approx(expected_tour_count) + assert totals["trip_count"][0] == pytest.approx(expected_trip_count) + assert totals["stop_count"][0] == pytest.approx(expected_tour_count) + + mode_count = tour_mode(model, _config()).filter( + pl.col("tour_purpose") == "shopping" + )["tour_count_all_households"][0] + assert mode_count == pytest.approx(expected_tour_count) + + trip_tod = trip_stop_tod(model, _config()).filter( + (pl.col("tour_purpose") == "shopping") & (pl.col("time_bin") == 6) + ) + assert trip_tod["departure_trip_count"][0] == pytest.approx( + expected_tour_count + ) + assert trip_tod["departure_stop_count"][0] == pytest.approx( + expected_tour_count + ) diff --git a/tests/test_prepare_cache.py b/tests/test_prepare_cache.py index a66cf1a..4c71606 100644 --- a/tests/test_prepare_cache.py +++ b/tests/test_prepare_cache.py @@ -20,6 +20,7 @@ ) from processor.prepare.cache import ( PreparedCacheError, + build_prepared_manifest_identity, build_run_fingerprint, load_custom_prepared_tables, load_prepared_run_cache, @@ -162,6 +163,25 @@ def _write_custom_prepared_tables( ), "joint_tour_participants": pl.DataFrame({"tour_id": [], "person_id": []}), "land_use": pl.DataFrame({"zone_id": [1], "TAZ": [1]}), + "trip_hypothetical_skims": pl.DataFrame( + { + "trip_id": [1000], + "hypothetical_mode": ["HOV2"], + "component": ["skim_auto_time"], + "value": [12.0], + "finalweight": [1.0], + } + ), + "tour_hypothetical_skims": pl.DataFrame( + { + "tour_id": [100], + "hypothetical_mode": ["HOV2"], + "component": ["skim_auto_time_outbound"], + "direction": ["outbound"], + "value": [6.0], + "finalweight": [1.0], + } + ), } paths: dict[str, str] = {} for table_id, table in tables.items(): @@ -388,6 +408,29 @@ def test_prepared_cache_round_trip_creates_default_layout(tmp_path: Path) -> Non assert loaded.trip_weight_col == "trip_weight" +def test_prepared_cache_removes_stale_hypothetical_skim_sidecars( + tmp_path: Path, +) -> None: + config = _write_config(tmp_path) + prepared = _prepared_run(config) + prepared.trip_hypothetical_skims = pl.DataFrame({"trip_id": [5001]}) + prepared.tour_hypothetical_skims = pl.DataFrame({"tour_id": [1001]}) + + entry = write_prepared_run_cache(prepared, config, run_key="base") + trip_path = entry.cache_dir / "trip_hypothetical_skims.parquet" + tour_path = entry.cache_dir / "tour_hypothetical_skims.parquet" + assert trip_path.exists() + assert tour_path.exists() + + prepared.trip_hypothetical_skims = pl.DataFrame() + prepared.tour_hypothetical_skims = pl.DataFrame() + entry = write_prepared_run_cache(prepared, config, run_key="base") + + assert entry.manifest["sidecar_files"] == {} + assert not trip_path.exists() + assert not tour_path.exists() + + def test_prepared_cache_round_trips_skimjoin_resolved_network_los( tmp_path: Path, ) -> None: @@ -473,6 +516,8 @@ def test_config_accepts_custom_prepared_table_map_and_csv_prepare_output( f" persons: {(custom_dir / 'persons.csv').as_posix()}", f" day: {(custom_dir / 'day.csv').as_posix()}", f" vehicles: {(custom_dir / 'vehicles.parquet').as_posix()}", + f" trip_hypothetical_skims: {(custom_dir / 'trip_hypothetical_skims.parquet').as_posix()}", + f" tour_hypothetical_skims: {(custom_dir / 'tour_hypothetical_skims.csv').as_posix()}", ], ) @@ -485,6 +530,12 @@ def test_config_accepts_custom_prepared_table_map_and_csv_prepare_output( assert config.runs[0]["prepared_table_map"]["vehicles"].endswith( "vehicles.parquet" ) + assert config.runs[0]["prepared_table_map"]["trip_hypothetical_skims"].endswith( + "trip_hypothetical_skims.parquet" + ) + assert config.runs[0]["prepared_table_map"]["tour_hypothetical_skims"].endswith( + "tour_hypothetical_skims.csv" + ) def test_config_accepts_prepare_relationship_validation_mode(tmp_path: Path) -> None: @@ -642,10 +693,52 @@ def test_load_custom_prepared_tables_supports_parquet_and_csv( assert csv_loaded.per["person_id"].to_list() == [10] assert parquet_loaded.day["day_id"].to_list() == [100] assert csv_loaded.vehicles["vehicle_id"].to_list() == [1001] + assert parquet_loaded.trip_hypothetical_skims["value"].to_list() == [12.0] + assert csv_loaded.tour_hypothetical_skims["direction"].to_list() == ["outbound"] assert table_availability(parquet_loaded)["households"] == "available" assert table_availability(csv_loaded)["trips"] == "available" +def test_load_custom_prepared_tables_reports_missing_configured_sidecar( + tmp_path: Path, +) -> None: + prepared_map = _write_custom_prepared_tables(tmp_path, file_format="parquet") + missing_path = tmp_path / "missing_trip_hypothetical_skims.parquet" + prepared_map["trip_hypothetical_skims"] = str(missing_path) + + loaded = load_custom_prepared_tables( + prepared_table_map=prepared_map, + label="Missing Sidecar", + ) + + assert loaded.trip_hypothetical_skims.is_empty() + assert table_availability(loaded)["trip_hypothetical_skims"] == "unavailable" + assert table_unavailable_reasons(loaded)["trip_hypothetical_skims"] == ( + f"Missing prepared sidecar file: {missing_path}" + ) + + +def test_prepared_manifest_identity_allows_missing_custom_sidecar( + tmp_path: Path, +) -> None: + config = _write_config(tmp_path) + prepared_map = _write_custom_prepared_tables(tmp_path / "prepared") + missing_path = tmp_path / "prepared" / "missing_tour_hypothetical_skims.parquet" + prepared_map["tour_hypothetical_skims"] = str(missing_path) + + identity = build_prepared_manifest_identity( + run_key="base", + config=config, + run_fingerprint={}, + source_type="custom_prepared_table_map", + prepared_table_map=prepared_map, + ) + + fingerprints = identity["prepared_table_fingerprints"] + assert fingerprints["tour_hypothetical_skims"] is None + assert fingerprints["trip_hypothetical_skims"]["size"] > 0 + + def test_load_custom_prepared_tables_retries_csv_with_full_schema_inference( tmp_path: Path, ) -> None: diff --git a/tests/test_prepare_weights.py b/tests/test_prepare_weights.py new file mode 100644 index 0000000..a6a9eff --- /dev/null +++ b/tests/test_prepare_weights.py @@ -0,0 +1,191 @@ +from __future__ import annotations + +from pathlib import Path + +import polars as pl + +from processor.models import RunData +from processor.prepare.enrichment.weights import compute_weights +from processor.summarize.cache_types import strip_weights +from runtime.config import Config + + +def _config(tmp_path: Path) -> Config: + config_path = tmp_path / "config.yaml" + config_path.write_text( + "\n".join( + [ + 'name: "Prepare Weight Test"', + "runs: []", + "root: summary_cache", + "dashboard:", + ' title: "Prepare Weight Test"', + ] + ), + encoding="utf-8", + ) + return Config.from_yaml(config_path) + + +def test_person_weights_flow_to_activity_tables_without_dropping_records( + tmp_path: Path, +) -> None: + hh = pl.DataFrame( + { + "household_id": [1, 2], + "hh_weight": [0.25, 2.0], + } + ) + per = pl.DataFrame( + { + "person_id": [10, 20], + "household_id": [1, 2], + "person_weight": [0.25, 2.0], + } + ) + day = pl.DataFrame( + { + "day_id": [100, 200], + "person_id": [10, 20], + "household_id": [1, 2], + } + ) + tours = pl.DataFrame( + { + "tour_id": [101, 102, 201], + "person_id": [10, 10, 20], + "household_id": [1, 1, 2], + "tour_weight": [1.0, 1.0, 1.0], + } + ) + trips = pl.DataFrame( + { + "trip_id": [1001, 1002, 1003, 2001], + "tour_id": [101, 101, 102, 201], + "person_id": [10, 10, 10, 20], + "household_id": [1, 1, 1, 2], + "linked_trip_weight": [1.0, 1.0, 1.0, 1.0], + } + ) + vehicles = pl.DataFrame( + { + "vehicle_id": [1, 2], + "household_id": [1, 2], + } + ) + + weighted_tables = compute_weights( + hh, + per, + day, + tours, + trips, + vehicles, + _config(tmp_path), + hh_weight_col="hh_weight", + person_weight_col="person_weight", + trip_weight_col=None, + ) + ( + weighted_hh, + weighted_per, + weighted_day, + weighted_tours, + weighted_trips, + weighted_vehicles, + ) = weighted_tables + + assert weighted_tours.height == tours.height + assert weighted_trips.height == trips.height + assert ( + weighted_tours.sort("tour_id")["tour_id"].to_list() + == tours.sort("tour_id")["tour_id"].to_list() + ) + assert ( + weighted_trips.sort("trip_id")["trip_id"].to_list() + == trips.sort("trip_id")["trip_id"].to_list() + ) + assert weighted_tours.sort("tour_id")["finalweight"].to_list() == [0.25, 0.25, 2.0] + assert weighted_trips.sort("trip_id")["finalweight"].to_list() == [ + 0.25, + 0.25, + 0.25, + 2.0, + ] + + weighted = RunData( + label="Weighted", + run_dir=".", + skim_file=None, + hh=weighted_hh, + per=weighted_per, + day=weighted_day, + tours=weighted_tours, + trips=weighted_trips, + vehicles=weighted_vehicles, + joint_participants=pl.DataFrame(), + land_use=pl.DataFrame(), + skim_matrix=None, + hh_weight_col="hh_weight", + person_weight_col="person_weight", + ) + unweighted = strip_weights(weighted) + + for weighted_frame, unweighted_frame in ( + (weighted.hh, unweighted.hh), + (weighted.per, unweighted.per), + (weighted.day, unweighted.day), + (weighted.tours, unweighted.tours), + (weighted.trips, unweighted.trips), + (weighted.vehicles, unweighted.vehicles), + ): + assert unweighted_frame.height == weighted_frame.height + assert ( + unweighted_frame["finalweight"].to_list() == [1.0] * weighted_frame.height + ) + + +def test_day_weight_source_can_explicitly_inherit_person_weights(tmp_path: Path) -> None: + hh = pl.DataFrame({"household_id": [1], "hh_weight": [1.0]}) + per = pl.DataFrame( + { + "household_id": [1, 1], + "person_id": [10, 20], + "person_weight": [1 / 7, 2.0], + } + ) + day = pl.DataFrame( + { + "household_id": [1, 1], + "person_id": [10, 20], + "day_weight": [1.0, 1.0], + } + ) + empty = pl.DataFrame() + + default_day = compute_weights( + hh, + per, + day, + empty, + empty, + empty, + _config(tmp_path), + hh_weight_col="hh_weight", + person_weight_col="person_weight", + )[2] + inherited_day = compute_weights( + hh, + per, + day, + empty, + empty, + empty, + _config(tmp_path), + hh_weight_col="hh_weight", + person_weight_col="person_weight", + day_weight_col=None, + )[2] + + assert default_day["finalweight"].to_list() == [1.0, 1.0] + assert inherited_day["finalweight"].to_list() == [1 / 7, 2.0] diff --git a/tests/test_processor_prepare.py b/tests/test_processor_prepare.py index a2b783a..0df4fca 100644 --- a/tests/test_processor_prepare.py +++ b/tests/test_processor_prepare.py @@ -46,7 +46,10 @@ univ_tlfd, work_tlfd, ) -from processor.summarize.summaries.long_term_person import telecommute +from processor.summarize.summaries.long_term_person import ( + license_holding_status, + telecommute, +) from processor.summarize.summaries.long_term_vehicle import ( vehicle_char_age, vehicle_char_body, @@ -590,6 +593,214 @@ def test_processor_prepare_data_exposes_the_same_prepared_contract( assert prepared.land_use["employment_count"].to_list() == [7.0, 8.0] +def test_processor_prepare_data_normalizes_toc_raw_statewide_survey( + tmp_path: Path, +) -> None: + config = _write_config( + tmp_path, + extra_lines=[ + "prepare:", + " category_mappings:", + " raw_statewide_survey:", + " persons:", + " has_license:", + " source: can_drive", + " output_type: boolean", + " preserve_unmapped: false", + " mapping:", + " 1: true", + " 3: false", + " 995: false", + " tours:", + " tour_mode:", + " mapping:", + " 2: WALK", + " 7: BIKE", + " 8: EBIKE", + " 13: SOV", + " tour_purpose:", + " mapping:", + " 1: loop", + " 2: work", + " 3: othmaint", + " 6: escort", + " 8: eatout", + " 14: loop", + " tour_category:", + " mapping:", + " 1: mandatory", + " 2: non_mandatory", + " 3: atwork", + " 4: joint", + " trips:", + " trip_mode:", + " source: linked_trip_mode", + " mapping:", + " 2: WALK", + " 7: BIKE", + " 13: SOV", + " 995: MISSING", + " trip_purpose:", + " source: d_purpose_category", + " mapping:", + " 1: home", + " 6: escort", + " 14: home", + " 995: null", + " vot_bins:", + " source_column: income_segment", + " output_column: vot_bin", + " fallback_value: M", + " mappings:", + " raw_statewide_survey:", + " 2: L", + ], + ) + category_mappings = config.prepare_category_mappings.mapping_for_run( + "Raw Statewide Survey" + ) + assert category_mappings is not None + assert category_mappings["tours"]["tour_mode"].mapping["13"] == "SOV" + assert ( + config.prepare_signature_payload()["prepare"]["category_mappings"] + ["raw-statewide-survey"]["persons"]["has_license"]["output_type"] + == "boolean" + ) + raw = _raw_run() + raw.label = "Raw Statewide Survey" + raw.hh = pl.DataFrame( + { + "household_id": [1], + "num_people": [6], + "num_vehicles": [5], + "num_workers": [1], + "num_adults": [3], + "income_broad": [2], + "home_county": ["Lane"], + "home_mpo": ["Central Lane"], + } + ) + raw.per = pl.DataFrame( + { + "person_id": [101, 102, 103], + "household_id": [1, 1, 1], + "person_type": [1, 4, 7], + "can_drive": [1, 3, 995], + "age": [45, 35, 10], + } + ) + raw.tours = pl.DataFrame( + { + "tour_id": [1001, 1002, 1003, 1004], + "person_id": [101, 101, 102, 103], + "household_id": [1, 1, 1, 1], + "tour_purpose": [2, 6, 3, 14], + "tour_category": [1, 2, 3, 4], + "tour_mode": [13, 7, 2, 8], + "tour_start_hour": [3, 2, 19, 8], + "tour_start_minute": [0, 30, 56, 15], + "tour_end_hour": [4, 3, 20, 9], + "tour_end_minute": [0, 0, 30, 0], + "duration_minutes": [60, 30, 34, 90], + "distance_miles": [1.5, 2.5, 3.5, 4.5], + "joint_num_participants": [1, 1, 1, 3], + "stop_frequency": ["0out_0in"] * 4, + "school_esc_outbound": [None, "ride_share", None, None], + "school_esc_inbound": [None] * 4, + "out_escorted_tour_ids": [None, "1004", None, None], + "inb_escorted_tour_ids": [None] * 4, + } + ) + raw.trips = pl.DataFrame( + { + "trip_id": [5001, 5002, 5003, 5004], + "tour_id": [1001, 1002, 1003, 1004], + "person_id": [101, 101, 102, 103], + "household_id": [1, 1, 1, 1], + "linked_trip_mode": [13, 7, 2, 995], + "d_purpose_category": [1, 14, 6, 995], + "depart_hour": [3, 2, 19, 8], + "depart_minute": [0, 30, 56, 15], + "distance_miles": [1.25, 2.25, 3.25, 4.25], + "outbound": [True, True, True, True], + "trip_num": [1, 1, 1, 1], + } + ) + + prepared = processor_prepare_data(raw, config) + + assert prepared.hh["HHSIZE"].to_list() == [5] + assert prepared.hh["HHVEH"].to_list() == [4] + assert prepared.hh["LICENSEDDRIVERS"].to_list() == [1] + assert prepared.per["has_license"].to_list() == [True, False, False] + + tours = prepared.tours.sort("tour_id") + assert tours["tour_mode"].to_list() == ["SOV", "BIKE", "WALK", "EBIKE"] + assert tours["tour_purpose"].to_list() == [ + "work", + "escort", + "othmaint", + "loop", + ] + assert tours["tour_category"].to_list() == [ + "mandatory", + "non_mandatory", + "atwork", + "joint", + ] + assert tours["start"].to_list() == [1, 48, 34, 11] + assert tours["end"].to_list() == [3, 1, 36, 13] + assert tours["tourdur"].to_list() == [3, 1, 3, 3] + assert "tour_distance" not in tours.columns + assert tours["SKIMDIST"].to_list() == [1.5, 2.5, 3.5, 4.5] + assert tours["NUMBER_HH"].to_list() == [1, 1, 1, 3] + assert tours["vot_bin"].to_list() == ["L", "L", "L", "L"] + + trips = prepared.trips.sort("trip_id") + assert trips["trip_mode"].to_list() == ["SOV", "BIKE", "WALK", "MISSING"] + assert trips["trip_purpose"].to_list() == ["home", "home", "escort", None] + assert trips["depart"].to_list() == [1, 48, 34, 11] + assert trips["od_dist"].to_list() == [1.25, 2.25, 3.25, 4.25] + assert trips["prepared_non_motorized_distance"].to_list() == [ + 1.25, + 2.25, + 3.25, + 4.25, + ] + assert trips["vot_bin"].to_list() == ["L", "L", "L", "L"] + assert set(license_holding_status(prepared, config)["license_holding_status"]) == { + "has_license", + "no_license", + } + + raw.label = "Regional: Filtered & Attributed" + raw.tours = raw.tours.with_columns( + pl.Series("tour_mode", ["SOV", "BIKE", "WALK", "EBIKE"]), + pl.Series( + "tour_category", + ["mandatory", "non_mandatory", "atwork", "joint"], + ), + pl.Series( + "primary_purpose", ["work", "escort", "othmaint", "eatout"] + ), + ) + labeled = processor_prepare_data(raw, config).tours.sort("tour_id") + + assert labeled["tour_mode"].to_list() == ["SOV", "BIKE", "WALK", "EBIKE"] + assert labeled["tour_purpose"].to_list() == [ + "work", + "escort", + "othmaint", + "eatout", + ] + assert labeled["tour_category"].to_list() == [ + "mandatory", + "non_mandatory", + "atwork", + "joint", + ] + + def test_processor_prepare_data_derives_first_inbound_trip_depart_on_tours( tmp_path: Path, ) -> None: @@ -629,6 +840,129 @@ def test_processor_prepare_data_derives_first_inbound_trip_depart_on_tours( assert prepared.tours["start_hour"].to_list() == [8] +def test_processor_prepare_data_constructs_tour_origins_and_boundary_trip_endpoints( + tmp_path: Path, +) -> None: + config = _write_config( + tmp_path, + extra_lines=[ + "zones:", + " use_maz: true", + " maz_col: [zone_id]", + " taz_col: [TAZ]", + ], + ) + raw = _raw_run() + raw = ProcessorRunData( + label=raw.label, + run_dir=raw.run_dir, + skim_file=raw.skim_file, + hh=raw.hh.with_columns(pl.lit(100).alias("home_zone_id")), + per=raw.per, + tours=pl.DataFrame( + { + "tour_id": [1, 2], + "parent_tour_id": [None, 1], + "person_id": [101, 101], + "household_id": [1, 1], + "tour_type": ["work", "eatout"], + "tour_mode": ["DRIVE", "WALK"], + "tour_category": ["mandatory", "atwork"], + "start": [8, 12], + "end": [10, 13], + "duration": [2, 1], + "origin": [999, 888], + "destination": [200, 300], + "stop_frequency": ["0out_0in", "0out_0in"], + } + ), + trips=pl.DataFrame( + { + "trip_id": [11, 12, 21, 22], + "tour_id": [1, 1, 2, 2], + "person_id": [101] * 4, + "household_id": [1] * 4, + "trip_mode": ["DRIVEALONE", "DRIVEALONE", "WALK", "WALK"], + "purpose": ["work", "home", "eatout", "work"], + "depart": [8, 10, 12, 13], + "outbound": [True, False, True, False], + "trip_num": [1, 1, 1, 1], + "origin": [999, 200, 888, 300], + "destination": [200, 999, 300, 888], + } + ), + joint_participants=raw.joint_participants, + land_use=pl.DataFrame( + { + "zone_id": [100, 200, 300, 888, 999], + "TAZ": [10, 20, 30, 88, 99], + "EMPLOY_TOT": [1, 1, 1, 1, 1], + } + ), + skim_matrix=None, + skim_zone_map=None, + ) + + prepared = processor_prepare_data(raw, config) + + assert prepared.tours.sort("tour_id").select( + "origin", "destination", "OTAZ", "DTAZ", "tourdur" + ).to_dicts() == [ + {"origin": 100, "destination": 200, "OTAZ": 10, "DTAZ": 20, "tourdur": 3}, + {"origin": 200, "destination": 300, "OTAZ": 20, "DTAZ": 30, "tourdur": 2}, + ] + assert prepared.trips.sort("trip_id").select("origin", "destination").to_dicts() == [ + {"origin": 100, "destination": 200}, + {"origin": 200, "destination": 100}, + {"origin": 200, "destination": 300}, + {"origin": 300, "destination": 200}, + ] + + +def test_processor_prepare_data_derives_missing_joint_composition_from_ages( + tmp_path: Path, +) -> None: + config = _write_config(tmp_path) + raw = _raw_run() + tours = pl.concat([raw.tours, raw.tours, raw.tours]).with_columns( + pl.Series("tour_id", [1001, 1002, 1003]), + pl.lit("joint").alias("tour_category"), + pl.Series("composition", [None, None, "adults"], dtype=pl.Utf8), + ) + raw = ProcessorRunData( + label=raw.label, + run_dir=raw.run_dir, + skim_file=raw.skim_file, + hh=raw.hh.with_columns(pl.lit(2).alias("hhsize")), + per=pl.DataFrame( + { + "person_id": [101, 102, 103], + "household_id": [1, 1, 1], + "ptype": [1, 7, 4], + "home_zone_id": [10, 10, 10], + # Raw survey ages are category codes rather than ages in years. + "age": [9, 3, 10], + } + ), + tours=tours, + trips=raw.trips, + joint_participants=pl.DataFrame( + { + "tour_id": [1001, 1001, 1002, 1002, 1003, 1003], + "person_id": [101, 102, 101, 103, 101, 102], + } + ), + land_use=raw.land_use, + skim_matrix=None, + skim_zone_map=None, + ) + + prepared = processor_prepare_data(raw, config).tours.sort("tour_id") + + assert prepared["composition"].to_list() == ["mixed", "adults", "adults"] + assert prepared["NUMBER_HH"].to_list() == [2, 2, 2] + + def test_processor_prepare_data_materializes_shared_zone_columns_for_maz_models( tmp_path: Path, ) -> None: @@ -957,6 +1291,242 @@ def test_processor_prepare_data_derives_num_joint_tours_from_joint_participants( assert prepared["num_joint_tours"].to_list() == [2, 1] +def test_processor_prepare_data_derives_external_worker_flag(tmp_path: Path) -> None: + config = _write_config(tmp_path) + raw = _raw_run() + raw = ProcessorRunData( + label=raw.label, + run_dir=raw.run_dir, + skim_file=raw.skim_file, + hh=raw.hh, + per=pl.DataFrame( + { + "person_id": [101, 102, 103, 104], + "household_id": [1, 1, 1, 1], + "ptype": [1, 1, 1, 1], + "home_zone_id": [10, 10, 10, 10], + "is_worker": [True, True, True, False], + "work_from_home": [False, False, True, False], + "external_worker_identification": [False, True, True, True], + } + ), + tours=raw.tours, + trips=raw.trips, + joint_participants=raw.joint_participants, + land_use=raw.land_use, + skim_matrix=raw.skim_matrix, + skim_zone_map=raw.skim_zone_map, + ) + + prepared = processor_prepare_data(raw, config) + + assert prepared.per.sort("person_id")["is_external_worker"].to_list() == [ + False, + True, + False, + False, + ] + + +def test_distance_and_duration_summaries_include_zero_bins(tmp_path: Path) -> None: + config = _write_config(tmp_path) + raw = _raw_run() + raw = ProcessorRunData( + label=raw.label, + run_dir=raw.run_dir, + skim_file=raw.skim_file, + hh=raw.hh, + per=raw.per, + tours=raw.tours.with_columns( + pl.lit(0).alias("duration"), + pl.lit(5.0).alias("SKIMDIST"), + ), + trips=raw.trips.with_columns(pl.lit(5.0).alias("od_dist")), + joint_participants=raw.joint_participants, + land_use=raw.land_use, + skim_matrix=raw.skim_matrix, + skim_zone_map=raw.skim_zone_map, + ) + + prepared = processor_prepare_data(raw, config) + assert "tour_distance" not in prepared.tours.columns + duration = tour_profiles.tour_tod(prepared, config).filter( + (pl.col("tour_purpose") != "all_tour_purposes") & (pl.col("time_bin") == 0) + ) + tour_distance = tour_profiles.tour_distance(prepared, config).filter( + pl.col("distance_bin") == "0" + ) + trip_distance = trip_distributions.trip_distance(prepared, config).filter( + pl.col("distance_bin") == "0" + ) + + assert duration["duration_tour_count"].to_list() == [0.0] + assert duration["departure_tour_count"].to_list() == [0.0] + assert duration["arrival_tour_count"].to_list() == [0.0] + assert tour_distance.height == 2 + assert "all_tour_purposes" in tour_distance["tour_purpose"].to_list() + assert tour_distance["tour_count"].to_list() == [0.0, 0.0] + assert trip_distance.height == 2 + assert "all_tour_purposes" in trip_distance["tour_purpose"].to_list() + assert trip_distance["trip_count"].to_list() == [0.0, 0.0] + + +def test_tour_distance_uses_skim_distance_and_excludes_mandatory_tours( + tmp_path: Path, +) -> None: + config = _write_config( + tmp_path, + extra_lines=[ + "zones:", + " use_maz: true", + " maz_col: [zone_id]", + " taz_col: [TAZ]", + ], + ) + raw = _raw_run() + raw = ProcessorRunData( + label=raw.label, + run_dir=raw.run_dir, + skim_file=raw.skim_file, + hh=raw.hh, + per=raw.per, + tours=pl.concat([raw.tours] * 2).with_columns( + pl.Series("tour_id", [1001, 1002]), + pl.Series("tour_type", ["eatout", "work"]), + pl.Series("tour_category", ["non-mandatory", "mandatory"]), + pl.Series("tour_distance", [2.5, None], dtype=pl.Float64), + ), + trips=pl.concat([raw.trips] * 2).with_columns( + pl.Series("trip_id", [5001, 5002]), + pl.Series("tour_id", [1001, 1002]), + ), + joint_participants=raw.joint_participants, + land_use=raw.land_use.with_columns(pl.Series("TAZ", [1, 2])), + skim_matrix=pl.DataFrame([[0.0, 12.5], [12.5, 0.0]]).to_numpy(), + skim_zone_map=None, + ) + + prepared = processor_prepare_data(raw, config) + + assert prepared.tours.sort("tour_id")["tour_distance"].to_list() == [2.5, 12.5] + assert prepared.tours.sort("tour_id")["SKIMDIST"].to_list() == [12.5, 12.5] + distance = tour_profiles.tour_distance(prepared, config) + assert set(distance["tour_purpose"]) == {"eatout", "all_tour_purposes"} + assert distance.filter(pl.col("distance_bin") != "0").select( + "distance_bin", "tour_purpose", "tour_count" + ).to_dicts() == [ + { + "distance_bin": "12", + "tour_purpose": "all_tour_purposes", + "tour_count": 1.0, + }, + {"distance_bin": "12", "tour_purpose": "eatout", "tour_count": 1.0}, + ] + average = avg_non_mand_tour_distance(prepared, config).filter( + pl.col("geography_type") == "all_geographies" + ) + assert average.select("average_tour_distance", "tour_count").to_dicts() == [ + {"average_tour_distance": 12.5, "tour_count": 1.0} + ] + + prepared.tours = prepared.tours.drop("tour_distance") + cached_distance = tour_profiles.tour_distance(prepared, config).filter( + (pl.col("tour_purpose") == "eatout") & (pl.col("distance_bin") != "0") + ) + assert cached_distance.select("distance_bin", "tour_count").to_dicts() == [ + {"distance_bin": "12", "tour_count": 1.0} + ] + cached_average = avg_non_mand_tour_distance(prepared, config).filter( + pl.col("geography_type") == "all_geographies" + ) + assert cached_average.select( + "average_tour_distance", "tour_count" + ).to_dicts() == [{"average_tour_distance": 12.5, "tour_count": 1.0}] + + +def test_processor_prepare_data_uses_joint_participant_count_with_tour_fallback( + tmp_path: Path, +) -> None: + config = _write_config(tmp_path) + raw = _raw_run() + tours = pl.concat([raw.tours] * 3).with_columns( + pl.Series("tour_id", [1001, 1002, 1003]), + pl.Series("tour_category", ["joint", "joint", "joint"]), + pl.Series( + "number_of_participants", + [4, 3, None], + dtype=pl.Int64, + ), + pl.Series("SKIMDIST", [5.0, 10.0, 15.0]), + ) + trips = pl.concat([raw.trips] * 3).with_columns( + pl.Series("trip_id", [5001, 5002, 5003]), + pl.Series("tour_id", [1001, 1002, 1003]), + ) + raw = ProcessorRunData( + label=raw.label, + run_dir=raw.run_dir, + skim_file=raw.skim_file, + hh=raw.hh, + per=pl.concat( + [ + raw.per, + raw.per.with_columns( + pl.lit(102, dtype=raw.per.schema["person_id"]).alias("person_id"), + pl.lit(2, dtype=raw.per.schema["ptype"]).alias("ptype"), + ), + ] + ), + tours=tours, + trips=trips, + joint_participants=pl.DataFrame( + { + "tour_id": [1001, 1001], + "person_id": [101, 102], + } + ), + land_use=raw.land_use, + skim_matrix=raw.skim_matrix, + skim_zone_map=raw.skim_zone_map, + ) + + prepared = processor_prepare_data(raw, config) + + assert prepared.tours.sort("tour_id")["NUMBER_HH"].to_list() == [2, 3, 1] + assert prepared.trips.sort("tour_id")["num_participants"].to_list() == [2, 3, 1] + + distance = tour_profiles.tour_distance(prepared, config).filter( + pl.col("tour_purpose") != "all_tour_purposes" + ) + assert distance.filter(pl.col("distance_bin") == "0")["tour_count"].to_list() == [ + 0.0 + ] + distance = distance.filter(pl.col("distance_bin") != "0") + assert distance.select("distance_bin", "tour_count").to_dicts() == [ + {"distance_bin": "5", "tour_count": 1.0}, + {"distance_bin": "10", "tour_count": 1.0}, + {"distance_bin": "15", "tour_count": 1.0}, + ] + + tod = tour_profiles.tour_tod(prepared, config).filter( + (pl.col("tour_purpose") != "all_tour_purposes") + & (pl.col("time_bin") == 8) + ) + assert tod["departure_tour_count"].to_list() == [3.0] + + prepared.tours = prepared.tours.with_columns( + pl.lit(2).alias("AUTOSUFF"), + pl.when(pl.col("tour_id") == 1003) + .then(995) + .otherwise(pl.col("NUMBER_HH")) + .alias("NUMBER_HH"), + ) + mode = tour_profiles.tour_mode(prepared, config).filter( + pl.col("tour_purpose") != "all_tour_purposes" + ) + assert mode["tour_count_all_households"].to_list() == [6.0] + + def test_processor_prepare_data_uses_zone_id_as_maz_fallback_for_trip_skim_distance( tmp_path: Path, ) -> None: @@ -970,6 +1540,7 @@ def test_processor_prepare_data_uses_zone_id_as_maz_fallback_for_trip_skim_dista ], ) raw = _raw_run() + raw.hh = raw.hh.with_columns(pl.lit(100).alias("home_zone_id")) raw = ProcessorRunData( label=raw.label, run_dir=raw.run_dir, @@ -1231,6 +1802,7 @@ def test_processor_prepare_data_keeps_legitimate_zero_skim_values( ], ) raw = _raw_run() + raw.hh = raw.hh.with_columns(pl.lit(100).alias("home_zone_id")) raw = ProcessorRunData( label=raw.label, run_dir=raw.run_dir, @@ -2879,6 +3451,9 @@ def test_mandatory_distance_summaries_include_configured_geography_levels( "workplace_zone_id": [30, 30, None], "school_zone_id": [None, 20, 30], "is_worker": [True, True, False], + "work_from_home": [False, False, False], + "is_external_worker": [False, False, False], + "external_workplace_zone_id": [-1, -1, -1], "is_student": [False, True, True], "person_type": ["1", "7", "3"], "distance_to_work": [0.5, 20.0, None], @@ -2902,22 +3477,22 @@ def test_mandatory_distance_summaries_include_configured_geography_levels( assert work_distance.filter( (pl.col("geography_type") == "county") & (pl.col("geography_id") == "West") - & (pl.col("distance_bin") == 0) + & (pl.col("distance_bin") == ">0-<1") )["person_count"].to_list() == [1.0] assert work_distance.filter( (pl.col("geography_type") == "county") & (pl.col("geography_id") == "East") - & (pl.col("distance_bin") == 20) + & (pl.col("distance_bin") == "20") )["person_count"].to_list() == [2.0] assert school_distance.filter( (pl.col("geography_type") == "county") & (pl.col("geography_id") == "East") - & (pl.col("distance_bin") == 5) + & (pl.col("distance_bin") == "5") )["person_count"].to_list() == [2.0] assert university_distance.filter( (pl.col("geography_type") == "county") & (pl.col("geography_id") == "East") - & (pl.col("distance_bin") == 15) + & (pl.col("distance_bin") == "15") )["person_count"].to_list() == [3.0] assert mandatory_distance.filter( (pl.col("mandatory_tour_purpose") == "work") @@ -2957,6 +3532,80 @@ def test_mandatory_distance_summaries_include_configured_geography_levels( ] +def test_mandatory_distance_summary_requires_home_geography( + tmp_path: Path, +) -> None: + config = _write_config(tmp_path) + prepared = ProcessorRunData( + label="Statewide", + run_dir="C:/runs/statewide", + skim_file=None, + hh=pl.DataFrame(), + per=pl.DataFrame({"finalweight": [1.0]}), + tours=pl.DataFrame(), + trips=pl.DataFrame(), + joint_participants=pl.DataFrame(), + land_use=pl.DataFrame(), + skim_matrix=None, + skim_zone_map=None, + ) + + assert missing_summary_inputs(avg_mand_tour_distance, prepared) == { + "per": "missing required columns: home_zone_id" + } + assert avg_mand_tour_distance(prepared, config).is_empty() + + +def test_work_distance_summaries_exclude_wfh_external_and_unresolved_workers( + tmp_path: Path, +) -> None: + config = _write_config(tmp_path) + prepared = ProcessorRunData( + label="Prepared", + run_dir="C:/runs/prepared", + skim_file=None, + hh=pl.DataFrame(), + per=pl.DataFrame( + { + "person_id": [1, 2, 3, 4, 5, 6], + "home_zone_id": [1, 1, 1, 1, 1, 1], + "workplace_zone_id": [10, 10, 10, 10, 10, 10], + "is_worker": [True, True, True, True, True, True], + "work_from_home": [False, False, True, False, False, False], + "is_external_worker": [False, False, False, True, False, False], + "external_workplace_zone_id": [-1, -1, -1, -1, 2100, -1], + "distance_to_work": [0.0, 0.5, 5.0, 10.0, 20.0, None], + "finalweight": [1.0, 2.0, 4.0, 8.0, 16.0, 32.0], + } + ), + tours=pl.DataFrame(), + trips=pl.DataFrame(), + joint_participants=pl.DataFrame(), + land_use=pl.DataFrame(), + skim_matrix=None, + skim_zone_map=None, + ) + + distribution = work_tlfd(prepared, config).filter( + pl.col("geography_type") == "all_geographies" + ) + average = avg_mand_tour_distance(prepared, config).filter( + (pl.col("mandatory_tour_purpose") == "work") + & (pl.col("geography_type") == "all_geographies") + ) + + assert distribution.filter(pl.col("distance_bin").is_in(["0", ">0-<1"])).select( + "distance_bin", "person_count" + ).to_dicts() == [ + {"distance_bin": "0", "person_count": 1.0}, + {"distance_bin": ">0-<1", "person_count": 2.0}, + ] + assert distribution["person_count"].sum() == 3.0 + assert average.select("average_tour_distance", "person_count").to_dicts() == [ + {"average_tour_distance": 1.0 / 3.0, "person_count": 3.0} + ] + + def test_telecommute_summary_includes_configured_geography_levels( tmp_path: Path, ) -> None: diff --git a/tests/test_run_cli.py b/tests/test_run_cli.py index ef7b7c9..59630c9 100644 --- a/tests/test_run_cli.py +++ b/tests/test_run_cli.py @@ -169,6 +169,19 @@ def test_main_rejects_from_csvs_with_write_csvs(monkeypatch, capsys) -> None: assert captured.err == "Error: --from-csvs cannot be combined with --write-csvs.\n" +def test_parse_args_accepts_explicit_log_path(monkeypatch, tmp_path: Path) -> None: + log_path = tmp_path / "scenario.log" + monkeypatch.setattr( + sys, + "argv", + ["activitysim-viz", "--log-path", str(log_path)], + ) + + args = run.parse_args() + + assert args.log_path == str(log_path) + + def test_main_rejects_refresh_summary_cache_without_summarize( tmp_path: Path, monkeypatch, @@ -267,6 +280,67 @@ def fake_build_dashboard(runs, config, summary_runs=None): assert dashboard_calls == [{"runs": [], "summary_labels": ["Run A"]}] +def test_main_dashboard_only_loads_optional_page_summaries( + tmp_path: Path, + monkeypatch, +) -> None: + run_dir = tmp_path / "run_a" + _write_cli_config( + tmp_path, + runs=[{"dir": str(run_dir), "label": "Run A"}], + dashboard_pages=["escorted_tours"], + ) + loaded_optional_summary_ids: list[str] = [] + + def fake_load_summary_runs_from_cache( + *, required_summary_ids, optional_summary_ids, **kwargs + ): + loaded_optional_summary_ids.extend(optional_summary_ids) + all_summary_ids = [*required_summary_ids, *optional_summary_ids] + tables = { + summary_id: pl.DataFrame({"value": [1.0]}) + for summary_id in all_summary_ids + } + return [ + create_summary_run( + label="Run A", + run_key="run-a", + summaries_by_mode={"weighted": tables, "unweighted": tables}, + ) + ] + + monkeypatch.setattr( + runtime_workflows, + "load_summary_runs_from_cache", + fake_load_summary_runs_from_cache, + ) + monkeypatch.setattr( + runtime_workflows, + "run_dashboard_workflow", + lambda **kwargs: None, + ) + monkeypatch.setattr( + sys, + "argv", + [ + "activitysim-viz", + "--config", + str(tmp_path / "config.yaml"), + "--dashboard", + "--no-show", + ], + ) + + run.main() + + assert { + "student_school_escort_status_by_direction", + "student_households_by_student_count", + "households_with_school_escorting_by_student_count_and_direction", + "schoolkids_per_escorted_tour_by_student_count_and_direction", + }.issubset(loaded_optional_summary_ids) + + def test_main_uses_config_dashboard_step_for_live_dashboard_only_run( tmp_path: Path, monkeypatch, @@ -1021,6 +1095,77 @@ def test_main_refresh_summary_cache_rebuilds_and_rewrites_run_cache( assert (summary_cache_dir / "manifest.json").exists() +def test_main_refresh_summary_cache_preserves_prepared_cache( + tmp_path: Path, + monkeypatch, +) -> None: + run_dir = tmp_path / "run_a" + config = _write_cli_config( + tmp_path, + runs=[{"dir": str(run_dir), "label": "Run A"}], + ) + fingerprint = build_run_fingerprint( + label="Run A", + run_dir=config.runs[0]["dir"], + skim_file=None, + hh_weight_col=None, + person_weight_col=None, + trip_weight_col=None, + ) + prepared_entry = write_prepared_run_cache( + _fake_run_data("Run A", str(run_dir)), + config, + run_key="run-a", + run_fingerprint=fingerprint, + ) + write_summary_run_cache( + _simple_summary_run("Run A", "run-a"), + config, + run_fingerprint=fingerprint, + prepared_manifest_identity=_prepared_identity( + config=config, + run_key="run-a", + label="Run A", + run_dir=config.runs[0]["dir"], + ), + ) + summary_build_calls: list[str] = [] + monkeypatch.setattr(summary_builder, "DEFAULT_SUMMARY_IDS", ["population_totals"]) + _patch_prepare_pipeline( + monkeypatch, + read_run=lambda *args, **kwargs: (_ for _ in ()).throw( + AssertionError("summary refresh must preserve the prepared cache") + ), + prepare_data=lambda *args, **kwargs: (_ for _ in ()).throw( + AssertionError("summary refresh must not rerun preparation") + ), + ) + monkeypatch.setattr( + summary_builder, + "build_mode_summaries_with_metadata", + lambda rd, config, **kwargs: ( + summary_build_calls.append(rd.label), + _simple_summary_mode_build(rd.label, Path(rd.run_dir).name), + )[1], + ) + monkeypatch.setattr( + sys, + "argv", + [ + "activitysim-viz", + "--config", + str(tmp_path / "config.yaml"), + "--summarize", + "--refresh-summary-cache", + ], + ) + + run.main() + + assert prepared_entry.cache_dir.exists() + assert summary_build_calls == ["Run A"] + + def test_main_refresh_prepared_cache_rebuilds_prepared_tables_before_summarize( tmp_path: Path, monkeypatch, @@ -1091,6 +1236,37 @@ def test_main_refresh_prepared_cache_rebuilds_prepared_tables_before_summarize( assert (prepared_dir / "run-a" / "prepared_tables" / "manifest.json").exists() +def test_explain_cache_reports_plan_without_creating_cache_root( + tmp_path: Path, + monkeypatch, + capsys: pytest.CaptureFixture[str], +) -> None: + run_dir = tmp_path / "run_a" + config = _write_cli_config( + tmp_path, + runs=[{"dir": str(run_dir), "label": "Run A"}], + ) + cache_root = Path(config.summary_root) + monkeypatch.setattr( + sys, + "argv", + [ + "activitysim-viz", + "--config", + str(tmp_path / "config.yaml"), + "--explain-cache", + ], + ) + + run.main() + + output = capsys.readouterr().out + assert "Pipeline plan — Run A" in output + assert "prepare REBUILD" in output + assert "summarize REBUILD" in output + assert not cache_root.exists() + + def test_main_uses_cache_hit_for_one_run_and_raw_fallback_for_another( tmp_path: Path, monkeypatch, diff --git a/tests/test_run_lock.py b/tests/test_run_lock.py new file mode 100644 index 0000000..0fd2f6c --- /dev/null +++ b/tests/test_run_lock.py @@ -0,0 +1,19 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest + +from runtime.run_lock import RunLockError, acquire_run_lock, release_run_lock + + +def test_run_lock_blocks_same_output_root_until_released(tmp_path: Path) -> None: + first = acquire_run_lock(tmp_path) + try: + with pytest.raises(RunLockError, match="already using output root"): + acquire_run_lock(tmp_path) + finally: + release_run_lock(first) + + second = acquire_run_lock(tmp_path) + release_run_lock(second) diff --git a/tests/test_runtime_canonical_columns.py b/tests/test_runtime_canonical_columns.py index e7526f7..258b245 100644 --- a/tests/test_runtime_canonical_columns.py +++ b/tests/test_runtime_canonical_columns.py @@ -1657,6 +1657,22 @@ def test_escorted_tour_summaries_exclude_child_person_types(tmp_path: Path) -> N trip_distance = daily_travel_escort_distributions.adult_escorted_trip_distance_distribution_by_direction( rd, config ).sort(["direction", "distance_bin"]) + assert tour_distance.filter(pl.col("distance_bin") == "0").to_dict( + as_series=False + ) == { + "distance_bin": ["0", "0", "0"], + "direction": ["both", "inbound", "outbound"], + "tour_count": [0.0, 0.0, 0.0], + } + assert trip_distance.filter(pl.col("distance_bin") == "0").to_dict( + as_series=False + ) == { + "distance_bin": ["0", "0", "0"], + "direction": ["both", "inbound", "outbound"], + "trip_count": [0.0, 0.0, 0.0], + } + tour_distance = tour_distance.filter(pl.col("distance_bin") != "0") + trip_distance = trip_distance.filter(pl.col("distance_bin") != "0") stop_frequency = daily_travel_escort_distributions.adult_escort_trip_stop_frequency( rd, config ).sort( @@ -1712,7 +1728,7 @@ def test_escorted_tour_summaries_exclude_child_person_types(tmp_path: Path) -> N "tour_count": [3.0, 3.0, 2.0, 3.0], } assert trip_distance.to_dict(as_series=False) == { - "distance_bin": ["40+", "9", "40+", "5", "9"], + "distance_bin": ["40+", "8", "40+", "5", "8"], "direction": [ "both", "both", @@ -1856,13 +1872,29 @@ def test_adult_escort_distance_distributions_filter_to_explicit_escort_types( rd, config ).sort(["direction", "distance_bin"]) + assert tour_distance.filter(pl.col("distance_bin") == "0").to_dict( + as_series=False + ) == { + "distance_bin": ["0", "0", "0"], + "direction": ["both", "inbound", "outbound"], + "tour_count": [0.0, 0.0, 0.0], + } + assert trip_distance.filter(pl.col("distance_bin") == "0").to_dict( + as_series=False + ) == { + "distance_bin": ["0", "0", "0"], + "direction": ["both", "inbound", "outbound"], + "trip_count": [0.0, 0.0, 0.0], + } + tour_distance = tour_distance.filter(pl.col("distance_bin") != "0") + trip_distance = trip_distance.filter(pl.col("distance_bin") != "0") assert tour_distance.to_dict(as_series=False) == { - "distance_bin": ["40+", "19", "40+", "12", "40+"], + "distance_bin": ["40+", "18", "40+", "12", "40+"], "direction": ["both", "inbound", "inbound", "outbound", "outbound"], "tour_count": [3.0, 5.0, 3.0, 2.0, 3.0], } assert trip_distance.to_dict(as_series=False) == { - "distance_bin": ["40+", "9", "40+", "5", "9"], + "distance_bin": ["40+", "8", "40+", "5", "8"], "direction": [ "both", "both", @@ -2180,7 +2212,7 @@ def test_prepare_data_uses_default_fallbacks_for_purpose_timing_and_employment( assert prepared.tours["tour_purpose"].to_list() == ["eatout"] assert prepared.tours["start_hour"].to_list() == [8] assert prepared.tours["end_hour"].to_list() == [10] - assert prepared.tours["tourdur"].to_list() == [2] + assert prepared.tours["tourdur"].to_list() == [3] assert prepared.trips["tour_purpose"].to_list() == ["eatout", "eatout"] assert prepared.trips["trip_purpose"].to_list() == ["shop", "home"] assert prepared.trips["depart_hour"].to_list() == [8, 9] @@ -2414,7 +2446,7 @@ def test_prepare_data_materializes_canonical_summary_columns_from_config_overrid assert prepared.tours["tour_category"].to_list() == ["non-mandatory"] assert prepared.tours["start_hour"].to_list() == [8] assert prepared.tours["end_hour"].to_list() == [10] - assert prepared.tours["tourdur"].to_list() == [2] + assert prepared.tours["tourdur"].to_list() == [3] assert prepared.trips["trip_purpose"].to_list() == ["shop", "home"] assert prepared.trips["trip_mode"].to_list() == ["DRIVEALONE", "WALK"] assert prepared.trips["tour_purpose"].to_list() == ["eatout", "eatout"] diff --git a/tests/test_runtime_config_package.py b/tests/test_runtime_config_package.py index a547fab..75047d1 100644 --- a/tests/test_runtime_config_package.py +++ b/tests/test_runtime_config_package.py @@ -65,9 +65,7 @@ def test_repository_example_configs_match_current_schemas() -> None: assert config.pipeline.steps == ("summarize", "dashboard") assert config.pipeline.dashboard_mode == "live" assert config.skimjoin.enabled is False - assert config.skimjoin.config_path == str( - (ROOT / "example_skimjoin_config.yaml").resolve() - ) + assert config.skimjoin.config_path is None assert config.include_notes is True assert config.missing_data_display == "card" diff --git a/tests/test_runtime_workflows.py b/tests/test_runtime_workflows.py index 7d0903b..8977d84 100644 --- a/tests/test_runtime_workflows.py +++ b/tests/test_runtime_workflows.py @@ -348,8 +348,8 @@ def test_validation_scaffold_summaries_are_registered_with_empty_contracts( "count_location_volumes_validation_summary", "count_location_scatter_validation_summary", "count_location_fit_validation_summary", - "county_flows_validation_summary", - "county_flows_joja_validation_summary", + "district_commuting_flows_validation_summary", + "county_commuting_flows_validation_summary", "commercial_vehicle_validation_summary", "commercial_vehicle_vmt_validation_summary", "external_trip_validation_summary", @@ -754,6 +754,109 @@ def fake_resolve_skimjoin(config, entry): assert skimjoin_labels == ["Run A", "Run B"] +def test_refresh_skimjoin_reuses_base_prepared_cache( + tmp_path: Path, + monkeypatch, +) -> None: + run_dir = tmp_path / "run_a" + config = _write_config( + tmp_path, + runs=[{"dir": str(run_dir), "label": "Run A"}], + extra_lines=[ + "pipeline:", + " steps: [prepare, skimjoin]", + "skimjoin:", + " defaults:", + ], + ) + read_labels: list[str] = [] + prepare_labels: list[str] = [] + skimjoin_labels: list[str] = [] + + def fake_resolve_skimjoin(config, entry): + return SkimjoinSettings( + enabled=True, + config_path="mock_skimjoin.yaml", + config_digest="mock-digest", + ) + + monkeypatch.setattr( + "runtime.config.resolve_run_skimjoin_settings", + fake_resolve_skimjoin, + ) + monkeypatch.setattr( + "runtime.config.normalize_prepare.resolve_run_skimjoin_settings", + fake_resolve_skimjoin, + ) + _patch_prepare_pipeline( + monkeypatch, + read_run=lambda run_dir, config, label=None, **kwargs: ( + read_labels.append(label or Path(run_dir).name), + _fake_run_data(label or Path(run_dir).name, str(run_dir)), + )[1], + prepare_data=lambda rd, config: ( + prepare_labels.append(rd.label), + rd, + )[1], + ) + monkeypatch.setattr( + prepare_workflow, + "apply_skimjoin", + lambda rd, config: (skimjoin_labels.append(rd.label), rd)[1], + ) + + plan = _workflow_plan(config, skimjoin=True) + runtime_workflows.run_prepare_workflow( + config=config, + prepared_root=Path(config.summary_root), + run_entries=config.runs, + prefer_cache=False, + write_cache=True, + plan=plan, + ) + refreshed_plan = replace(plan, refresh_steps=("skimjoin",)) + runtime_workflows.run_prepare_workflow( + config=config, + prepared_root=Path(config.summary_root), + run_entries=config.runs, + prefer_cache=False, + write_cache=True, + plan=refreshed_plan, + ) + + assert read_labels == ["Run A"] + assert prepare_labels == ["Run A"] + assert skimjoin_labels == ["Run A", "Run A"] + assert ( + Path(config.summary_root) / "run-a" / "base_prepared_tables" / "manifest.json" + ).exists() + + +def test_raw_input_file_identity_changes_run_fingerprint(tmp_path: Path) -> None: + run_dir = tmp_path / "run_a" + run_dir.mkdir() + households = run_dir / "final_households.csv" + households.write_text("household_id\n1\n", encoding="utf-8") + config = _write_config( + tmp_path, + runs=[{"dir": str(run_dir), "label": "Run A"}], + ) + + first = prepare_workflow._run_cache_metadata( + entry=config.runs[0], + run_key="run-a", + config=config, + )["run_fingerprint"] + households.write_text("household_id\n1\n2\n", encoding="utf-8") + second = prepare_workflow._run_cache_metadata( + entry=config.runs[0], + run_key="run-a", + config=config, + )["run_fingerprint"] + + assert first["raw_file_identities"] != second["raw_file_identities"] + + def test_run_summary_workflow_does_not_build_non_default_registered_summaries( tmp_path: Path, monkeypatch, @@ -998,6 +1101,51 @@ def test_dashboard_only_loads_non_default_summary_table_map_id( ] +def test_dashboard_only_loads_available_optional_cache_summaries( + tmp_path: Path, +) -> None: + config = _write_config(tmp_path, runs=[]) + vmt_summary = pl.DataFrame( + { + "TOD": ["Daily"], + "SOV": [2.0], + "HOV2": [0.0], + "HOV3": [0.0], + "Truck": [0.0], + "Total": [2.0], + } + ) + summary_run = create_summary_run( + label="Base", + run_key="base", + summaries_by_mode={ + mode: { + "population_totals": pl.DataFrame({"person_count": [1.0]}), + "auto_vmt_validation_summary": vmt_summary, + } + for mode in config.weighting_modes + }, + ) + cache_dir = write_summary_run_cache(summary_run, config) + + loaded = runtime_workflows.load_summary_runs_from_cache( + config=config, + cache_root=Path(config.summary_root), + explicit_cache_dirs=[str(cache_dir)], + run_entries=[], + required_summary_ids=("population_totals",), + optional_summary_ids=( + "auto_vmt_validation_summary", + "link_validation_summary", + ), + ) + + assert set(loaded[0].summaries_by_mode["weighted"]) == { + "population_totals", + "auto_vmt_validation_summary", + } + + def test_dashboard_only_respects_empty_required_summary_ids_for_optional_only_page( tmp_path: Path, ) -> None: @@ -1185,6 +1333,60 @@ def _segmented_run_data(label: str, run_dir: str) -> RunData: skim_zone_map=None, ) + +def _market_segmentation_lines( + segments: list[tuple[str, str, str]], +) -> list[str]: + lines = [ + "pipeline:", + " steps: [segment, summarize]", + "segment:", + " definitions:", + " market:", + " source:", + " type: prepared_column", + " source_table: hh", + " column: market", + " segments:", + ] + for segment_id, label, value in segments: + lines.extend( + [ + f" - id: {segment_id}", + f" label: {label}", + f" values: [{value}]", + ] + ) + return lines + + +def _recording_summary_builder(calls: list[tuple[str, ...]]): + def build(rd, config, summary_ids=None): + markets = ( + tuple(sorted(rd.hh["market"].to_list())) + if "market" in rd.hh.columns + else () + ) + calls.append(markets) + requested = list(summary_ids or summary_builder.DEFAULT_SUMMARY_IDS) + tables = { + mode: { + summary_id: pl.DataFrame({"value": [float(rd.hh.height)]}) + for summary_id in requested + } + for mode in config.weighting_modes + } + metadata = { + mode: { + summary_id: {"state": "available"} + for summary_id in requested + } + for mode in config.weighting_modes + } + return tables, metadata + + return build + def test_run_prepare_workflow_rebuilds_and_writes_prepared_cache_on_cache_miss( tmp_path: Path, monkeypatch, @@ -1437,6 +1639,201 @@ def test_run_summary_workflow_with_segment_step_builds_full_and_segmented_summar ] +def test_enabling_segmentation_reuses_full_summaries_and_builds_only_segments( + tmp_path: Path, + monkeypatch, +) -> None: + run_dir = tmp_path / "run_a" + runs = [{"dir": str(run_dir), "label": "Run A"}] + config = _write_config(tmp_path, runs=runs) + read_calls: list[str] = [] + build_calls: list[tuple[str, ...]] = [] + monkeypatch.setattr(summary_builder, "DEFAULT_SUMMARY_IDS", ["population_totals"]) + monkeypatch.setattr( + summary_builder, + "build_mode_summaries_with_metadata", + _recording_summary_builder(build_calls), + ) + _patch_prepare_pipeline( + monkeypatch, + read_run=lambda run_dir, config, label=None, **kwargs: ( + read_calls.append(label or Path(run_dir).name), + _segmented_run_data(label or Path(run_dir).name, str(run_dir)), + )[1], + prepare_data=lambda rd, config: rd, + ) + + runtime_workflows.run_summary_workflow( + config=config, + cache_root=Path(config.summary_root), + run_entries=config.runs, + prefer_cache=False, + write_cache=True, + ) + build_calls.clear() + segmented_config = _write_config( + tmp_path, + runs=runs, + extra_lines=_market_segmentation_lines( + [("urban", "Urban", "Urban"), ("rural", "Rural", "Rural")] + ), + ) + + result = runtime_workflows.run_summary_workflow( + config=segmented_config, + cache_root=Path(segmented_config.summary_root), + run_entries=segmented_config.runs, + prefer_cache=True, + write_cache=True, + ) + + assert build_calls == [("Urban",), ("Rural",)] + assert read_calls == ["Run A"] + assert [(run.segmentation_type, run.segment_id) for run in result.runs] == [ + ("full", "full"), + ("market", "urban"), + ("market", "rural"), + ] + build_calls.clear() + + runtime_workflows.run_summary_workflow( + config=segmented_config, + cache_root=Path(segmented_config.summary_root), + run_entries=segmented_config.runs, + prefer_cache=False, + write_cache=True, + ) + + assert build_calls == [("Rural", "Urban"), ("Urban",), ("Rural",)] + assert read_calls == ["Run A"] + + +def test_changing_one_segment_rebuilds_only_that_segment( + tmp_path: Path, + monkeypatch, +) -> None: + run_dir = tmp_path / "run_a" + runs = [{"dir": str(run_dir), "label": "Run A"}] + config = _write_config( + tmp_path, + runs=runs, + extra_lines=_market_segmentation_lines( + [("urban", "Urban", "Urban"), ("rural", "Rural", "Rural")] + ), + ) + build_calls: list[tuple[str, ...]] = [] + monkeypatch.setattr(summary_builder, "DEFAULT_SUMMARY_IDS", ["population_totals"]) + monkeypatch.setattr( + summary_builder, + "build_mode_summaries_with_metadata", + _recording_summary_builder(build_calls), + ) + _patch_prepare_pipeline( + monkeypatch, + read_run=lambda run_dir, config, label=None, **kwargs: _segmented_run_data( + label or Path(run_dir).name, str(run_dir) + ), + prepare_data=lambda rd, config: rd, + ) + runtime_workflows.run_summary_workflow( + config=config, + cache_root=Path(config.summary_root), + run_entries=config.runs, + prefer_cache=False, + write_cache=True, + ) + build_calls.clear() + changed_config = _write_config( + tmp_path, + runs=runs, + extra_lines=_market_segmentation_lines( + [("urban", "Urban households", "Urban"), ("rural", "Rural", "Rural")] + ), + ) + + result = runtime_workflows.run_summary_workflow( + config=changed_config, + cache_root=Path(changed_config.summary_root), + run_entries=changed_config.runs, + prefer_cache=True, + write_cache=True, + ) + + assert build_calls == [("Urban",)] + assert next(run for run in result.runs if run.segment_id == "urban").segment_label == ( + "Urban households" + ) + + +def test_removing_segment_reuses_current_units_and_cleans_obsolete_cache( + tmp_path: Path, + monkeypatch, +) -> None: + run_dir = tmp_path / "run_a" + runs = [{"dir": str(run_dir), "label": "Run A"}] + config = _write_config( + tmp_path, + runs=runs, + extra_lines=_market_segmentation_lines( + [("urban", "Urban", "Urban"), ("rural", "Rural", "Rural")] + ), + ) + monkeypatch.setattr(summary_builder, "DEFAULT_SUMMARY_IDS", ["population_totals"]) + monkeypatch.setattr( + summary_builder, + "build_mode_summaries_with_metadata", + _recording_summary_builder([]), + ) + _patch_prepare_pipeline( + monkeypatch, + read_run=lambda run_dir, config, label=None, **kwargs: _segmented_run_data( + label or Path(run_dir).name, str(run_dir) + ), + prepare_data=lambda rd, config: rd, + ) + runtime_workflows.run_summary_workflow( + config=config, + cache_root=Path(config.summary_root), + run_entries=config.runs, + prefer_cache=False, + write_cache=True, + ) + changed_config = _write_config( + tmp_path, + runs=runs, + extra_lines=_market_segmentation_lines([("urban", "Urban", "Urban")]), + ) + monkeypatch.setattr( + summary_builder, + "build_mode_summaries_with_metadata", + lambda *args, **kwargs: (_ for _ in ()).throw( + AssertionError("no current analysis unit should be rebuilt") + ), + ) + + result = runtime_workflows.run_summary_workflow( + config=changed_config, + cache_root=Path(changed_config.summary_root), + run_entries=changed_config.runs, + prefer_cache=True, + write_cache=True, + ) + + assert [(run.segmentation_type, run.segment_id) for run in result.runs] == [ + ("full", "full"), + ("market", "urban"), + ] + assert not ( + Path(changed_config.summary_root) + / "run-a" + / "summary_tables" + / "weighted" + / "segments" + / "market" + / "rural" + ).exists() + + def test_run_prepare_workflow_loads_custom_prepared_tables_without_raw_prepare( tmp_path: Path, monkeypatch, @@ -2340,7 +2737,7 @@ def test_resolve_requested_steps_uses_config_pipeline_defaults(tmp_path: Path) - ] -def test_resolve_effective_plan_uses_pipeline_dashboard_mode_and_overwrite( +def test_resolve_effective_plan_uses_pipeline_dashboard_mode_and_refresh( tmp_path: Path, ) -> None: config = _write_config( @@ -2352,7 +2749,7 @@ def test_resolve_effective_plan_uses_pipeline_dashboard_mode_and_overwrite( " - summarize", " - dashboard", " dashboard_mode: export", - " overwrite: true", + " refresh: [summarize]", ], ) @@ -2375,7 +2772,7 @@ def test_resolve_effective_plan_uses_pipeline_dashboard_mode_and_overwrite( assert plan.runtime_steps == ("summarize", "dashboard") assert plan.logical_steps == ("summarize", "dashboard") assert plan.dashboard_mode == "export" - assert plan.overwrite is True + assert plan.refresh_steps == ("summarize",) def test_resolve_effective_plan_drops_dashboard_when_config_dashboard_mode_is_none( diff --git a/tests/test_segmentation_feature.py b/tests/test_segmentation_feature.py index 7af02bb..b414e36 100644 --- a/tests/test_segmentation_feature.py +++ b/tests/test_segmentation_feature.py @@ -271,7 +271,7 @@ def test_config_requires_dashboard_segmentation_type_to_exist(tmp_path: Path) -> ) -def test_summary_digest_changes_for_definition_but_not_dashboard_selection( +def test_summary_digest_excludes_segmentation_definition_and_dashboard_selection( tmp_path: Path, ) -> None: base_lines = [ @@ -324,7 +324,7 @@ def test_summary_digest_changes_for_definition_but_not_dashboard_selection( config_a.presentation_config_digest != config_b.presentation_config_digest ) - assert config_a.summary_config_digest != config_c.summary_config_digest + assert config_a.summary_config_digest == config_c.summary_config_digest def test_build_analysis_units_supports_multiple_segmentation_types(tmp_path: Path) -> None: diff --git a/tests/test_simor_scenario_runner.py b/tests/test_simor_scenario_runner.py new file mode 100644 index 0000000..c9d7d4b --- /dev/null +++ b/tests/test_simor_scenario_runner.py @@ -0,0 +1,133 @@ +from __future__ import annotations + +from argparse import Namespace +from pathlib import Path +import sys + +from scripts import run_simor_scenarios as runner + + +def _args(tmp_path: Path, *, dry_run: bool = False) -> Namespace: + return Namespace( + max_parallel=2, + log_root=tmp_path / "logs", + refresh_caches=False, + exports_only=False, + no_skimjoin=False, + dry_run=dry_run, + ) + + +def _configure_test_scenarios(monkeypatch, tmp_path: Path) -> None: + configs = [] + for name in ("metro", "lcog", "skats", "comparison"): + path = tmp_path / f"{name}.yaml" + path.write_text("name: test\n", encoding="utf-8") + configs.append((name, path)) + monkeypatch.setattr(runner, "AREA_CONFIGS", tuple(configs[:3])) + monkeypatch.setattr(runner, "COMPARISON_CONFIG", configs[3]) + + +def test_run_scenarios_waits_for_areas_before_comparison( + monkeypatch, + tmp_path: Path, +) -> None: + _configure_test_scenarios(monkeypatch, tmp_path) + batches: list[tuple[list[str], int]] = [] + + def fake_run_batch(jobs, *, max_parallel, poll_interval=0.25): + batches.append(([job.name for job in jobs], max_parallel)) + return True + + monkeypatch.setattr(runner, "_run_batch", fake_run_batch) + + assert runner.run_scenarios(_args(tmp_path)) == 0 + assert batches == [ + (["metro", "lcog", "skats"], 2), + (["comparison"], 1), + ] + + +def test_run_scenarios_skips_comparison_after_area_failure( + monkeypatch, + tmp_path: Path, +) -> None: + _configure_test_scenarios(monkeypatch, tmp_path) + batches: list[list[str]] = [] + + def fake_run_batch(jobs, *, max_parallel, poll_interval=0.25): + batches.append([job.name for job in jobs]) + return False + + monkeypatch.setattr(runner, "_run_batch", fake_run_batch) + + assert runner.run_scenarios(_args(tmp_path)) == 1 + assert batches == [["metro", "lcog", "skats"]] + + +def test_run_batch_records_console_output(tmp_path: Path) -> None: + console_log = tmp_path / "probe.console.log" + job = runner.ScenarioJob( + name="probe", + config_path=tmp_path / "probe.yaml", + runtime_log=tmp_path / "probe.runtime.log", + console_log=console_log, + command=(sys.executable, "-c", "print('runner-probe')"), + ) + + assert runner._run_batch([job], max_parallel=1, poll_interval=0.01) is True + assert console_log.read_text(encoding="utf-8").strip() == "runner-probe" + + +def test_dry_run_does_not_create_log_directory( + monkeypatch, + tmp_path: Path, +) -> None: + _configure_test_scenarios(monkeypatch, tmp_path) + + assert runner.run_scenarios(_args(tmp_path, dry_run=True)) == 0 + assert not (tmp_path / "logs").exists() + + +def test_exports_only_flags_are_passed_to_all_jobs( + monkeypatch, + tmp_path: Path, +) -> None: + _configure_test_scenarios(monkeypatch, tmp_path) + args = _args(tmp_path) + args.exports_only = True + commands: list[tuple[str, ...]] = [] + + def fake_run_batch(jobs, *, max_parallel, poll_interval=0.25): + commands.extend(job.command for job in jobs) + return True + + monkeypatch.setattr(runner, "_run_batch", fake_run_batch) + + assert runner.run_scenarios(args) == 0 + assert len(commands) == 4 + assert all(command[-2:] == ("--dashboard", "--export-html") for command in commands) + + +def test_no_skimjoin_flags_are_passed_to_all_jobs( + monkeypatch, + tmp_path: Path, +) -> None: + _configure_test_scenarios(monkeypatch, tmp_path) + args = _args(tmp_path) + args.no_skimjoin = True + commands: list[tuple[str, ...]] = [] + + def fake_run_batch(jobs, *, max_parallel, poll_interval=0.25): + commands.extend(job.command for job in jobs) + return True + + monkeypatch.setattr(runner, "_run_batch", fake_run_batch) + + assert runner.run_scenarios(args) == 0 + assert len(commands) == 4 + assert all( + command[-4:] + == ("--prepare", "--summarize", "--dashboard", "--export-html") + for command in commands + ) diff --git a/tests/test_skimjoin_integration.py b/tests/test_skimjoin_integration.py index cd504bf..e9c9c6b 100644 --- a/tests/test_skimjoin_integration.py +++ b/tests/test_skimjoin_integration.py @@ -19,8 +19,14 @@ from processor.skimjoin.annotate.tours import annotate_tours from processor.skimjoin.annotate.trips import annotate_trips from processor.skimjoin.config.validation import ConfigValidationError, load_config, validate_config +from processor.skimjoin.hypothetical_sidecars import ( + TOUR_HYPOTHETICAL_SIDECAR_SCHEMA, + TRIP_HYPOTHETICAL_SIDECAR_SCHEMA, + build_hypothetical_sidecars, +) from processor.skimjoin.inventory import inventory_skim_files from processor.skimjoin.pipeline import apply_skimjoin +from processor.skimjoin.runtime_execution import _validate_runtime_inventory from processor.skimjoin.skimstore.omx import OmxSkimStore from processor.summarize import cache_types as summary_cache_types from processor.summarize import builder as summary_builder @@ -569,6 +575,53 @@ def test_run_level_skimjoin_requires_resolvable_config_path_when_enabled( config_for_run(config, config.runs[0]) +def test_run_level_skimjoin_can_be_disabled_without_a_config_path( + tmp_path: Path, +) -> None: + config = _write_main_config( + tmp_path, + skimjoin_enabled=True, + skimjoin_config_name=None, + run_skimjoin_lines=["enabled: false"], + ) + + resolved = config_for_run(config, config.runs[0]) + + assert config.skimjoin_step_enabled() is True + assert resolved.skimjoin.enabled is False + assert resolved.skimjoin_step_enabled() is False + + +def test_run_level_skimjoin_enabled_cannot_override_the_global_pipeline( + tmp_path: Path, +) -> None: + config = _write_main_config( + tmp_path, + skimjoin_enabled=False, + skimjoin_config_name=None, + run_skimjoin_lines=["enabled: true"], + ) + + resolved = config_for_run(config, config.runs[0]) + + assert resolved.skimjoin_step_enabled() is False + + +def test_run_level_skimjoin_enabled_rejects_non_boolean_values( + tmp_path: Path, +) -> None: + with pytest.raises( + ValueError, + match=r"runs\[0\]\.skimjoin\.enabled must be true or false", + ): + _write_main_config( + tmp_path, + skimjoin_enabled=True, + skimjoin_config_name=None, + run_skimjoin_lines=["enabled: sometimes"], + ) + + def test_run_level_skimjoin_missing_config_file_raises_clear_error( tmp_path: Path, ) -> None: @@ -887,6 +940,66 @@ def test_prepare_workflow_supports_two_runs_with_different_skimjoin_config_files assert outputs["Run B"].trips["skim_time"].to_list() == [101.0] +def test_prepare_workflow_can_skip_skimjoin_for_one_run( + tmp_path: Path, +) -> None: + run_a_dir = tmp_path / "run_a" + run_b_dir = tmp_path / "run_b" + _write_prepare_run_inputs(run_a_dir) + _write_prepare_run_inputs(run_b_dir) + + skim_path = tmp_path / "shared.omx" + _write_omx_with_lookup( + skim_path, + matrix_name="SOV_TIME", + lookup_name="taz", + values=np.array([[1.0, 12.0], [22.0, 32.0]]), + ) + _write_skimjoin_config(tmp_path, skim_file=skim_path) + config_path = tmp_path / "config.yaml" + config_path.write_text( + "\n".join( + [ + 'name: "Mixed Skimjoin Test"', + "root: summary_cache", + "dashboard:", + ' title: "Mixed Skimjoin Test"', + "zones:", + " use_maz: false", + "runs:", + f' - dir: "{run_a_dir.as_posix()}"', + ' label: "Run A"', + " skimjoin:", + " enabled: false", + f' - dir: "{run_b_dir.as_posix()}"', + ' label: "Run B"', + "skimjoin:", + " defaults:", + " config_path: skimjoin.yaml", + "pipeline:", + " steps: [prepare, skimjoin]", + ] + ), + encoding="utf-8", + ) + config = Config.from_yaml(config_path) + + prepared_root = runtime_workflows.prepared_cache_root(config, create=True) + result = runtime_workflows.run_prepare_workflow( + config=config, + prepared_root=prepared_root, + run_entries=config.runs, + prefer_cache=False, + write_cache=False, + ) + + outputs = {label: prepared for label, prepared in result.runs} + assert "skim_time" not in outputs["Run A"].trips.columns + assert outputs["Run B"].trips["skim_time"].to_list() == [12.0] + assert result.fingerprints_by_key["run-a"]["skimjoin"] is None + assert result.fingerprints_by_key["run-b"]["skimjoin"]["enabled"] is True + + def test_prepare_workflow_supports_two_runs_sharing_one_skimjoin_config_with_different_skim_files( tmp_path: Path, ) -> None: @@ -1125,7 +1238,9 @@ def test_config_accepts_mixed_omx_and_csv_skim_inputs(tmp_path: Path) -> None: assert config.skimjoin.normalized_config is not None -def test_validate_config_rejects_duplicate_matrix_names_across_sources(tmp_path: Path) -> None: +def test_validate_config_rejects_ambiguous_unqualified_matrix_reference( + tmp_path: Path, +) -> None: skim_path = tmp_path / "auto.omx" csv_path = tmp_path / "auto.csv" _write_omx(skim_path, matrix_name="auto__time") @@ -1174,13 +1289,121 @@ def test_validate_config_rejects_duplicate_matrix_names_across_sources(tmp_path: }, } inventory = inventory_skim_files([skim_path, csv_path]) - trips = pl.read_parquet(tmp_path / "run" / "final_trips.parquet") - tours = pl.read_parquet(tmp_path / "run" / "final_tours.parquet") + trips = pl.read_parquet(tmp_path / "run" / "final_trips.parquet").with_columns( + pl.lit("WALK_TRANSIT").alias("trip_mode") + ) + tours = pl.read_parquet(tmp_path / "run" / "final_tours.parquet").with_columns( + pl.lit("WALK_TRANSIT").alias("tour_mode"), + pl.lit(101).alias("o_maz"), + pl.lit(102).alias("d_maz"), + ) - with pytest.raises(ConfigValidationError, match="Duplicate matrix names"): + with pytest.raises(ConfigValidationError, match="ambiguous matrix reference 'auto__time'"): validate_config(config_data, inventory, trips, tours=tours) +def test_qualified_matrix_references_select_duplicate_names_by_file( + tmp_path: Path, +) -> None: + commute_path = tmp_path / "bike_commute.omx" + noncommute_path = tmp_path / "bike_noncommute.omx" + _write_omx_with_lookup( + commute_path, + matrix_name="distance", + lookup_name="taz", + values=np.array([[1.0, 2.0], [3.0, 4.0]]), + ) + _write_omx_with_lookup( + noncommute_path, + matrix_name="distance", + lookup_name="taz", + values=np.array([[10.0, 20.0], [30.0, 40.0]]), + ) + _write_skimjoin_config( + tmp_path, + skim_files=[commute_path, noncommute_path], + include_default_mode=False, + extra_lines=[ + "modes:", + " BIKE:", + " commute_distance:", + " output: skim_bike_commute_distance", + ' matrix: "bike_commute.omx::distance"', + " noncommute_distance:", + " output: skim_bike_noncommute_distance", + ' matrix: "bike_noncommute.omx::distance"', + ], + ) + config = _write_main_config(tmp_path, skimjoin_enabled=True) + normalized = config.skimjoin.normalized_config + assert normalized is not None + inventory = inventory_skim_files(normalized.skim_files) + _validate_runtime_inventory(inventory) + + trips = pl.DataFrame( + { + "trip_id": [1], + "trip_mode": ["BIKE"], + "OTAZ": [101], + "DTAZ": [102], + } + ) + annotated, lookup_summary, missing = annotate_trips( + trips, + normalized, + inventory, + skim_store=OmxSkimStore(), + ) + + assert annotated["skim_bike_commute_distance"].to_list() == [2.0] + assert annotated["skim_bike_noncommute_distance"].to_list() == [20.0] + assert sorted(lookup_summary["matrix_name"].to_list()) == [ + "bike_commute.omx::distance", + "bike_noncommute.omx::distance", + ] + assert missing.is_empty() + + +def test_annotate_trips_rejects_ambiguous_unqualified_matrix_reference( + tmp_path: Path, +) -> None: + first_path = tmp_path / "first.omx" + second_path = tmp_path / "second.omx" + _write_omx(first_path, matrix_name="distance") + _write_omx(second_path, matrix_name="distance") + _write_skimjoin_config( + tmp_path, + skim_files=[first_path, second_path], + include_default_mode=False, + extra_lines=[ + "modes:", + " BIKE:", + " distance:", + " matrix: distance", + ], + ) + config = _write_main_config(tmp_path, skimjoin_enabled=True) + normalized = config.skimjoin.normalized_config + assert normalized is not None + inventory = inventory_skim_files(normalized.skim_files) + trips = pl.DataFrame( + { + "trip_id": [1], + "trip_mode": ["BIKE"], + "OTAZ": [101], + "DTAZ": [102], + } + ) + + with pytest.raises(ValueError, match="Ambiguous matrix reference 'distance'"): + annotate_trips( + trips, + normalized, + inventory, + skim_store=OmxSkimStore(), + ) + + def test_validate_config_allows_summed_output_overlap_but_rejects_replace_overlap( tmp_path: Path, ) -> None: @@ -1515,6 +1738,7 @@ def test_run_prepare_workflow_applies_mapping_aware_skimjoin(tmp_path: Path) -> "n_total": 1.0, "n_valid": 1.0, "mean": 2.0, + "mean_nonzero": 2.0, "std": 0.0, "min": 2.0, "max": 2.0, @@ -1530,6 +1754,7 @@ def test_run_prepare_workflow_applies_mapping_aware_skimjoin(tmp_path: Path) -> "n_total": 1.0, "n_valid": 1.0, "mean": 2.0, + "mean_nonzero": 2.0, "std": 0.0, "min": 2.0, "max": 2.0, @@ -1547,6 +1772,7 @@ def test_run_prepare_workflow_applies_mapping_aware_skimjoin(tmp_path: Path) -> "n_total": 1.0, "n_valid": 1.0, "mean": 3.0, + "mean_nonzero": 3.0, "std": 0.0, "min": 3.0, "max": 3.0, @@ -1562,6 +1788,7 @@ def test_run_prepare_workflow_applies_mapping_aware_skimjoin(tmp_path: Path) -> "n_total": 1.0, "n_valid": 1.0, "mean": 2.0, + "mean_nonzero": 2.0, "std": 0.0, "min": 2.0, "max": 2.0, @@ -1577,6 +1804,7 @@ def test_run_prepare_workflow_applies_mapping_aware_skimjoin(tmp_path: Path) -> "n_total": 1.0, "n_valid": 1.0, "mean": 3.0, + "mean_nonzero": 3.0, "std": 0.0, "min": 3.0, "max": 3.0, @@ -1592,6 +1820,7 @@ def test_run_prepare_workflow_applies_mapping_aware_skimjoin(tmp_path: Path) -> "n_total": 1.0, "n_valid": 1.0, "mean": 2.0, + "mean_nonzero": 2.0, "std": 0.0, "min": 2.0, "max": 2.0, @@ -2012,6 +2241,295 @@ def test_annotate_trips_can_return_fallback_lookup_report(tmp_path: Path) -> Non } ] + fast_annotated, fast_lookup_summary, fast_missing = annotate_trips( + trips, + normalized, + inventory, + skim_store=OmxSkimStore(), + collect_reports=False, + ) + assert fast_annotated.to_dicts() == annotated.to_dicts() + assert fast_lookup_summary.is_empty() + assert fast_missing.is_empty() + + +def test_annotate_trips_can_skip_discarded_reports( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + skim_path = tmp_path / "skims.omx" + _write_omx(skim_path) + _write_skimjoin_config(tmp_path, skim_file=skim_path) + config = _write_main_config(tmp_path, skimjoin_enabled=True) + normalized = config.skimjoin.normalized_config + assert normalized is not None + trips = pl.DataFrame( + { + "trip_id": [1], + "trip_mode": ["SOV"], + "OTAZ": [101], + "DTAZ": [102], + } + ) + inventory = inventory_skim_files(normalized.skim_files) + + def _unexpected_report(*args, **kwargs): + raise AssertionError("report builder should not run") + + monkeypatch.setattr( + "processor.skimjoin.annotate.engine._build_lookup_summary_frame", + _unexpected_report, + ) + monkeypatch.setattr( + "processor.skimjoin.annotate.engine._concat_missing_frames", + _unexpected_report, + ) + monkeypatch.setattr( + "processor.skimjoin.annotate.engine._build_fallback_lookup_report", + _unexpected_report, + ) + + annotated, lookup_summary, missing = annotate_trips( + trips, + normalized, + inventory, + skim_store=OmxSkimStore(), + collect_reports=False, + ) + + assert annotated["skim_time"].to_list() == [2.0] + assert lookup_summary.is_empty() + assert missing.is_empty() + + +def test_hypothetical_sidecars_use_mode_specific_long_lookup_values( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + skim_path = tmp_path / "skims.omx" + _write_omx(skim_path) + _write_skimjoin_config( + tmp_path, + skim_file=skim_path, + include_default_mode=False, + extra_lines=[ + "modes:", + " SOV:", + " time: SOV_TIME", + " WALK:", + " time: SOV_TIME", + ], + ) + config = _write_main_config(tmp_path, skimjoin_enabled=True) + normalized = config.skimjoin.normalized_config + assert normalized is not None + inventory = inventory_skim_files(normalized.skim_files) + calls: list[tuple[str, str, set[str]]] = [] + + def _lookup_values(frame, _normalized, _inventory, **kwargs): + mode_column = "trip_mode" if "trip_mode" in frame.columns else "tour_mode" + table_name = "trips" if mode_column == "trip_mode" else "tours" + mode = str(frame.item(0, mode_column)) + rules = kwargs["rules"] + calls.append( + ( + table_name, + mode, + {str(rule.mode) for rule in rules}, + ) + ) + outputs = sorted({str(rule.output) for rule in rules}) + return pl.DataFrame( + [ + {"_row_id": row_id, "output": output, "value": float(index + 1)} + for index, output in enumerate(outputs) + for row_id in range(frame.height) + ] + ) + + monkeypatch.setattr( + "processor.skimjoin.hypothetical_sidecars.lookup_trip_output_values", + _lookup_values, + ) + monkeypatch.setattr( + "processor.skimjoin.hypothetical_sidecars.lookup_tour_output_values", + _lookup_values, + ) + + trip_sidecar, tour_sidecar = build_hypothetical_sidecars( + trips=pl.DataFrame( + { + "trip_id": [1], + "trip_mode": ["SOV"], + "finalweight": [2.0], + } + ), + tours=pl.DataFrame( + { + "tour_id": [10], + "tour_mode": ["WALK"], + "finalweight": [3.0], + } + ), + normalized=normalized, + inventory=inventory, + ) + + assert not trip_sidecar.is_empty() + assert not tour_sidecar.is_empty() + assert calls == [ + ("trips", "SOV", {"SOV"}), + ("trips", "WALK", {"WALK"}), + ("tours", "SOV", {"SOV"}), + ("tours", "WALK", {"WALK"}), + ] + + +def test_hypothetical_sidecars_preserve_values_nulls_and_schema( + tmp_path: Path, +) -> None: + skim_path = tmp_path / "skims.omx" + _write_omx(skim_path) + _write_skimjoin_config(tmp_path, skim_file=skim_path) + config = _write_main_config(tmp_path, skimjoin_enabled=True) + normalized = config.skimjoin.normalized_config + assert normalized is not None + + trip_sidecar, tour_sidecar = build_hypothetical_sidecars( + trips=pl.DataFrame( + { + "trip_id": [1, 2], + "trip_mode": ["WALK", "SOV"], + "OTAZ": [101, 999], + "DTAZ": [102, 102], + "finalweight": [2.0, 4.0], + } + ), + tours=pl.DataFrame( + { + "tour_id": [10], + "tour_mode": ["WALK"], + "OTAZ": [101], + "DTAZ": [102], + "finalweight": [3.0], + } + ), + normalized=normalized, + inventory=inventory_skim_files(normalized.skim_files), + skim_store=OmxSkimStore(), + ) + + assert trip_sidecar.schema == TRIP_HYPOTHETICAL_SIDECAR_SCHEMA + assert trip_sidecar.sort("trip_id").to_dicts() == [ + { + "trip_id": 1, + "observed_mode": "WALK", + "hypothetical_mode": "SOV", + "component": "skim_time", + "value": 2.0, + "finalweight": 2.0, + }, + { + "trip_id": 2, + "observed_mode": "SOV", + "hypothetical_mode": "SOV", + "component": "skim_time", + "value": None, + "finalweight": 4.0, + }, + ] + assert tour_sidecar.schema == TOUR_HYPOTHETICAL_SIDECAR_SCHEMA + assert tour_sidecar.sort("component").to_dicts() == [ + { + "tour_id": 10, + "observed_mode": "WALK", + "hypothetical_mode": "SOV", + "direction": "inbound", + "component": "skim_time_inbound", + "value": 3.0, + "finalweight": 3.0, + }, + { + "tour_id": 10, + "observed_mode": "WALK", + "hypothetical_mode": "SOV", + "direction": "outbound", + "component": "skim_time_outbound", + "value": 2.0, + "finalweight": 3.0, + }, + ] + + +def test_hypothetical_sidecars_filter_csv_cache_to_required_od_pairs( + tmp_path: Path, +) -> None: + csv_path = tmp_path / "maz_maz_walk.csv" + _write_csv_od_skim( + csv_path, + rows=[ + {"OMAZ": 101, "DMAZ": 102, "DISTWALK": 0.5, "actual": 10.0}, + {"OMAZ": 101, "DMAZ": 102, "DISTWALK": 0.6, "actual": 11.0}, + {"OMAZ": 102, "DMAZ": 101, "DISTWALK": 0.75, "actual": 15.0}, + {"OMAZ": 999, "DMAZ": 999, "DISTWALK": 9.0, "actual": 99.0}, + ], + ) + _write_skimjoin_config( + tmp_path, + skim_files=[csv_path], + include_default_mode=False, + extra_lines=[ + "modes:", + " WALK:", + " distance:", + " output: skim_walk_distance", + " origin: o_maz", + " destination: d_maz", + " matrix: maz_maz_walk__DISTWALK", + ], + ) + config = _write_main_config(tmp_path, skimjoin_enabled=True) + normalized = config.skimjoin.normalized_config + assert normalized is not None + inventory = inventory_skim_files(normalized.skim_files) + skim_store = OmxSkimStore() + + trip_sidecar, tour_sidecar = build_hypothetical_sidecars( + trips=pl.DataFrame( + { + "trip_id": [1], + "trip_mode": ["SOV"], + "o_maz": [101], + "d_maz": [102], + "OTAZ": [101], + "DTAZ": [102], + "finalweight": [2.0], + } + ), + tours=pl.DataFrame( + { + "tour_id": [10], + "tour_mode": ["SOV"], + "o_maz": [101], + "d_maz": [102], + "OTAZ": [101], + "DTAZ": [102], + "finalweight": [3.0], + } + ), + normalized=normalized, + inventory=inventory, + skim_store=skim_store, + ) + + assert trip_sidecar["value"].to_list() == [0.6] + assert tour_sidecar.sort("direction")["value"].to_list() == [0.75, 0.6] + cached = next(iter(skim_store._od_csv_cache.values())) + assert cached.height == 2 + assert cached.select("__lookup_origin", "__lookup_destination").sort( + "__lookup_origin" + ).rows() == [(101.0, 102.0), (102.0, 101.0)] + def test_annotate_trips_primary_lookup_success_does_not_emit_fallback_report( tmp_path: Path, @@ -2200,6 +2718,120 @@ def test_annotate_trips_supports_csv_od_lookup(tmp_path: Path) -> None: assert missing["destination"].to_list() == [101] +def test_annotate_trips_scans_each_multi_value_csv_once( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + keyed_path = tmp_path / "maz_stop_walk.csv" + _write_csv_skim( + keyed_path, + rows=[ + { + "maz": 101, + "walk_dist_local_bus": 0.25, + "walk_dist_premium_transit": None, + "unused": 100.0, + }, + { + "maz": 101, + "walk_dist_local_bus": None, + "walk_dist_premium_transit": 0.6, + "unused": 200.0, + }, + ], + ) + od_path = tmp_path / "maz_maz_walk.csv" + _write_csv_od_skim( + od_path, + rows=[ + { + "OMAZ": 101, + "DMAZ": 102, + "DISTWALK": 0.5, + "actual": None, + "unused": 100.0, + }, + { + "OMAZ": 101, + "DMAZ": 102, + "DISTWALK": None, + "actual": 10.0, + "unused": 200.0, + }, + ], + ) + _write_skimjoin_config( + tmp_path, + skim_files=[keyed_path, od_path], + include_default_mode=False, + extra_lines=[ + "modes:", + " WALK:", + " local_stop:", + " output: skim_local_stop", + " lookup: key", + " key_column: o_maz", + " matrix: maz_stop_walk__walk_dist_local_bus", + " premium_stop:", + " output: skim_premium_stop", + " lookup: key", + " key_column: o_maz", + " matrix: maz_stop_walk__walk_dist_premium_transit", + " walk_distance:", + " output: skim_walk_distance", + " origin: o_maz", + " destination: d_maz", + " matrix: maz_maz_walk__DISTWALK", + " walk_actual:", + " output: skim_walk_actual", + " origin: o_maz", + " destination: d_maz", + " matrix: maz_maz_walk__actual", + ], + ) + config = _write_main_config(tmp_path, skimjoin_enabled=True) + normalized = config.skimjoin.normalized_config + assert normalized is not None + inventory = inventory_skim_files(normalized.skim_files) + + scan_calls: list[Path] = [] + original_scan_csv = pl.scan_csv + + def tracked_scan_csv(source, *args, **kwargs): + scan_calls.append(Path(source)) + return original_scan_csv(source, *args, **kwargs) + + monkeypatch.setattr(pl, "scan_csv", tracked_scan_csv) + skim_store = OmxSkimStore() + annotated, _, missing = annotate_trips( + pl.DataFrame( + { + "trip_id": [1], + "trip_mode": ["WALK"], + "o_maz": [101], + "d_maz": [102], + "OTAZ": [101], + "DTAZ": [102], + } + ), + normalized, + inventory, + skim_store=skim_store, + ) + + assert annotated.select( + "skim_local_stop", + "skim_premium_stop", + "skim_walk_distance", + "skim_walk_actual", + ).row(0) == (0.25, 0.6, 0.5, 10.0) + assert missing.is_empty() + assert scan_calls.count(keyed_path) == 1 + assert scan_calls.count(od_path) == 1 + assert "unused" not in next(iter(skim_store._keyed_csv_cache.values())).columns + assert "unused" not in next(iter(skim_store._od_csv_cache.values())).columns + + def test_annotate_trips_resolves_placeholders_into_multiple_matrix_groups( tmp_path: Path, ) -> None: @@ -2444,6 +3076,17 @@ def test_annotate_tours_produces_directional_outputs_and_reuses_segmentation( ] assert missing.is_empty() + fast_annotated, fast_lookup_summary, fast_missing = annotate_tours( + tours, + normalized, + inventory, + skim_store=OmxSkimStore(), + collect_reports=False, + ) + assert fast_annotated.to_dicts() == annotated.to_dicts() + assert fast_lookup_summary.is_empty() + assert fast_missing.is_empty() + def test_annotate_tours_runs_without_any_trip_inputs_or_trip_skim_columns( tmp_path: Path, @@ -3166,6 +3809,7 @@ def test_trip_skim_component_stats_follow_weighted_contract(tmp_path: Path) -> N "n_total": 6.0, "n_valid": 3.0, "mean": pytest.approx(20.0 / 3.0), + "mean_nonzero": 10.0, "std": pytest.approx(math.sqrt(200.0 / 9.0)), "min": 0.0, "max": 10.0, @@ -3181,6 +3825,7 @@ def test_trip_skim_component_stats_follow_weighted_contract(tmp_path: Path) -> N "n_total": 6.0, "n_valid": 6.0, "mean": pytest.approx(20.0 / 6.0), + "mean_nonzero": pytest.approx(20.0 / 6.0), "std": pytest.approx(math.sqrt(8.0 / 9.0)), "min": 2.0, "max": 4.0, @@ -3196,6 +3841,7 @@ def test_trip_skim_component_stats_follow_weighted_contract(tmp_path: Path) -> N "n_total": 8.0, "n_valid": 5.0, "mean": 6.0, + "mean_nonzero": 7.5, "std": pytest.approx(math.sqrt(14.0)), "min": 0.0, "max": 10.0, @@ -3259,12 +3905,14 @@ def test_tour_skim_component_summaries_follow_unweighted_mode(tmp_path: Path) -> assert drive_time["n_total"] == 3.0 assert drive_time["n_valid"] == 2.0 assert drive_time["mean"] == 5.0 + assert drive_time["mean_nonzero"] == 10.0 assert drive_time["mode"] == 0.0 assert drive_time["zero_share"] == 0.5 assert drive_time["missing_share"] == pytest.approx(1.0 / 3.0) assert all_modes_time["n_total"] == 5.0 assert all_modes_time["n_valid"] == 4.0 assert all_modes_time["mean"] == 5.0 + assert all_modes_time["mean_nonzero"] == pytest.approx(20.0 / 3.0) assert all_modes_time["mode"] == 5.0 assert all_modes_time["zero_share"] == 0.25 assert all_modes_time["missing_share"] == 0.2 diff --git a/tests/test_summary_cache.py b/tests/test_summary_cache.py index 3cc3e2d..91a14f2 100644 --- a/tests/test_summary_cache.py +++ b/tests/test_summary_cache.py @@ -29,8 +29,9 @@ from dashboard.pages.daily_travel.escorted_tours import EscortedToursPage from dashboard.pages.joint_travel import JointTravelPage from dashboard.pages.overview import OverviewPage -from dashboard.pages.skim_summaries.trip_skims import TripSkimsPage +from dashboard.pages.skim_summaries._shared import family_stats_table, skim_family_for_mode from dashboard.pages.skim_summaries.tour_skims import TourSkimsPage +from dashboard.pages.skim_summaries.trip_skims import TripSkimsPage from dashboard.pages.tour_summaries.tour_mode import ( TourModePage as TourSummariesTourModePage, ) @@ -57,9 +58,11 @@ from dashboard.pages.trip_summaries.trip_stop_time import TripStopTimePage from dashboard.pages.validation.traffic import TrafficValidationPage from dashboard.pages.validation.transit import TransitValidationPage -from dashboard.data_access import DashboardPreparedRunProvider +from dashboard.pages.validation.regional import RegionalValidationPage +from dashboard.pages.validation.vmt import VMTValidationPage +from dashboard.data_access import DashboardPreparedRunProvider, DashboardSummarySeries from dashboard.state import DashboardState -from dashboard.page_registry import page_definitions_for_group +from dashboard.page_registry import all_page_definitions, page_definitions_for_group from processor.models import RunData from processor.prepare.cache import build_prepared_manifest_identity from processor.prepare.enrichment.pipeline import prepare_data @@ -299,6 +302,20 @@ def _skim_summary_tables() -> tuple[dict[str, pl.DataFrame], dict[str, pl.DataFr 7.004, 18.5, ], + "mean_nonzero": [ + 15.126, + 3.452, + 99.111, + 1.827, + 12.233, + 1.604, + 28.06, + 8.887, + 34.221, + 28.781, + 7.004, + 18.5, + ], "std": [ 1.554, 0.882, @@ -422,6 +439,15 @@ def _skim_summary_tables() -> tuple[dict[str, pl.DataFrame], dict[str, pl.DataFr "n_total": [5.0, 5.0, 6.0, 4.0, 4.0, 3.0, 3.0], "n_valid": [5.0, 4.0, 6.0, 4.0, 4.0, 3.0, 3.0], "mean": [25.333, 21.112, 4.557, 41.221, 39.778, 9.115, 8.441], + "mean_nonzero": [ + 25.333, + 21.112, + 4.557, + 41.221, + 39.778, + 9.115, + 8.441, + ], "std": [2.111, 2.004, 0.631, 4.221, 3.992, 1.202, 1.103], "min": [21.0, 17.0, 3.4, 35.0, 34.0, 7.0, 6.7], "max": [29.0, 24.0, 5.3, 47.0, 45.0, 11.0, 10.2], @@ -2054,7 +2080,9 @@ def test_trip_stop_distance_live_page_uses_shared_summary_helpers( assert page.trip_stop_distance_range.current_range() == (0.0, 40.0) assert list(stop_ood_plot.object.data[0].x) == [0.0, 1.0, 40.0] assert list(stop_ood_plot.object.layout.xaxis.ticktext) == [ - *[str(value) for value in range(0, 40, 2)], + "0", + ">0-<1", + *[str(value) for value in range(2, 40, 2)], "40+", ] assert list(stop_ood_plot.object.layout.xaxis.range) == [0.0, 40.0] @@ -2559,7 +2587,128 @@ def test_skim_summaries_group_lists_tour_skims_before_trip_skims() -> None: ] -def test_trip_skims_page_uses_family_selector_and_two_digit_precision_summary_table( +def test_bike_transit_uses_transit_skim_family() -> None: + assert skim_family_for_mode("BIKE_TRANSIT") == "Transit Skims" + assert skim_family_for_mode("EBIKE") == "Bike Skims" + + +def test_skim_family_tables_only_show_outputs_configured_for_each_mode( + tmp_path: Path, +) -> None: + config = _write_config(tmp_path) + _attach_test_skimjoin_config(config) + normalized = config.skimjoin.normalized_config + series = DashboardSummarySeries(label="Base", summaries_by_mode={}) + for target_table, mode_column, direction_suffix in ( + ("trips", "trip_mode", ""), + ("tours", "tour_mode", "_outbound"), + ): + lookups = getattr(normalized, f"{target_table[:-1]}_lookups") + lookups.extend( + [ + SimpleNamespace(mode="BIKE", output=f"skim_bike_distance{direction_suffix}"), + SimpleNamespace(mode="BIKE", output=f"skim_bike_logsum{direction_suffix}"), + SimpleNamespace(mode="EBIKE", output=f"skim_bike_distance{direction_suffix}"), + ] + ) + stats = pl.DataFrame( + { + "component": [ + f"skim_bike_distance{direction_suffix}", + f"skim_bike_logsum{direction_suffix}", + ], + mode_column: ["EBIKE", "EBIKE"], + "n_valid": [2.0, 0.0], + } + ) + + result = family_stats_table( + config, + [("Base", series, stats)], + family="Bike Skims", + mode_column=mode_column, + target_table=target_table, + direction="outbound" if direction_suffix else None, + )[0][1] + + assert result["skim_name"].to_list() == ["Bike Distance (mi)"] + + +def test_skim_family_tables_use_all_records_pairs_without_skimjoin_config( + tmp_path: Path, +) -> None: + config = _write_config(tmp_path) + series = DashboardSummarySeries(label="Comparison", summaries_by_mode={}) + for target_table, mode_column, direction_suffix in ( + ("trips", "trip_mode", ""), + ("tours", "tour_mode", "_outbound"), + ): + auto_component = f"skim_auto_time{direction_suffix}" + transit_component = f"skim_transit_brt_ivtt{direction_suffix}" + walk_component = f"skim_walk_distance{direction_suffix}" + stats = pl.DataFrame( + { + "skim_scenario": [ + "chosen_mode", + "chosen_mode", + "chosen_mode", + "chosen_mode", + "chosen_mode", + "all_records", + "all_records", + ], + mode_column: [ + "SOV", + "SOV", + "HOV2", + "WALK_TRANSIT", + "WALK_TRANSIT", + "SOV", + "WALK_TRANSIT", + ], + "component": [ + auto_component, + walk_component, + walk_component, + transit_component, + auto_component, + auto_component, + transit_component, + ], + "n_valid": [5.0, 0.0, 0.0, 4.0, 0.0, 10.0, 10.0], + } + ) + data = [("Comparison", series, stats)] + + auto = family_stats_table( + config, + data, + family="Auto Skims", + mode_column=mode_column, + target_table=target_table, + direction="outbound" if direction_suffix else None, + )[0][1] + transit = family_stats_table( + config, + data, + family="Transit Skims", + mode_column=mode_column, + target_table=target_table, + direction="outbound" if direction_suffix else None, + )[0][1] + + expected_auto_name = ( + "Drive Time (min)" if target_table == "trips" else "Time (min)" + ) + assert auto.select(mode_column, "skim_name").rows() == [ + ("SOV", expected_auto_name) + ] + assert transit.select(mode_column, "skim_name").rows() == [ + ("WALK_TRANSIT", "BRT In-Vehicle Time (min)") + ] + + +def test_trip_skims_page_uses_family_selector_and_fixed_precision_summary_table( tmp_path: Path, ) -> None: config = _write_config(tmp_path) @@ -2589,8 +2738,13 @@ def test_trip_skims_page_uses_family_selector_and_two_digit_precision_summary_ta assert list(table.value.columns[:2]) == ["skim_name", "trip_mode"] assert set(table.value["trip_mode"].tolist()) == {"SOV", "HOV2", "HOV3"} assert "Bonus" not in table.value["skim_name"].tolist() - assert table.value["mean"].tolist() == ["3.5", "7", "15"] + assert table.value["mean"].tolist() == ["3.5", "7.0", "15.1"] + assert table.value["mean_nonzero"].tolist() == ["3.5", "7.0", "15.1"] + assert table.titles["mean_nonzero"] == "Mean Non-Zero" + assert table.value["std"].tolist() == ["0.9", "1.0", "1.6"] + assert table.value["min"].tolist() == ["1.2", "5.0", "11.0"] assert table.value["n_valid"].tolist() == ["11", "10", "9"] + assert table.value["zero_share"].tolist() == ["0.000", "0.000", "0.000"] if "Other Skims" in page.trip_family_sel.options: page.trip_family_sel.value = "Other Skims" @@ -2618,8 +2772,8 @@ def test_trip_walk_skims_use_explicit_walk_distance_and_time_labels( assert table.value["skim_name"].tolist() == [ "MAZ Actual Walk Time (min)", "MAZ Network Walk Distance (mi)", - "TAZ Skim Walk Distance (mi)", "Total Walk Access/Egress Time (min)", + "Walk Distance (mi)", ] @@ -2637,6 +2791,7 @@ def test_trip_skim_family_respects_skimjoin_ignored_modes( "n_total": [4.0, 3.0, 2.0], "n_valid": [4.0, 3.0, 2.0], "mean": [6.0, 7.0, 8.0], + "mean_nonzero": [6.0, 7.0, 8.0], "std": [1.0, 1.0, 1.0], "min": [5.0, 6.0, 7.0], "max": [7.0, 8.0, 9.0], @@ -2705,6 +2860,10 @@ def test_tour_skims_page_uses_family_and_direction_selectors_for_summary_table( assert list(table.value.columns[:2]) == ["skim_name", "tour_mode"] assert set(table.value["tour_mode"].tolist()) == {"SOV", "HOV2"} assert set(table.value["skim_name"].tolist()) == {"Cost ($)", "Time (min)"} + assert table.value["mean"].tolist() == ["4.6", "25.3"] + assert table.value["mean_nonzero"].tolist() == ["4.6", "25.3"] + assert table.value["n_valid"].tolist() == ["6", "5"] + assert table.value["min"].tolist() == ["3.4", "21.0"] assert "Outbound" == page.tour_direction_sel.value page.tour_family_sel.value = "Transit Skims" @@ -3179,7 +3338,9 @@ def counted_render(section_id=section_id, original_render=original_render): assert page.escort_distance_range.current_range() == (0.0, 40.0) assert list(distance_plot.object.layout.xaxis.range) == [0.0, 40.0] assert list(distance_plot.object.layout.xaxis.ticktext) == [ - *[str(value) for value in range(0, 40, 2)], + "0", + ">0-<1", + *[str(value) for value in range(2, 40, 2)], "40+", ] page.escort_distance_range.min_widget.value = 10.0 @@ -3273,6 +3434,17 @@ def test_escorted_tours_page_renders_core_charts_when_optional_summaries_missing assert "Adult Escort Stops Before Dropoff - Outbound" in titles assert "Adult Escort Trip Stop Frequency - Both Directions" not in titles assert all("Schoolkids Per Escorted Tour" not in title for title in titles) + card_text = [ + str(card.objects[0].object) + for card in _collect_cards(page.view) + if card.objects + ] + assert any("student_school_escort_status_by_direction" in text for text in card_text) + assert any("student_households_by_student_count" in text for text in card_text) + assert any( + "schoolkids_per_escorted_tour_by_student_count_and_direction" in text + for text in card_text + ) def test_escorted_tours_page_uses_configured_escort_labels_for_student_status( @@ -3474,6 +3646,14 @@ def test_overview_live_page_uses_shared_summary_helpers(tmp_path: Path) -> None: page.refresh(force=True) assert page.view.objects + kpi_titles = [card.title for card in _collect_cards(page._kpi_section)] + assert "Population (person-days)" in kpi_titles + assert "Households (HH-days)" in kpi_titles + percent_difference_table = _collect_tabulators(page._kpi_section)[0] + assert list(percent_difference_table.value["Metric"][:2]) == [ + "Population (person-days)", + "Households (HH-days)", + ] household_plot = next( plot for plot in _collect_plotly_panes(page._demographics_section) @@ -3663,9 +3843,9 @@ def test_tour_summaries_tour_mode_page_renders_main_chart_without_vehicle_summar ] assert chart_titles == [ "Tour Mode - All", - "Tour Mode - Zero Auto", - "Tour Mode - Fewer Vehicles Than Drivers", - "Tour Mode - At Least As Many Vehicles as Drivers", + "Tour Mode - Zero Auto Households", + "Tour Mode - Households with Fewer Vehicles Than Drivers", + "Tour Mode - Households with At Least as Many Vehicles as Drivers", ] vehicle_cards = _collect_cards(page._vehicle_section) assert len(vehicle_cards) == 3 @@ -3760,8 +3940,9 @@ def test_tour_mode_auto_sufficiency_definitions_follow_configured_basis( markdown = auto_sufficiency_definitions_markdown(config) - assert "**Fewer Vehicles Than Workers**" in markdown - assert "**At Least As Many Vehicles as Workers**" in markdown + assert "**Zero Auto Households**" in markdown + assert "**Households with Fewer Vehicles Than Workers**" in markdown + assert "**Households with At Least as Many Vehicles as Workers**" in markdown assert "household has fewer vehicles than workers." in markdown assert "household has at least as many vehicles as workers." in markdown @@ -3922,8 +4103,8 @@ def test_internal_external_tours_geo_selector_uses_union_levels_across_tables( weighted={ "internal_external_nonmandatory_tour_frequency_by_home_geography": pl.DataFrame( { - "geography_level": ["all_geographies", "maz", "district"], - "home_geography": ["all_geographies", "1", "A"], + "geography_type": ["all_geographies", "maz", "district"], + "geography_id": ["all_geographies", "1", "A"], "internal_tour_count": [5.0, 2.0, 3.0], "external_tour_count": [2.0, 1.0, 1.0], } @@ -4833,6 +5014,7 @@ def test_mandatory_location_choice_reorders_sections_and_shows_all_distance_plot assert list(distance_plots[0].object.data[0].x) == [1.0, 40.0] assert list(distance_plots[0].object.layout.xaxis.ticktext) == [ "0", + ">0-<1", "2", "4", "6", @@ -4856,6 +5038,12 @@ def test_mandatory_location_choice_reorders_sections_and_shows_all_distance_plot ] assert list(distance_plots[0].object.layout.xaxis.range) == [0.0, 40.0] assert list(distance_plots[0].object.data[0].y) == [60.0, 40.0] + assert list(distance_plots[1].object.data[0].x) == [1.0, 2.0] + assert list(distance_plots[1].object.data[0].y) == pytest.approx( + [83.33333333333334, 16.666666666666664] + ) + assert list(distance_plots[2].object.data[0].x) == [1.0, 2.0] + assert list(distance_plots[2].object.data[0].y) == [60.0, 40.0] page.mandatory_distance_range.min_widget.value = 2.0 page.mandatory_distance_range.max_widget.value = "10" page.refresh(force=True) @@ -4890,16 +5078,16 @@ def test_mandatory_location_choice_reorders_sections_and_shows_all_distance_plot "university", ] assert comparison_table["Average Mandatory Tour Distance"].tolist() == [ - "8", - "4", - "10", + "8.0", + "4.0", + "10.0", ] assert comparison_table["Base Run Average Mandatory Tour Distance"].tolist() == [ - "8", - "4", - "10", + "8.0", + "4.0", + "10.0", ] - assert comparison_table["Difference"].tolist() == ["0", "0", "0"] + assert comparison_table["Difference"].tolist() == ["0.0", "0.0", "0.0"] assert comparison_table["% Difference"].tolist() == ["0.00%", "0.00%", "0.00%"] @@ -4969,10 +5157,27 @@ def test_mandatory_location_choice_supports_configured_geography_levels_for_dist "school_district", "school_district", "school_district", + "region", + "region", + "region", + ], + "geography_id": [ + "North", + "North", + "South", + "Metro", + "Metro", + "Rural", + ], + "telecommute_frequency": [ + "never", + "often", + "never", + "never", + "often", + "never", ], - "geography_id": ["North", "North", "South"], - "telecommute_frequency": ["never", "often", "never"], - "person_count": [7.0, 5.0, 4.0], + "person_count": [7.0, 5.0, 4.0, 6.0, 2.0, 2.0], } ), "average_mandatory_tour_distance_by_purpose_and_geography": pl.DataFrame( @@ -5000,6 +5205,7 @@ def test_mandatory_location_choice_supports_configured_geography_levels_for_dist page.refresh(force=True) assert "School District" in list(page.geo_level_sel.options) + assert "Region" in list(page.geo_level_sel.options) assert list(page.geography_sel.options) == ["All Geographies"] page.geo_level_sel.value = "School District" page.refresh(force=True) @@ -5068,16 +5274,16 @@ def test_mandatory_location_choice_supports_configured_geography_levels_for_dist "% Difference", ] assert comparison_table["Average Mandatory Tour Distance"].tolist() == [ - "8", - "4", - "10", + "8.0", + "4.0", + "10.0", ] assert comparison_table["Base Run Average Mandatory Tour Distance"].tolist() == [ - "8", - "4", - "10", + "8.0", + "4.0", + "10.0", ] - assert comparison_table["Difference"].tolist() == ["0", "0", "0"] + assert comparison_table["Difference"].tolist() == ["0.0", "0.0", "0.0"] assert comparison_table["% Difference"].tolist() == ["0.00%", "0.00%", "0.00%"] page.geography_sel.value = "South" @@ -5091,6 +5297,29 @@ def test_mandatory_location_choice_supports_configured_geography_levels_for_dist assert list(south_telecommute_plot.object.data[0].x) == ["never", "often"] assert list(south_telecommute_plot.object.data[0].y) == [100.0, 0.0] + page.geo_level_sel.value = "Region" + page.refresh(force=True) + assert list(page.geography_sel.options) == ["All Regions", "Metro", "Rural"] + region_remote_work_plots = _collect_plotly_panes(page._remote_work_section) + region_telecommute_plot = next( + plot + for plot in region_remote_work_plots + if plot.object.layout.title.text == "Telecommute Rate" + ) + assert list(region_telecommute_plot.object.data[0].x) == ["never", "often"] + assert list(region_telecommute_plot.object.data[0].y) == [80.0, 20.0] + + page.geography_sel.value = "Rural" + page.refresh(force=True) + rural_remote_work_plots = _collect_plotly_panes(page._remote_work_section) + rural_telecommute_plot = next( + plot + for plot in rural_remote_work_plots + if plot.object.layout.title.text == "Telecommute Rate" + ) + assert list(rural_telecommute_plot.object.data[0].x) == ["never", "often"] + assert list(rural_telecommute_plot.object.data[0].y) == [100.0, 0.0] + def test_mandatory_location_choice_reuses_collected_data_on_selector_changes( tmp_path: Path, @@ -5225,6 +5454,7 @@ def test_traffic_validation_removes_direction_period_selectors_and_count_card( "direction": ["outbound"], "count_period": ["AM"], "screenline_id": ["A"], + "facility_type": [3], "observed_volume": [15.0], "modeled_volume": [14.0], } @@ -5244,6 +5474,8 @@ def test_traffic_validation_removes_direction_period_selectors_and_count_card( "demo_facility_type", "demo_top_period", "demo_top_n", + "screenline_period", + "screenline_facility_type", ] assert page.demo_period_sel.name == "Period" assert page.demo_top_period_sel.name == "Period" @@ -5261,12 +5493,20 @@ def test_traffic_validation_removes_direction_period_selectors_and_count_card( "demo_facility_type", ) assert sections["link_tables.volume"].selector_ids == ("demo_period",) - assert page.view.objects[-2].object == "### Screenline Flow Summaries" + assert sections["screenlines.body"].selector_ids == ( + "screenline_period", + "screenline_facility_type", + ) + assert page.view.objects[-3].object == "### Screenline Flow Summaries" + assert list(page.view.objects[-2].objects) == [ + page.screenline_period_sel, + page.screenline_facility_sel, + ] plot_titles = [ plot.object.layout.title.text for plot in _collect_plotly_panes(page._screenline_body) ] - assert plot_titles == ["Screenline Flow Comparisons"] + assert plot_titles == ["Screenline Observed vs Modeled - Day"] def test_traffic_validation_external_volume_table_compares_observed_and_modeled( @@ -5297,11 +5537,12 @@ def test_traffic_validation_external_volume_table_compares_observed_and_modeled( ), "screenline_flow_comparisons": pl.DataFrame( { - "direction": ["outbound"], - "count_period": ["AM"], - "screenline_id": ["A"], - "observed_volume": [15.0], - "modeled_volume": [14.0], + "direction": ["outbound", "outbound", "inbound"], + "count_period": ["AM", "AM", "AM"], + "screenline_id": ["A", "B", "C"], + "facility_type": [3, 3, 4], + "observed_volume": [15.0, 25.0, 35.0], + "modeled_volume": [14.0, 27.0, 30.0], } ), "link_validation_summary": pl.DataFrame( @@ -5353,6 +5594,8 @@ def test_traffic_validation_external_volume_table_compares_observed_and_modeled( assert page.demo_top_n_sel.name == "Top N by Modeled Volume" page.demo_period_sel.value = "AM" page.demo_facility_sel.value = "Principal Arterial" + page.screenline_period_sel.value = "AM" + page.screenline_facility_sel.value = "Principal Arterial" page.refresh(force=True) tables = _collect_tabulators(page._external_top_body) @@ -5401,7 +5644,8 @@ def test_traffic_validation_external_volume_table_compares_observed_and_modeled( "Total Modeled Count", "% Difference", "RMSE", - "R^2", + "RMSPE", + "R²", ] assert facility_table.to_dict("records") == [ { @@ -5411,7 +5655,8 @@ def test_traffic_validation_external_volume_table_compares_observed_and_modeled( "Total Modeled Count": "210", "% Difference": "5.00%", "RMSE": "10", - "R^2": None, + "RMSPE": "5.00%", + "R²": None, }, { "Facility Type": "Principal Arterial", @@ -5420,14 +5665,15 @@ def test_traffic_validation_external_volume_table_compares_observed_and_modeled( "Total Modeled Count": "110", "% Difference": "10.00%", "RMSE": "10", - "R^2": None, + "RMSPE": "10.00%", + "R²": None, }, ] assert facility_tables[0]._configuration == { "columns": [ {"field": "n", "sorter": "number"}, {"field": "RMSE", "sorter": "number"}, - {"field": "R^2", "sorter": "number"}, + {"field": "R²", "sorter": "number"}, ] } assert any( @@ -5459,26 +5705,53 @@ def test_traffic_validation_external_volume_table_compares_observed_and_modeled( reference_line = count_plot.object.data[-1] assert reference_line.name == "1:1 line" - assert list(reference_line.x) == [0.0, 11.0] - assert list(reference_line.y) == [0.0, 11.0] + assert list(reference_line.x) == [10.0, 11.0] + assert list(reference_line.y) == [10.0, 11.0] assert reference_line.line.color == "#BDBDBD" assert reference_line.line.dash == "dash" - assert reference_line.showlegend is False + assert reference_line.showlegend is True + assert list(count_plot.object.layout.xaxis.range) == [10.0, 11.0] + assert list(count_plot.object.layout.yaxis.range) == [10.0, 11.0] + assert count_plot.object.layout.xaxis.constrain == "domain" + assert count_plot.object.layout.yaxis.constrain == "domain" + assert count_plot.object.layout.yaxis.scaleanchor == "x" + assert count_plot.object.layout.legend.orientation == "v" + assert count_plot.object.layout.legend.x == 1.02 assert count_plot.sizing_mode == "scale_width" assert count_plot.aspect_ratio == 1.0 + assert "Observed Count (vehicles): %{x}" in count_plot.object.data[0].hovertemplate + assert "Modeled Volume (vehicles): %{y}" in count_plot.object.data[0].hovertemplate assert list(bar_plot.object.data[0].x) == [ "Minor Arterial", "Principal Arterial", ] assert bar_plot.object.layout.showlegend is True assert bar_plot.object.data[0].name == "Base" - assert plot_titles[-1] == "Screenline Flow Comparisons" + assert plot_titles[-1] == "Screenline Observed vs Modeled - AM" + screenline_plot = _collect_plotly_panes(page._screenline_body)[0] + assert screenline_plot.object.data[0].name == "Base" + assert screenline_plot.object.data[-1].name == "1:1 line" + assert screenline_plot.object.data[1].name == "Base fit" + assert len(screenline_plot.object.data[1].x) == 101 + assert "R²" in screenline_plot.object.data[1].hovertemplate + assert "y = 1.30x - 5.50" in screenline_plot.object.data[1].hovertemplate + assert not screenline_plot.object.layout.annotations + assert ( + "Observed Screenline Flow (vehicles): %{x}" + in screenline_plot.object.data[0].hovertemplate + ) + assert ( + "Modeled Screenline Flow (vehicles): %{y}" + in screenline_plot.object.data[0].hovertemplate + ) + assert screenline_plot.object.layout.yaxis.scaleanchor == "x" + assert screenline_plot.object.layout.legend.x == 1.02 assert "Traffic Count Comparisons" not in plot_titles assert "Demo Link Volume by Facility Type - Day" not in plot_titles assert "Link Volume by Facility Type - AM" in plot_titles -def test_transit_validation_technology_selector_uses_common_summary_options( +def test_transit_validation_places_each_selector_with_its_plot( tmp_path: Path, ) -> None: config = _write_config(tmp_path) @@ -5510,7 +5783,96 @@ def test_transit_validation_technology_selector_uses_common_summary_options( page = TransitValidationPage(state, config) page.refresh(force=True) - assert list(page.technology_sel.options) == ["All", "bus"] + assert list(page.technology_sel.options) == ["All", "bus", "rail"] + assert list(page.view.objects[2].objects) == [page.technology_sel] + assert list(page.view.objects[6].objects) == [page.access_mode_sel] + sections = {section.section_id: section for section in page.registered_sections} + assert sections["transit_boardings_body"].selector_ids == ("technology",) + assert sections["transit_transfer_body"].selector_ids == ("access_mode",) + + page.technology_sel.value = "rail" + page.access_mode_sel.value = "walk" + page.refresh(force=True) + transfer_plot = _collect_plotly_panes(page._transfer_body)[0] + assert transfer_plot.object.layout.title.text == "Transit Transfer Rate - walk" + + +@pytest.mark.parametrize( + ("page_type", "section_names"), + [ + ( + TrafficValidationPage, + ( + "_facility_summary_body", + "_external_volume_body", + "_link_volume_body", + "_external_top_body", + "_screenline_body", + ), + ), + ( + TransitValidationPage, + ("_boardings_body", "_transfer_body"), + ), + ( + VMTValidationPage, + ( + "_vmt_overview_body", + "_personal_vmt_body", + "_non_motorized_vmt_body", + "_external_vmt_body", + "_body", + "_bicycle_body", + ), + ), + (RegionalValidationPage, ("_body",)), + ], +) +def test_validation_visualizations_render_cards_when_data_is_unavailable( + tmp_path: Path, + page_type: type, + section_names: tuple[str, ...], +) -> None: + config = _write_config(tmp_path) + summary_run = _summary_run_with_tables(label="Base", weighted={}) + state = DashboardState( + summary_runs=[summary_run], + weighting_modes=config.weighting_modes, + ) + + page = page_type(state, config) + page.refresh(force=True) + + for section_name in section_names: + cards = _collect_cards(getattr(page, section_name)) + assert len(cards) == 1, section_name + assert cards[0].title == "Data Not Available" + + +def test_all_dashboard_sections_render_missing_data_content(tmp_path: Path) -> None: + config = _write_config(tmp_path) + summary_run = _summary_run_with_tables(label="Base", weighted={}) + + for definition in all_page_definitions(): + state = DashboardState( + summary_runs=[summary_run], + weighting_modes=config.weighting_modes, + prepared_run_provider=DashboardPreparedRunProvider.unavailable(), + ) + page = definition.page_cls(state, config) + page.refresh(force=True) + + for section in page.registered_sections: + context = f"{definition.page_id}.{section.section_id}" + assert section.container.objects, context + cards = _collect_cards(section.container) + plots = _collect_plotly_panes(section.container) + tables = _collect_tabulators(section.container) + assert cards or plots or tables, context + for plot in plots: + assert plot.object.data, context + for tabs in _collect_tabs(section.container): + assert tabs.objects, context def test_tour_distance_chart_casts_distance_bins_consistently_across_runs( @@ -5584,7 +5946,9 @@ def test_tour_distance_chart_casts_distance_bins_consistently_across_runs( assert traces["A"] == [0.0, 1.0] assert traces["B"] == [0.0] assert list(plot.object.layout.xaxis.ticktext) == [ - *[str(value) for value in range(0, 40, 2)], + "0", + ">0-<1", + *[str(value) for value in range(2, 40, 2)], "40+", ] assert list(plot.object.layout.xaxis.range) == [0.0, 40.0] @@ -5843,16 +6207,17 @@ def test_tour_time_live_page_uses_shared_summary_helpers(tmp_path: Path) -> None "tour_time_of_day_by_tour_purpose": pl.DataFrame( { "tour_purpose": [ + "work", "all_tour_purposes", "all_tour_purposes", "work", "work", "work", ], - "time_bin": [1, 2, 1, 2, 48], - "departure_tour_count": [5.0, 6.0, 3.0, 4.0, 0.0], - "arrival_tour_count": [4.0, 5.0, 2.0, 3.0, 0.0], - "duration_tour_count": [2.0, 3.0, 1.0, 2.0, 0.0], + "time_bin": [0, 1, 2, 1, 2, 48], + "departure_tour_count": [0.0, 5.0, 6.0, 3.0, 4.0, 0.0], + "arrival_tour_count": [0.0, 4.0, 5.0, 2.0, 3.0, 0.0], + "duration_tour_count": [2.0, 2.0, 3.0, 1.0, 2.0, 0.0], } ), }, @@ -5876,6 +6241,8 @@ def test_tour_time_live_page_uses_shared_summary_helpers(tmp_path: Path) -> None departure_hover = str(departure_chart.object.data[0].customdata[0]) assert "Clock Time: 03:00" in departure_hover assert "start at 03:00" not in departure_hover + duration_chart = _collect_plotly_panes(page._body)[2] + assert list(duration_chart.object.data[0].x)[0] == 0.0 def test_vehicle_ownership_type_live_page_uses_shared_summary_helpers( diff --git a/tests/test_summary_declarations.py b/tests/test_summary_declarations.py index 7fba1c0..a58d8c6 100644 --- a/tests/test_summary_declarations.py +++ b/tests/test_summary_declarations.py @@ -12,6 +12,12 @@ from processor.models import RunData from processor.summarize.catalog import build_summary_catalog from processor.summarize.contracts import SummaryResultError, summary +from scripts.generate_wiki_catalogs import ( + _validate_processor_output_reference, + _validate_summary_reference, + build_processor_output_reference, + build_summary_catalog as build_wiki_summary_catalog, +) def _run(**tables) -> RunData: @@ -100,3 +106,88 @@ def second(run, config): with pytest.raises(ValueError, match="Duplicate summary id 'duplicate'"): build_summary_catalog((module,)) + + +def test_wiki_summary_catalog_documents_build_status_and_all_fields() -> None: + _validate_summary_reference() + + catalog = build_wiki_summary_catalog() + + assert "### Population and Demographics" in catalog + assert "Related dashboard pages: Overview." in catalog + assert ( + "| Summary Table | Information and Analytical Use | " + "Required Tables/Fields | Output Fields | Summary Function |" + ) in catalog + population_row = next( + line for line in catalog.splitlines() if "`population_totals`
    " in line + ) + external_row = next( + line + for line in catalog.splitlines() + if "`auto_vmt_validation_summary`
    " in line + ) + + assert "`population_totals.csv`" in population_row + assert "`per`: Prepared person table." in population_row + assert "`finalweight` (`Float64`): Person expansion weight" in population_row + assert "`person_count` (`Float64`): weighted persons." in population_row + assert "`auto_vmt_validation_summary.csv`" in external_row + assert "Default build: **no**" in external_row + assert sum(line.startswith("| `") for line in catalog.splitlines()) == 100 + + +def test_processor_output_reference_covers_each_output_boundary() -> None: + _validate_processor_output_reference() + reference = build_processor_output_reference() + + assert "## 1. Prepared Output Tables" in reference + assert "## 2. Summary Output Tables" in reference + assert "### Population and Demographics" in reference + assert "## 3. Hypothetical All-Modes Skim Tables" in reference + assert "## 4. Externally Supplied Table Schemas" in reference + assert "**These tables are not created by the processor.**" in reference + assert "### `households` Prepared Table" in reference + assert "_Processor runtime name: `hh`._" in reference + assert "| Field | Type | Description |" in reference + assert "| Summary Table | Information and Analytical Use | Fields |" in reference + assert ( + "Every summary table is saved as a CSV file named " + "`.csv`." in reference + ) + assert "Required Tables/Fields" not in reference + assert "Summary Function" not in reference + household_row = next( + line for line in reference.splitlines() if line.startswith("| `household_id`") + ) + population_row = next( + line + for line in reference.splitlines() + if line.startswith("| `population_totals` |") + ) + sidecar_row = next( + line + for line in reference.splitlines() + if line.startswith("| `trip_hypothetical_skims` |") + ) + external_row = next( + line + for line in reference.splitlines() + if line.startswith("| `auto_vmt_validation_summary` |") + ) + assert "| `household_id` | `Int64` |" in household_row + assert "`population_totals.csv`" not in population_row + assert "`person_count` (`Float64`): weighted persons." in population_row + assert "`trip_id` (`Int64`)" in sidecar_row + assert "`auto_vmt_validation_summary.csv`" not in external_row + assert "skim_lookup_summary.csv" not in reference + + summary_section, remainder = reference.split( + "## 2. Summary Output Tables", maxsplit=1 + )[1].split("## 3. Hypothetical All-Modes Skim Tables", maxsplit=1) + sidecar_section, external_section = remainder.split( + "## 4. Externally Supplied Table Schemas", maxsplit=1 + ) + assert sum(line.startswith("| `") for line in summary_section.splitlines()) == 87 + assert sum(line.startswith("| `") for line in sidecar_section.splitlines()) == 2 + assert sum(line.startswith("| `") for line in external_section.splitlines()) == 13 diff --git a/tests/test_summary_helpers.py b/tests/test_summary_helpers.py index 169c96e..0c475b7 100644 --- a/tests/test_summary_helpers.py +++ b/tests/test_summary_helpers.py @@ -12,6 +12,8 @@ ALL_TOUR_PURPOSES, _all_person_types_rollup, _all_purpose_rollup, + _distance_bin_expr, + _distance_bin_sort_expr, ) @@ -77,3 +79,36 @@ def test_all_person_types_rollup_preserves_concat_compatible_column_order() -> N "mandatory_tour_frequency": [1, 2], "person_count": [5.0, 4.0], } + + +def test_distance_bins_separate_exact_zero_from_positive_submile_values() -> None: + result = ( + pl.DataFrame( + {"distance": [None, -1.0, 0.0, 0.25, 0.99, 1.0, 1.99, 39.99, 40.0]} + ) + .with_columns(_distance_bin_expr("distance")) + .with_columns(_distance_bin_sort_expr().alias("_sort_distance")) + ) + + assert result["distance_bin"].to_list() == [ + None, + None, + "0", + ">0-<1", + ">0-<1", + "1", + "1", + "39", + "40+", + ] + assert result["_sort_distance"].to_list() == [ + float("inf"), + float("inf"), + 0.0, + 0.5, + 0.5, + 1.0, + 1.0, + 39.0, + 40.0, + ] diff --git a/tests/test_summary_regressions.py b/tests/test_summary_regressions.py index 9ff3f7b..15c487f 100644 --- a/tests/test_summary_regressions.py +++ b/tests/test_summary_regressions.py @@ -5,14 +5,27 @@ import polars as pl from processor.models import RunData +from processor.summarize.cache_types import strip_weights +from processor.summarize.summaries.daily_travel_activity import ( + dap_summary, + indiv_nm_summary, + mandatory_tour_freq, + tour_rate_per_person, + trip_rate_per_person, +) from processor.summarize.summaries.demographics import hh_size, person_type from processor.summarize.summaries.long_term_geography import free_parking +from processor.summarize.summaries.validation import screenline_flow_comparisons def _run( *, hh: pl.DataFrame | None = None, per: pl.DataFrame | None = None, + day: pl.DataFrame | None = None, + tours: pl.DataFrame | None = None, + trips: pl.DataFrame | None = None, + joint_participants: pl.DataFrame | None = None, ) -> RunData: return RunData( label="Test", @@ -20,15 +33,449 @@ def _run( skim_file=None, hh=hh if hh is not None else pl.DataFrame(), per=per if per is not None else pl.DataFrame(), - tours=pl.DataFrame(), - trips=pl.DataFrame(), - joint_participants=pl.DataFrame(), + day=day if day is not None else pl.DataFrame(), + tours=tours if tours is not None else pl.DataFrame(), + trips=trips if trips is not None else pl.DataFrame(), + joint_participants=( + joint_participants if joint_participants is not None else pl.DataFrame() + ), land_use=pl.DataFrame(), skim_matrix=None, skim_zone_map=None, ) +def test_dap_summary_uses_person_day_activity_patterns_when_available() -> None: + result = dap_summary( + _run( + per=pl.DataFrame( + { + "person_id": [1, 2], + "person_type": ["1", "2"], + "finalweight": [10.0, 20.0], + } + ), + day=pl.DataFrame( + { + "person_id": [1, 1, 2], + "cdap_activity": ["M", "N", "H"], + "finalweight": [1.0, 2.0, 3.0], + } + ), + ), + _config(), + ) + + assert result.to_dicts() == [ + {"person_type": "1", "daily_activity_pattern": "M", "person_count": 1.0}, + {"person_type": "1", "daily_activity_pattern": "N", "person_count": 2.0}, + {"person_type": "2", "daily_activity_pattern": "H", "person_count": 3.0}, + { + "person_type": "all_person_types", + "daily_activity_pattern": "H", + "person_count": 3.0, + }, + { + "person_type": "all_person_types", + "daily_activity_pattern": "M", + "person_count": 1.0, + }, + { + "person_type": "all_person_types", + "daily_activity_pattern": "N", + "person_count": 2.0, + }, + ] + + +def test_dap_summary_falls_back_to_person_activity_patterns() -> None: + result = dap_summary( + _run( + per=pl.DataFrame( + { + "person_type": ["1", "2"], + "cdap_activity": ["M", "N"], + "finalweight": [1.0, 2.0], + } + ) + ), + _config(), + ) + + assert result.filter(pl.col("person_type") != "all_person_types").to_dicts() == [ + {"person_type": "1", "daily_activity_pattern": "M", "person_count": 1.0}, + {"person_type": "2", "daily_activity_pattern": "N", "person_count": 2.0}, + ] + + +def test_dap_summary_prefers_person_activity_patterns_when_both_exist() -> None: + result = dap_summary( + _run( + per=pl.DataFrame( + { + "person_id": [1], + "person_type": ["1"], + "cdap_activity": ["M"], + "finalweight": [2.0], + } + ), + day=pl.DataFrame( + { + "person_id": [1], + "cdap_activity": ["N"], + } + ), + ), + _config(), + ) + + assert result.filter(pl.col("person_type") != "all_person_types").to_dicts() == [ + {"person_type": "1", "daily_activity_pattern": "M", "person_count": 2.0} + ] + + +def test_dap_summary_falls_back_rowwise_and_uses_each_day_weight() -> None: + result = dap_summary( + _run( + per=pl.DataFrame( + { + "person_id": [1, 2], + "person_type": ["1", "2"], + "cdap_activity": ["M", None], + "finalweight": [100.0, 200.0], + } + ), + day=pl.DataFrame( + { + "person_id": [1, 2, 2], + "day_id": [11, 21, 22], + "cdap_activity": ["N", "H", "N"], + "finalweight": [10.0, 3.0, 4.0], + } + ), + ), + _config(), + ) + + assert result.filter(pl.col("person_type") != "all_person_types").to_dicts() == [ + {"person_type": "1", "daily_activity_pattern": "M", "person_count": 100.0}, + {"person_type": "2", "daily_activity_pattern": "H", "person_count": 3.0}, + {"person_type": "2", "daily_activity_pattern": "N", "person_count": 4.0}, + ] + + +def test_mandatory_tour_frequency_falls_back_to_person_day_tour_counts() -> None: + result = mandatory_tour_freq( + _run( + per=pl.DataFrame( + { + "person_id": [1, 2], + "person_type": ["1", "2"], + "finalweight": [0.5, 2.0], + } + ), + day=pl.DataFrame( + { + "person_id": [1, 1, 2], + "day_id": [11, 12, 21], + } + ), + tours=pl.DataFrame( + { + "person_id": [1, 1, 2, 2], + "day_id": [11, 12, 21, 21], + "tour_purpose": ["work", "school", "work", "work"], + "tour_category": ["mandatory"] * 4, + } + ), + ), + _config(), + ) + + assert result.filter(pl.col("person_type") != "all_person_types").to_dicts() == [ + {"person_type": "1", "mandatory_tour_frequency": 1, "person_count": 0.5}, + {"person_type": "1", "mandatory_tour_frequency": 3, "person_count": 0.5}, + {"person_type": "2", "mandatory_tour_frequency": 2, "person_count": 2.0}, + ] + + +def test_mandatory_frequency_falls_back_rowwise_and_uses_day_weight() -> None: + result = mandatory_tour_freq( + _run( + per=pl.DataFrame( + { + "person_id": [1, 2], + "person_type": ["1", "2"], + "imf_choice": [1, None], + "finalweight": [100.0, 200.0], + } + ), + day=pl.DataFrame( + { + "person_id": [1, 2, 2], + "day_id": [11, 21, 22], + "finalweight": [10.0, 3.0, 4.0], + } + ), + tours=pl.DataFrame( + { + "person_id": [1, 2, 2, 2], + "day_id": [11, 21, 21, 22], + "tour_purpose": ["school", "work", "work", "school"], + "tour_category": ["mandatory"] * 4, + } + ), + ), + _config(), + ) + + assert result.filter(pl.col("person_type") != "all_person_types").to_dicts() == [ + {"person_type": "1", "mandatory_tour_frequency": 1, "person_count": 100.0}, + {"person_type": "2", "mandatory_tour_frequency": 2, "person_count": 3.0}, + {"person_type": "2", "mandatory_tour_frequency": 3, "person_count": 4.0}, + ] + + +def test_nonmandatory_frequency_uses_person_days_and_joint_participant_days() -> None: + result = indiv_nm_summary( + _run( + per=pl.DataFrame( + { + "person_id": [1, 2], + "person_type": ["1", "2"], + "finalweight": [100.0, 200.0], + } + ), + day=pl.DataFrame( + { + "person_id": [1, 1, 2], + "day_id": [11, 12, 21], + "finalweight": [2.0, 3.0, 5.0], + } + ), + tours=pl.DataFrame( + { + "tour_id": [101, 102, 201, 202], + "person_id": [1, 1, 1, 1], + "day_id": [11, 11, 12, 12], + "tour_category": [ + "non_mandatory", + "non_mandatory", + "joint", + "joint", + ], + } + ), + joint_participants=pl.DataFrame( + { + "tour_id": [201, 202], + "person_id": [1, 1], + } + ), + ), + _config(), + ) + + assert result.filter(pl.col("person_type") != "all_person_types").to_dicts() == [ + { + "person_type": "1", + "nonmandatory_tour_frequency": "2", + "person_count": 5.0, + }, + { + "person_type": "2", + "nonmandatory_tour_frequency": "0", + "person_count": 5.0, + }, + ] + + +def test_nonmandatory_frequency_uses_person_fallback_when_joint_day_is_unsafe() -> None: + result = indiv_nm_summary( + _run( + per=pl.DataFrame( + { + "person_id": [1], + "person_type": ["1"], + "finalweight": [7.0], + } + ), + day=pl.DataFrame( + { + "person_id": [1, 1], + "day_id": [11, 12], + "finalweight": [2.0, 3.0], + } + ), + tours=pl.DataFrame( + { + "person_id": [1, 1], + "day_id": [11, 12], + "tour_category": ["non_mandatory", "non_mandatory"], + } + ), + joint_participants=pl.DataFrame({"person_id": [1]}), + ), + _config(), + ) + + assert result.filter(pl.col("person_type") != "all_person_types").to_dicts() == [ + { + "person_type": "1", + "nonmandatory_tour_frequency": "3+", + "person_count": 7.0, + } + ] + + +def test_daily_rates_sum_every_activity_weight_and_retain_person_day_rows() -> None: + run = _run( + per=pl.DataFrame( + { + "person_id": [1, 2], + "person_type": ["1", "2"], + "finalweight": [2.0, 3.0], + } + ), + day=pl.DataFrame( + { + "person_id": [1, 1, 2], + "day_id": [11, 12, 21], + } + ), + tours=pl.DataFrame( + { + "person_id": [1, 1, 2], + "tour_purpose": ["work", "work", "work"], + "finalweight": [2.0, 4.0, 3.0], + } + ), + trips=pl.DataFrame( + { + "person_id": [1, 1, 1, 2], + "trip_purpose": ["work", "work", "work", "work"], + "finalweight": [1.0, 2.0, 3.0, 6.0], + } + ), + ) + + tour_result = tour_rate_per_person(run, _config()) + trip_result = trip_rate_per_person(run, _config()) + unweighted_tour_result = tour_rate_per_person(strip_weights(run), _config()) + unweighted_trip_result = trip_rate_per_person(strip_weights(run), _config()) + + assert tour_result.filter( + pl.col("person_type") == "all_person_types" + ).to_dicts() == [ + {"person_type": "all_person_types", "tour_purpose": "work", "tour_rate": 9 / 7} + ] + assert trip_result.filter( + pl.col("person_type") == "all_person_types" + ).to_dicts() == [ + {"person_type": "all_person_types", "trip_purpose": "work", "trip_rate": 12 / 7} + ] + assert unweighted_tour_result.filter( + pl.col("person_type") == "all_person_types" + ).to_dicts() == [ + {"person_type": "all_person_types", "tour_purpose": "work", "tour_rate": 1.0} + ] + assert unweighted_trip_result.filter( + pl.col("person_type") == "all_person_types" + ).to_dicts() == [ + { + "person_type": "all_person_types", + "trip_purpose": "work", + "trip_rate": 4 / 3, + } + ] + + +def test_daily_rates_use_multiday_weights_without_deduplicating_days() -> None: + run = _run( + per=pl.DataFrame( + { + "person_id": [1], + "person_type": ["1"], + "finalweight": [100.0], + } + ), + day=pl.DataFrame( + { + "person_id": [1, 1], + "day_id": [11, 12], + "finalweight": [2.0, 3.0], + } + ), + tours=pl.DataFrame( + { + "person_id": [1, 1], + "tour_purpose": ["work", "work"], + "finalweight": [2.0, 3.0], + } + ), + trips=pl.DataFrame( + { + "person_id": [1, 1, 1, 1], + "trip_purpose": ["work", "work", "work", "work"], + "finalweight": [2.0, 2.0, 3.0, 3.0], + } + ), + ) + + tour_result = tour_rate_per_person(run, _config()).filter( + pl.col("person_type") == "all_person_types" + ) + trip_result = trip_rate_per_person(run, _config()).filter( + pl.col("person_type") == "all_person_types" + ) + + assert tour_result["tour_rate"].to_list() == [1.0] + assert trip_result["trip_rate"].to_list() == [2.0] + + +def test_daily_rates_use_day_rows_even_when_person_ids_are_unique() -> None: + run = _run( + per=pl.DataFrame( + { + "person_id": [1, 2], + "person_type": ["1", "1"], + "finalweight": [0.5, 2.0], + } + ), + day=pl.DataFrame( + { + "person_id": [1], + "day_id": [11], + "finalweight": [0.5], + } + ), + tours=pl.DataFrame( + { + "person_id": [1], + "tour_purpose": ["work"], + "finalweight": [0.5], + } + ), + trips=pl.DataFrame( + { + "person_id": [1, 1], + "trip_purpose": ["work", "home"], + "finalweight": [0.5, 0.5], + } + ), + ) + + tour_result = tour_rate_per_person(run, _config()).filter( + pl.col("person_type") == "all_person_types" + ) + trip_result = trip_rate_per_person(run, _config()).filter( + pl.col("person_type") == "all_person_types" + ) + + assert tour_result["tour_rate"].to_list() == [1.0] + assert trip_result["trip_rate"].to_list() == [1.0, 1.0] + + def _config(): return SimpleNamespace( use_maz=False, @@ -39,6 +486,38 @@ def _config(): ) +def test_screenline_comparison_uses_available_facility_type() -> None: + run = _run() + run.observed_screenline_flows = pl.DataFrame( + { + "screenline_id": ["A"], + "direction": ["NB"], + "count_period": ["AM"], + "volume": [100.0], + } + ) + run.visum_screenline_flows = pl.DataFrame( + { + "screenline_id": ["A"], + "direction": ["NB"], + "count_period": ["AM"], + "facility_type": [3], + "volume": [110.0], + } + ) + + assert screenline_flow_comparisons(run, None).to_dicts() == [ + { + "screenline_id": "A", + "direction": "NB", + "count_period": "AM", + "facility_type": "3", + "observed_volume": 100.0, + "modeled_volume": 110.0, + } + ] + + def test_household_size_summary_normalizes_integer_width_to_contract() -> None: result = hh_size( _run( diff --git a/tests/test_validation_derived.py b/tests/test_validation_derived.py index ace89f0..b67f7c5 100644 --- a/tests/test_validation_derived.py +++ b/tests/test_validation_derived.py @@ -18,6 +18,7 @@ ) from processor.summarize.external import load_summary_table_map, merge_summary_table_map_run from runtime.config import Config +from scripts import generate_validation_demo_fixtures as fixture_generator def _write_config(tmp_path: Path) -> Config: @@ -204,3 +205,85 @@ def test_summary_table_map_run_builds_and_caches_count_location_validation_deriv loaded.summary_metadata_by_mode["weighted"][COUNT_LOCATION_FIT_ID]["state"] == "available" ) + + +def test_validation_demo_fixture_generator_writes_distinct_run_tables( + tmp_path: Path, + monkeypatch, +) -> None: + source_dir = tmp_path / "source" + output_dir = source_dir / "estimated_fixtures" + source_dir.mkdir() + pl.DataFrame( + { + "id": [1], + "FACTYPE": [3], + "am_vol": [100.0], + "md_vol": [200.0], + "pm_vol": [150.0], + "day_vol": [500.0], + } + ).write_csv(source_dir / "countLocCounts.csv") + pl.DataFrame( + { + "id": [1], + "From_Node": [10], + "To_Node": [20], + "FACTYPE": [3], + "am_vol": [100.0], + "md_vol": [200.0], + "pm_vol": [150.0], + "day_vol": [500.0], + } + ).write_csv(source_dir / "allLinkSummary.csv") + for filename in ("cvm_summary.csv", "cvm_vmt_summary.csv"): + pl.DataFrame( + {"tod": ["AM"], "car": [10.0], "mu": [3.0], "su": [5.0], "Total": [18.0]} + ).write_csv(source_dir / filename) + external = { + "tod": ["AM"], + "hbcoll": [1.0], + "hbo": [2.0], + "hbr": [3.0], + "hbs": [4.0], + "hbsch": [5.0], + "hbw": [6.0], + "nhbnw": [7.0], + "nhbw": [8.0], + "truck": [9.0], + "Total": [45.0], + } + for filename in ("ext_summary.csv", "ext_vmt_summary.csv"): + pl.DataFrame(external).write_csv(source_dir / filename) + flow_matrix = pl.DataFrame( + {"": ["A", "Total"], "A": [10.0, 10.0], "Total": [10.0, 10.0]} + ) + flow_matrix.write_csv(source_dir / "countyFlows.csv") + flow_matrix.write_csv(source_dir / "countyFlows_JoJa.csv") + + monkeypatch.setattr(fixture_generator, "SOURCE_DIR", source_dir) + monkeypatch.setattr(fixture_generator, "OUTPUT_DIR", output_dir) + fixture_generator.generate() + + day_values = [] + for run in fixture_generator.RUNS: + run_dir = output_dir / run + assert len(list(run_dir.glob("*.csv"))) == 11 + day_values.append( + pl.read_csv(run_dir / "count_location_volumes_validation_summary.csv")[ + "day_vol" + ][0] + ) + assert not pl.read_csv(run_dir / "screenline_flow_comparisons.csv").is_empty() + assert set(pl.read_csv(run_dir / "commuting_flows.csv")[ + "origin_geography_type" + ]) == {"district", "county"} + commercial = pl.read_csv( + run_dir / "commercial_vehicle_validation_summary.csv" + ) + assert commercial["Total"][0] == commercial.select( + pl.sum_horizontal("car", "mu", "su") + ).item() + + assert len(set(day_values)) == 4 + assert len(list((output_dir / "observed").glob("*.csv"))) == 3 diff --git a/tests/test_validation_vmt_summary.py b/tests/test_validation_vmt_summary.py index 9cdafb3..ed73782 100644 --- a/tests/test_validation_vmt_summary.py +++ b/tests/test_validation_vmt_summary.py @@ -4,6 +4,7 @@ import sys import polars as pl +import pytest sys.path.insert(0, str(Path(__file__).resolve().parents[1])) @@ -290,6 +291,174 @@ def test_auto_vmt_segment_summary_uses_daily_and_dimension_fallbacks() -> None: ] +def test_derived_daily_vmt_uses_period_rows_and_preserves_daily_only_groups() -> None: + period_rows = pl.DataFrame( + { + "geography_type": ["home_taz"] * 4, + "geography_id": ["101"] * 4, + "income_segment": ["2"] * 4, + "household_size": ["3"] * 4, + "time_period": ["AM", "PM", "Daily", "Daily"], + "mode": ["SOV", "SOV", "SOV", "HOV2"], + "auto_vmt": [10.0, 20.0, 999.0, 7.0], + "trip_count": [1.0, 2.0, 99.0, 4.0], + "distance_source": ["od_dist"] * 4, + "time_period_source": ["trip_period"] * 4, + } + ) + + result = validation._with_derived_daily_vmt_rows(period_rows).sort( + ["mode", "time_period"] + ) + + assert result.select( + "mode", "time_period", "auto_vmt", "trip_count" + ).to_dicts() == [ + { + "mode": "HOV2", + "time_period": "Daily", + "auto_vmt": 7.0, + "trip_count": 4.0, + }, + {"mode": "SOV", "time_period": "AM", "auto_vmt": 10.0, "trip_count": 1.0}, + { + "mode": "SOV", + "time_period": "Daily", + "auto_vmt": 30.0, + "trip_count": 3.0, + }, + {"mode": "SOV", "time_period": "PM", "auto_vmt": 20.0, "trip_count": 2.0}, + ] + + +def test_auto_vmt_summaries_share_legacy_occupancy_and_daily_total() -> None: + rd = _run_data( + trips=pl.DataFrame( + { + "trip_mode": [ + "SOV", + "HOV2", + "HOV3", + "KNR_TRANSIT", + "PNR_TRANSIT", + "HOV3", + "HOV2", + "TNC_SHARED", + "AUTO_PASSENGER", + "WALK", + ], + "skim_auto_distance": [ + 10.0, + 12.0, + 9.99, + 8.0, + 5.0, + 7.0, + 6.0, + 6.0, + 4.0, + None, + ], + "od_dist": [99.0] * 10, + "trip_period": ["AM"] * 5 + ["PM"] * 5, + "tour_category": [ + "mandatory", + "mandatory", + "mandatory", + "mandatory", + "mandatory", + "joint", + "mandatory", + "non_mandatory", + "non_mandatory", + "non_mandatory", + ], + "escort_event_role": [ + None, + None, + None, + None, + None, + None, + "dropoff", + None, + None, + None, + ], + "num_participants": [995.0] * 8 + [0.0, 1.0], + "finalweight": [1.0] * 10, + } + ) + ) + + total = validation.auto_vmt_totals(rd, None)["auto_vmt"][0] + detailed = validation.auto_vmt_by_home_geography_income_hhsize_time_period( + rd, None + ) + detailed_daily_total = ( + detailed.filter( + (pl.col("geography_type") == "all_geographies") + & (pl.col("time_period") == "Daily") + )["auto_vmt"].sum() + ) + + assert total == pytest.approx(51.0) + assert detailed_daily_total == pytest.approx(total) + + +def test_auto_vmt_invalid_participant_codes_fall_back_to_one() -> None: + rd = _run_data( + trips=pl.DataFrame( + { + "trip_mode": ["AUTO_PASSENGER"] * 5, + "od_dist": [5.0] * 5, + "num_participants": [995.0, None, 0.0, -2.0, 4.0], + "finalweight": [1.0] * 5, + } + ) + ) + + result = validation.auto_vmt_totals(rd, None) + + assert result["auto_vmt"][0] == pytest.approx(21.25) + + +def test_auto_vmt_od_distance_does_not_count_full_drive_transit_trip() -> None: + rd = _run_data( + trips=pl.DataFrame( + { + "trip_mode": ["SOV", "KNR_TRANSIT", "PNR_TRANSIT"], + "od_dist": [10.0, 50.0, 60.0], + "num_participants": [1.0, 2.0, 1.0], + "finalweight": [1.0, 1.0, 1.0], + } + ) + ) + + result = validation.auto_vmt_totals(rd, None) + + assert result["auto_vmt"][0] == pytest.approx(10.0) + + +def test_auto_vmt_preserves_person_level_joint_rows_and_applies_mode_occupancy() -> None: + rd = _run_data( + trips=pl.DataFrame( + { + "trip_mode": ["HOV2", "HOV2", "HOV3", "HOV3", "HOV3"], + "tour_category": ["joint"] * 5, + "joint_trip_id": [10, 10, 20, 20, 20], + "od_dist": [10.0, 10.0, 9.99, 9.99, 9.99], + "num_participants": [1.0] * 5, + "finalweight": [1.0] * 5, + } + ) + ) + + result = validation.auto_vmt_totals(rd, None) + + assert result["auto_vmt"][0] == pytest.approx(19.0) + + def test_non_motorized_vmt_summary_prefers_skimjoin_distances() -> None: rd = _run_data( trips=pl.DataFrame( diff --git a/wiki/00-home.md b/wiki/00-home.md index ddf390c..feeb5dc 100644 --- a/wiki/00-home.md +++ b/wiki/00-home.md @@ -1,47 +1,63 @@ # ActivitySim Visualizer Wiki -This wiki is the main documentation home for the ActivitySim Visualizer. It is -written for two audiences: +This wiki is the main documentation for ActivitySim Visualizer. It covers two +common tasks: -- users who need to run the visualizer on ActivitySim outputs -- developers who need to extend the processor, summaries, skimjoin, or dashboard +- running the visualizer with ActivitySim output +- extending the processor, summaries, skimjoin, or dashboard -The short mental model: +The main data flow is: ```text ActivitySim outputs -> Output Processor - -> prepared tables and summary caches + -> canonical prepared tables + -> optional skimjoin enrichment + -> summary caches -> Output Visualizer -> live dashboard or standalone HTML export ``` -For the subsystem boundaries and complete repository map, see +For subsystem boundaries and a complete repository map, see [01 - Architecture](01-architecture.md). ## I Am Using The Visualizer -You only need three short chapters for normal use: +For a standard setup, read these three chapters in order: 1. [Get a dashboard running](10-getting-started.md). 2. [Choose raw, prepared, or summary inputs](11-configuring-your-data.md). 3. [Configure a live, export, or processor workflow](12-running-workflows.md). -Use [Troubleshooting](90-troubleshooting.md) when something is missing. The -[Configuration Reference](13-configuration-reference.md) is there when you need -an exact field or default; it is not required reading. +After the dashboard starts, use the +[Dashboard User Guide](16-dashboard-user-guide.md) to choose an analysis and +interpret its controls and results. + +To publish a standalone dashboard at no cost, see +[17 - Publish An Export With Posit Connect Cloud](17-posit-connect-cloud.md). + +Use [14 - Input Data Contract](14-input-data-contract.md) when you need exact +table, key, relationship, or bypass-prepare rules. Use +[15 - Cache And Manifest Reference](15-cache-manifest-reference.md) when you +need to interpret stored identities and diagnostics. + +If data is missing, see [Troubleshooting](90-troubleshooting.md). Use the +[Configuration Reference](13-configuration-reference.md) to look up a field or +default value; you do not need to read it from beginning to end. ## I Am Extending The Visualizer -| If you want to... | Read | +| Task | Read | |---|---| | Find every main config field and option | [13 - Configuration Reference](13-configuration-reference.md) | | Understand the Output Processor | [20 - Output Processor](20-output-processor.md) | | Add a prepared column | [41 - Data Extension Cookbook](41-data-extension-cookbook.md#worked-example-add-a-column-to-an-existing-prepared-table) | | Add or debug skimjoin outputs | [22 - Skimjoin](22-skimjoin.md) | -| Find every skimjoin config field and lookup option | [25 - Skimjoin Config Reference](25-skimjoin-config-reference.md) | +| Find every skimjoin config field and lookup option | [23 - Skimjoin Config Reference](23-skimjoin-config-reference.md) | +| Build summaries for configured subsets | [24 - Segmentation](24-segmentation.md) | +| Add custom zone-based geographies | [27 - Geography](27-geography.md) | | Add a summary function | [44 - Summary Function Cookbook](44-summary-function-cookbook.md) | -| Find every registered summary table | [24 - Summary Catalog](24-summary-catalog.md) | +| Find every registered summary table | [26 - Summary Catalog](26-summary-catalog.md) | | Understand the Output Visualizer | [30 - Output Visualizer](30-output-visualizer.md) | | Add a dashboard page or page group | [45 - Dashboard Extension Cookbook](45-dashboard-extension-cookbook.md) | | Add a figure, table, selector, or widget | [45 - Dashboard Extension Cookbook](45-dashboard-extension-cookbook.md) | @@ -59,23 +75,31 @@ an exact field or default; it is not required reading. - [11 - Configuring Your Data](11-configuring-your-data.md) - [12 - Running Workflows](12-running-workflows.md) - [13 - Configuration Reference](13-configuration-reference.md) +- [14 - Input Data Contract](14-input-data-contract.md) +- [15 - Cache And Manifest Reference](15-cache-manifest-reference.md) +- [16 - Dashboard User Guide](16-dashboard-user-guide.md) +- [17 - Publish An Export With Posit Connect Cloud](17-posit-connect-cloud.md) ### Output Processor - [20 - Output Processor](20-output-processor.md) - [21 - Prepared Tables](21-prepared-tables.md) - [22 - Skimjoin](22-skimjoin.md) -- [23 - Summary Functions](23-summary-functions.md) -- [24 - Summary Catalog](24-summary-catalog.md) -- [25 - Skimjoin Config Reference](25-skimjoin-config-reference.md) +- [23 - Skimjoin Config Reference](23-skimjoin-config-reference.md) +- [24 - Segmentation](24-segmentation.md) +- [25 - Summary Functions](25-summary-functions.md) +- [26 - Summary Catalog](26-summary-catalog.md) +- [27 - Geography](27-geography.md) ### Output Visualizer - [30 - Output Visualizer](30-output-visualizer.md) -- [31 - Dashboard Pages](31-dashboard-pages.md) +- [31 - Dashboard Page Contract](31-dashboard-pages.md) - [32 - Figures and Widgets](32-figures-and-widgets.md) - [33 - Dashboard Page Recipes](33-dashboard-page-recipes.md) - [34 - HTML Export](34-html-export.md) +- [35 - Plotting Reference](35-plotting-reference.md) +- [36 - HTML Export Schema](36-html-export-schema.md) ### Developer Reference @@ -91,14 +115,15 @@ an exact field or default; it is not required reading. ## Generated Pages -Some wiki sections are generated from code to keep reference material from -drifting: +The project generates some wiki sections directly from the code so that the +reference material stays accurate: -- [24 - Summary Catalog](24-summary-catalog.md) -- the generated page catalog in [31 - Dashboard Pages](31-dashboard-pages.md) +- [26 - Summary Catalog](26-summary-catalog.md) +- the generated page catalog in + [31 - Dashboard Page Contract](31-dashboard-pages.md) -Regenerate them after changing summary declarations/contracts, dashboard page -definitions, or page data requirements: +Regenerate these sections after you change a summary declaration, a summary +contract, a dashboard page definition, or a page data requirement: ```bash uv run python scripts/generate_wiki_catalogs.py diff --git a/wiki/01-architecture.md b/wiki/01-architecture.md index 85e46b8..e5e64a7 100644 --- a/wiki/01-architecture.md +++ b/wiki/01-architecture.md @@ -2,16 +2,16 @@ `activitysim_visualizer` has three main jobs: -1. Read and normalize raw ActivitySim outputs. +1. Read and normalize raw ActivitySim outputs, with optional skim enrichment. 2. Build and cache summary tables. 3. Render those summaries in a live Panel dashboard or a standalone HTML export. -The codebase is organized around those jobs rather than around one monolithic app layer. +Each job has its own subsystem. -The config surface is now intentionally split into top-level domains such as -`pipeline`, `dashboard`, `display`, `summarize`, `segment`, and `skimjoin`. -`runtime.config.load_config_from_yaml()` validates that canonical schema before -any workflow code sees it. Removed and unknown keys fail with a focused error. +The configuration has top-level sections such as `pipeline`, `dashboard`, +`display`, `summarize`, `segment`, and `skimjoin`. +`runtime.config.load_config_from_yaml()` validates the canonical schema before +the workflow uses it. Removed keys and unknown keys cause a specific error. ## Main Subsystems @@ -20,6 +20,9 @@ any workflow code sees it. Removed and unknown keys fail with a focused error. | CLI and workflow orchestration | Parse step selections, choose cache-first vs rebuild flow, and hand off to prepare/summarize/dashboard workflows | `run.py`, `runtime/workflows/` | | Shared runtime contracts | Normalize YAML config and expose shared cross-cutting contracts used by both processor and dashboard | public surface `runtime.config`, implementation in `runtime/config/` | | Processor prepare step | Read raw ActivitySim outputs, materialize canonical prepared columns, and manage prepared-table cache I/O | `processor/models.py`, `processor/prepare/*` | +| Skim enrichment | Resolve per-run lookup rules and add skim-derived prepared fields | `runtime/config/normalize_skimjoin.py`, `processor/skimjoin/` | +| Segmentation | Slice related prepared tables into analysis units before summary generation | `runtime/config/normalize_segmentation.py`, `processor/segmentation.py` | +| Geography | Normalize zone lookups and add role-specific spatial fields during prepare | `runtime/config/normalize_geography.py`, `processor/prepare/enrichment/zones.py` | | Summary generation | Declare builders and compute weighted/unweighted tables | `processor/summarize/contracts.py`, `processor/summarize/catalog.py`, `processor/summarize/summaries/*.py` | | Summary cache I/O | Inspect, write, and load cache manifests and CSVs | `processor/summarize/cache.py`, `processor/summarize/cache_storage.py` | | Dashboard page runtime | Discover pages, validate contracts, refresh declared features, and memoize section queries | `dashboard/page_registry.py`, `dashboard/page_definitions.py`, `dashboard/page_lifecycle.py`, `dashboard/page_declarations.py` | @@ -34,98 +37,113 @@ run.py -> resolve_effective_plan() from CLI overrides + config.pipeline defaults -> zero or more runtime steps: A. run_prepare_workflow() - -> processor.prepare.cache.load_prepared_run_cache() + -> inspect/load the final prepared cache + -> when skimjoin is enabled, inspect/load the base prepared cache separately -> processor.prepare.reader.read_run() -> processor.prepare.enrichment.pipeline.prepare_data() + -> processor.skimjoin.pipeline.apply_skimjoin() when selected -> processor.prepare.cache.write_prepared_run_cache() B. run_summary_workflow() - -> processor.summarize.cache.load_summary_run_cache() + -> inspect reusable/stale tables in the summary bundle -> run_prepare_workflow() on summary-cache miss + -> processor.segmentation.build_analysis_units_for_run() when selected -> processor.summarize.builder.build_mode_summaries_with_metadata() - -> processor.summarize.cache.write_summary_run_cache() + for the full run and each segment analysis unit + -> merge reusable and rebuilt tables + -> processor.summarize.cache.write_summary_run_bundle() C. load_summary_runs_from_cache() for dashboard-only cache runs - -> processor.summarize.cache.load_summary_run_cache() - D. dashboard.app.build_dashboard() - E. dashboard.export.html.build_export_html_document() + -> processor.summarize.cache.load_summary_run_bundle() + D. run_dashboard_workflow() + -> dashboard.app.build_dashboard() and Panel serve for live mode + -> dashboard.export.write_export_html_document() for export mode ``` -`WorkflowPlan` is the single resolved execution plan passed into these -operations. `run_prepare_workflow()` returns `PreparedRunsArtifact`, and -`run_summary_workflow()` returns `SummaryRunsArtifact`. Cache policy stays in -these runtime workflows; processor functions only transform tables. +The runtime passes one resolved `WorkflowPlan` to these operations. +`run_prepare_workflow()` returns `PreparedRunsArtifact`, and +`run_summary_workflow()` returns `SummaryRunsArtifact`. Runtime workflows own +the cache policy; processor functions only transform tables. ## Core Runtime Contracts ### `Config` -`runtime.config.Config` is the normalized application configuration. The public -import surface remains `runtime.config`, while the implementation now lives in -the `runtime/config/` package. +`runtime.config.Config` is the normalized application configuration. Import +the public API from `runtime.config`. The implementation is in the +`runtime/config/` package. Treat it as the contract for: -- which files are read -- which logical pipeline steps are requested by default -- which dashboard mode is used by default (`none`, `live`, `export`, `host`) -- whether a run should prefer cache reuse or overwrite behavior by default -- how schema aliases are resolved +- files that the application reads +- logical pipeline steps that the application requests by default +- default dashboard mode (`none`, `live`, `export`, `host`) +- stored stages that require a refresh +- rules to resolve schema aliases - which weighting modes exist -- which pages are enabled -- how export selector requests are configured +- enabled pages +- export selector request configuration -`dashboard.host` is a reserved placeholder for a future hosting integration. -The schema accepts `account`, `app_id`, `title`, and `verify`, but the current -runtime deliberately does not store or act on them. +`dashboard.host` is reserved for a future hosting integration. The schema +accepts `account`, `app_id`, `title`, and `verify`. The runtime does not store +or use these values. -If a new feature adds a config key or changes config behavior, update the README -and the relevant wiki chapters in the same change. +If a feature adds a configuration key or changes configuration behavior, +update the README and the related wiki chapters in the same change. `Config.pipeline` is the canonical home for workflow defaults. Today the logical step names are: - `prepare` - `skimjoin` -- `summarize` - `segment` +- `summarize` - `dashboard` -The runtime still executes three coarse workflow boundaries (`prepare`, -`summarize`, `dashboard`). `skimjoin` currently resolves inside the prepare -workflow, and `segment` currently resolves inside the summarize workflow. +The runtime executes three main workflow boundaries: `prepare`, `summarize`, +and `dashboard`. The runtime resolves `skimjoin` in the prepare workflow. It +resolves `segment` in the summarize workflow. + +For detailed flows, see [22 - Skimjoin](22-skimjoin.md), +[24 - Segmentation](24-segmentation.md), and +[27 - Geography](27-geography.md). ### `RunData` -`processor.models.RunData` is the prepared-data contract consumed by summary builders and prepared-data dashboard pages. Summary code should rely on canonical prepared columns rather than guessing raw ActivitySim column names directly. `processor/prepare/` is the layer that materializes those canonical fields and owns prepared-table cache helpers. +`processor.models.RunData` is the prepared-data contract used by summary +builders and prepared-data dashboard pages. Summary code uses canonical +prepared columns and never guesses the names of raw ActivitySim columns. The +`processor/prepare/` subsystem creates these canonical fields and contains the +prepared-table cache helpers. ### `@summary` and the summary catalog -Each persisted summary is declared beside its builder with `@summary(...)`. The +Declare each persistent summary next to its builder with `@summary(...)`. The declaration defines: - the stable summary id used by dashboard pages -- the CSV filename stem used in cache directories +- the CSV file-name stem used in cache directories - its ordered output schema and prepared-input prerequisites -- whether it is built by default +- default build status -`processor.summarize.catalog` imports the owning domain modules explicitly, -collects those declarations deterministically, and rejects duplicate ids. -Successful builder results are validated for exact columns, order, and dtypes. -Unexpected builder exceptions follow `summarize.failure_policy`: `record` keeps -typed failure metadata for an interactive dashboard, while `error` is the -fail-fast setting for validation and batch workflows. +`processor.summarize.catalog` imports the relevant domain modules. It +collects the declarations in a repeatable order and rejects duplicate IDs. The +system validates the columns, column order, and data types of each successful +builder result. The `summarize.failure_policy` setting controls unexpected +builder exceptions. The `record` value keeps typed failure metadata for an +interactive dashboard. The `error` value stops validation and batch workflows +immediately. ### `DashboardPageDefinition` and `DashboardPage` -Dashboard pages are registered with `@dashboard_page(...)` on the page class in -`dashboard/pages/`. The decorator holds identity, navigation grouping, ordering, -and the summary/prepared-data contract through `required_summary_ids`, +Register a dashboard page with `@dashboard_page(...)` on its page class in +`dashboard/pages/`. The decorator defines identity, navigation group, order, +and the summary or prepared-data contract through `required_summary_ids`, `optional_summary_ids`, `prepared_data_mode`, and `required_prepared_tables`. `dashboard.page_base` is the small public facade. Lifecycle, declarations, diagnostics, feature composition, data access, and grouped navigation live in separate implementation modules. -Page authors are expected to: +Page authors must: - implement `build_page()` to declare selectors, features, sections, and layout - give selectors an option provider and default policy when their domain is dynamic @@ -133,10 +151,9 @@ Page authors are expected to: - memoize chart-ready transformations with `self.query(...)` - keep section render methods to lookup/query/render -Large controllers may keep their registered page module as a compatibility -facade and compose page-local implementation mixins from a private `_/` -package. This convention, its constraints, and its distinction from -`PageFeature` are documented in +Large controllers can keep the registered page module as a compatibility +facade. They can use page-local implementation mixins from a private `_/` +package. For the rules and the difference from `PageFeature`, see [Figures And Widgets](32-figures-and-widgets.md#sections-and-features). The framework now owns: @@ -150,10 +167,10 @@ The framework now owns: - export selector metadata - export region metadata -That means live refresh behavior and export behavior both derive from the same selector/section registration graph rather than from separate page metadata declarations. +The same selector and section registration graph controls both live refresh and +export behavior; separate page metadata does not. -The shared helper layer under `dashboard/helpers/` is now part of that page -authoring model: +The page authoring model includes the shared helpers in `dashboard/helpers/`: - `category_helpers.py` centralizes selector domains, labels, and category completion - `geography_helpers.py` centralizes geography normalization, option discovery, and filters @@ -161,48 +178,97 @@ authoring model: - `time_distance_helpers.py` centralizes repeated time-bin and distance-bin behavior - `comparison_helpers.py` centralizes percent-error formatting and base-run comparisons -For page-local table shaping, `dashboard.data_access.RunTables` applies one -fluent query to every run while preserving run labels. Pages should prefer its +For page-local table changes, `dashboard.data_access.RunTables` applies one +query to every run while preserving the run labels. Pages should use its `where`, `with_columns`, `group`, `select`, `sort`, `join`, `requiring`, -`drop_empty`, and `map` operations over open-coded loops through -run/dataframe pairs. - -The skim pages share their family-specific model/query service while exposing -small summary and distribution features. This is the reference pattern for -logic reusable within one page family but not broad enough for -`dashboard/helpers/`. +`drop_empty`, and `map` operations when possible. Do not write equivalent loops +through run and data frame pairs. + +The skim pages use a model and query service for their page family. Each page +provides small summary and distribution features. Use this pattern for logic +that one page family shares. Put more general logic in `dashboard/helpers/`. + +## Public Python APIs + +Use these facades when you extend or embed the visualizer. Files they do not +export are implementation details unless a cookbook identifies a specific +extension point. + +| Import surface | Public contract | +|---|---| +| `runtime.config` | `Config`, `load_config_from_yaml()`, `config_for_run()`, `resolve_run_skimjoin_settings()`, normalized export/pipeline/prepare/segmentation setting types, and weighting registry types. `Config.from_yaml()` is the equivalent class entry point. | +| `runtime.workflows` | Config/run resolution; prepared and summary cache roots/loaders; `run_prepare_workflow()`, `run_summary_workflow()`, and `run_dashboard_workflow()`; consumer pruning; `WorkflowPlan`, `PreparedRunsArtifact`, `SummaryRunsArtifact`, and `SummaryCacheInspection`. Workflow functions are keyword-oriented and return artifacts rather than hidden module state. | +| `processor` | `RunData`, the canonical prepared-run data contract. | +| `processor.summarize` | `summary`, the declaration decorator for registered summary builders. | +| `dashboard` | `DashboardPage`, `dashboard_page`, `DashboardState`, `PageData`, `RunTables`, and prepared/summary provider types used by page and embedding code. | +| `dashboard.page_base` | `GroupedDashboardPage`, `PageFeature`, selector/section declaration types, and `PAGE_SELECTOR_STYLESHEET`, in addition to `DashboardPage`. | +| `dashboard.rendering` | `RenderContext`, `FigureBuilder`, `Plotter`, table/formatting helpers, selector/control rows, legends, and the standard unavailable card. | +| `dashboard.export` | `build_export_html_document()` for an in-memory document and `write_export_html_document()` for the streamed file/diagnostics workflow. | + +The normalized config value objects exported alongside `Config` are +`CategorySpec`, `PipelineSettings`, `ExportDashboardSettings`, +`ExportHTMLSettings`, `ExportSelectorRequest`, +`PrepareCategoryMappingsSettings`, `PrepareNonMotorizedDistanceSkimSettings`, +`SegmentationDefinition`, +`PreparedColumnSegmentationSource`, `CsvLookupSegmentationSource`, and +`StudentTypeConfig`. These are read-only runtime contracts populated through +YAML normalization; do not assemble a `Config` manually. + +The workflow facade also exports `effective_processor_config()`, +`run_entries_with_keys()`, `prepared_cache_root()`, `summary_cache_root()`, +`prune_summary_runs()`, and `prune_summary_artifact()`. Embedding code can use +them to get the same identity and removal behavior as `run.py`. The +dashboard facade exports `DashboardPreparedRunProvider` and +`DashboardSummarySeries`; the page-base facade exports the typed +`RegisteredPageSelector`, `RegisteredPageSection`, and `SectionContent` +declaration records. + +Chapter 32 describes the page-facing `PageData` and `RunTables` API, chapter 35 +covers chart keywords, and chapter 25 defines the `@summary` contract. The +subsystem sections above describe workflow arguments and artifacts. When public +code needs behavior that differs from the loaded configuration, it must pass an +explicit `WorkflowPlan` that records logical steps, runtime boundaries, +dashboard mode, and refresh targets. ## Repository Map ```text activitysim_visualizer/ |-- run.py -|-- runtime/ -| |-- workflows/ |-- config.yaml |-- runtime/ -| `-- config/ +| |-- config/ # canonical schema, normalizers, models, signatures +| |-- workflows/ # prepare/summarize/dashboard orchestration and artifacts +| |-- logging.py +| |-- run_lock.py +| `-- weighting.py |-- processor/ +| |-- analysis_units.py +| |-- cache_identity.py +| |-- cache_infra.py | |-- models.py +| |-- segmentation.py | |-- prepare/ -| | |-- __init__.py | | |-- availability.py | | |-- cache.py | | |-- enrichment/ -| | | |-- __init__.py | | | |-- canonicalize.py | | | |-- columns.py | | | |-- domains.py | | | |-- finalize.py | | | |-- households_persons.py +| | | |-- non_motorized_distance.py | | | |-- pipeline.py +| | | |-- student_enrollment.py +| | | |-- time_periods.py | | | |-- tours.py | | | |-- trips.py -| | | |-- types.py | | | |-- weights.py | | | `-- zones.py | | |-- reader.py +| | |-- validation.py | | `-- writer.py +| |-- skimjoin/ # config, inventory, annotation, stores, QA reports, CLI | `-- summarize/ | |-- builder.py | |-- cache.py @@ -211,25 +277,20 @@ activitysim_visualizer/ | |-- catalog.py | |-- contracts.py | |-- csv_export.py +| |-- external.py | |-- schema.py +| |-- validation_derived.py | `-- summaries/ -| |-- daily_travel_activity.py -| |-- daily_travel_escort_counts.py -| |-- daily_travel_escort_distributions.py -| |-- demographics.py -| |-- joint_travel.py -| |-- long_term_person.py -| |-- long_term_vehicle.py -| |-- long_term_geography.py -| |-- long_term_distance.py -| |-- tour.py -| |-- trip.py -| `-- validation.py +| `-- |-- dashboard/ | |-- app.py +| |-- calculation_notes.py / calculation_notes.yaml +| |-- data_access.py +| |-- helpers/ | |-- rendering/ | | |-- context.py | | |-- figures.py +| | |-- labels.py | | |-- plotter.py | | |-- layout.py | | `-- tables.py @@ -242,6 +303,7 @@ activitysim_visualizer/ | | |-- traversal.py | | |-- runtime_assets.py | | |-- types.py +| | |-- js_runtime/ | | `-- assets/ | |-- page_base.py | |-- page_declarations.py @@ -253,6 +315,15 @@ activitysim_visualizer/ | |-- page_registry.py | |-- state.py | `-- pages/ +|-- scripts/ +| |-- generate_wiki_catalogs.py +| |-- generate_validation_demo_fixtures.py +| |-- processor_output_catalog_metadata.yaml +| |-- run_simor_scenarios.py +| `-- summary_catalog_metadata.yaml +|-- reference/ +| `-- processor-output-table-reference.md +|-- wiki/ `-- tests/ ``` diff --git a/wiki/10-getting-started.md b/wiki/10-getting-started.md index dee4f37..29fa6bb 100644 --- a/wiki/10-getting-started.md +++ b/wiki/10-getting-started.md @@ -1,6 +1,6 @@ # 10 - Getting Started -This is the shortest path from a clone to a local dashboard. +Follow these steps to start a local dashboard from a repository clone. ## 1. Install @@ -18,8 +18,7 @@ uv sync --locked --link-mode=copy ## 2. Create A Small Config -Create `local_config.yaml`. This file defines both the inputs and what the run -should produce: +Create `local_config.yaml` to define the input and output: ```yaml root: artifacts @@ -27,7 +26,7 @@ root: artifacts pipeline: steps: [prepare, summarize, dashboard] dashboard_mode: live - overwrite: false + refresh: [] runs: - dir: C:\models\base\output @@ -46,18 +45,20 @@ dashboard: output_path: exports/dashboard.html ``` -Change the two `dir` values to real ActivitySim output folders. The default -input names are `final_households`, `final_persons`, `final_tours`, -`final_trips`, `final_joint_tour_participants`, and `final_land_use`; each may be -CSV or Parquet. +Set the two `dir` values to ActivitySim output directories. The default +input names are `final_households`, `final_persons`, `final_day`, +`final_tours`, `final_trips`, `final_vehicles`, +`final_joint_tour_participants`, and `final_land_use`. Day, vehicle, joint +participant, and land-use tables are optional. Each input file can be CSV or +Parquet. If your files have different names, read [File Names](11-configuring-your-data.md#raw-activitysim-output). -`root` is the visualizer's artifact location. Summary caches are written below -it, and relative export paths resolve below it. Keep the export path configured -even for a live workflow; switching from a live dashboard to an HTML file then -requires changing only `pipeline.dashboard_mode` from `live` to `export`. +`root` is the artifact directory, where the visualizer writes summary caches +and resolves relative export paths. Keep the export path in the configuration +for a live workflow. To create HTML later, you only need to change +`pipeline.dashboard_mode` from `live` to `export`. ## 3. Run The Config @@ -65,25 +66,25 @@ requires changing only `pipeline.dashboard_mode` from `live` to `export`. uv run activitysim-viz --config local_config.yaml ``` -The first run prepares data, builds summaries, and starts the dashboard at -[http://localhost:5006](http://localhost:5006). Later runs reuse valid caches. +On the first run, the visualizer prepares the data, builds the summaries, and +starts the dashboard at [http://localhost:5006](http://localhost:5006). Later +runs reuse valid caches. -Stop the server with `Ctrl+C`. +To stop the server, press `Ctrl+C`. -Use this same command for live dashboards, HTML exports, and processor-only -workflows. Change the `pipeline` and `dashboard` sections in the config instead -of maintaining different launch commands. +The command is the same for live dashboards, HTML exports, and processor-only +workflows. Select the workflow in the `pipeline` and `dashboard` sections. -## If The First Run Fails +## If the first execution fails -Check these first: +Check the following: -1. each `runs[*].dir` exists; -2. the expected tables are present as `.csv` or `.parquet`; -3. `zones.use_maz`, `maz_col`, and `taz_col` match the model; and -4. the log names the missing file or column. +1. Make sure each `runs[*].dir` exists. +2. Make sure each required table is a `.csv` or `.parquet` file. +3. Make sure `zones.use_maz`, `maz_col`, and `taz_col` agree with the model. +4. Find the missing file or column in the log. -Then use [Troubleshooting](90-troubleshooting.md). +For more help, see [Troubleshooting](90-troubleshooting.md). ## Next diff --git a/wiki/11-configuring-your-data.md b/wiki/11-configuring-your-data.md index 8390ddb..b05c082 100644 --- a/wiki/11-configuring-your-data.md +++ b/wiki/11-configuring-your-data.md @@ -1,11 +1,31 @@ # 11 - Configuring Your Data -Most users only need to choose an input type and name their runs. Use one of the -three patterns below. +Choose one of the three input types below, and give each run a label. For exact +table, key, relationship, and type rules, see +[14 - Input Data Contract](14-input-data-contract.md). + +## Run Input Decision Matrix + +The fields on one run can be combined only when their boundaries make sense: + +| Run input | Prepare source | Skimjoin | Segmentation | Generated summaries | Mapped-summary behavior | +|---|---|---|---|---|---| +| `dir` | Raw CSV/Parquet files | Available when the step and paths are configured | Available | Available | Optional mapped IDs replace generated IDs. | +| `prepared_table_map` | Supplied canonical tables; raw prepare is skipped | Skipped for this run | Available | Available | Optional mapped IDs replace generated IDs. | +| `summary_table_map` only | None | Not available | Not available | Not available | Mapped IDs are the run's summaries. | +| `dir` plus `summary_table_map` | Raw files | Available | Available | Available for unmapped IDs | The same mapped table replaces its ID for full and segmented units. | +| `prepared_table_map` plus `summary_table_map` | Supplied canonical tables | Skipped | Available | Available for unmapped IDs | The same mapped table replaces its ID for full and segmented units. | +| `--from-csvs ` | Existing manifested cache bundle | Already complete | Already complete | Not run | Loads the bundle; it is not a loose-file mapping. | + +`file_map`, `skim_file`, and the run weight fields apply to raw `dir` +input. `file_map` cannot be combined with `prepared_table_map`. A +`summary_table_map` entry is mode-independent: built-in weighted and unweighted +modes copy it, while declarative named modes reject it because they cannot +recalculate an aggregated file. ## Raw ActivitySim Output -Use this when you have normal ActivitySim output folders: +Use this configuration for standard ActivitySim output directories: ```yaml root: artifacts @@ -17,7 +37,7 @@ runs: label: Build ``` -The label is what appears in the dashboard. +The dashboard uses the label to identify the run. ### File Names @@ -27,14 +47,18 @@ The default files are: files: households: final_households persons: final_persons + day: final_day tours: final_tours trips: final_trips + vehicles: final_vehicles joint_tour_participants: final_joint_tour_participants land_use: final_land_use ``` -A bare name accepts either `.parquet` or `.csv`. Override one unusual run with -`file_map`: +The day, vehicle, joint-participant, and land-use tables are optional. + +A name without an extension selects a `.parquet` or `.csv` file. Use `file_map` +to set nonstandard file names for one run: ```yaml runs: @@ -49,7 +73,8 @@ runs: ### Column Names -If a model uses different column names, list the candidates in preferred order: +If a model uses different column names, list the possible names in order of +preference: ```yaml columns: @@ -58,13 +83,14 @@ columns: trip_mode: mode ``` -Prepare converts the selected source to the visualizer's canonical column. See -chapter 13 for the [complete column list](13-configuration-reference.md#columns). +The prepare step copies the first available source into the canonical +visualizer column. See the +[complete column list](13-configuration-reference.md#columns) in chapter 13. ## Already-Prepared Tables -Use `prepared_table_map` for canonical tables that were prepared, skimjoined, -or filtered elsewhere: +Use `prepared_table_map` for canonical tables created by another process. That +process can prepare, skimjoin, or filter the tables: ```yaml runs: @@ -77,14 +103,19 @@ runs: land_use: prepared/land_use.parquet ``` -Paths must end in `.csv` or `.parquet` and are relative to the config file. -These tables must already use the canonical prepared columns expected by -summaries. Raw prepare and integrated skimjoin are skipped for this run. +Each path must end in `.csv` or `.parquet`. Relative paths start from the +configuration file directory. The tables must contain the canonical prepared +columns required by the summaries. For this type of run, the visualizer skips +raw preparation and integrated skimjoin. + +The visualizer also skips canonicalization, derived columns, standard weight +creation, and geography mapping. The supplied files must already satisfy those +parts of the prepared contract. ## Dashboard-Ready Summary Tables -Use `summary_table_map` when another process has already produced registered -summary tables: +Use `summary_table_map` for registered summary tables created by another +process: ```yaml runs: @@ -94,13 +125,18 @@ runs: traffic_count_comparisons: summaries/traffic_counts.parquet ``` -Keys must appear in the [Summary Catalog](24-summary-catalog.md). Files must -match the registered columns exactly. A run may contain only outside summaries, -or they may override selected summaries generated from raw/prepared data. +Each key must appear in the [Summary Catalog](26-summary-catalog.md), and each +file must have the registered columns in the specified order. A run can contain +only external summaries, or external summaries can replace selected summaries +from raw or prepared data. + +Mapped summary tables cannot be segmented or reweighted from their rows. When +combined with buildable input, one mapped table is overlaid unchanged on every +full or segmented analysis unit. ## Weights -The normal modes are configured with: +Configure the standard modes with: ```yaml summarize: @@ -116,13 +152,23 @@ runs: hh_weight_col: household_weight person_weight_col: person_weight trip_weight_col: trip_weight + day_weight_col: diary_day_weight ``` -Otherwise prepare uses a configured sample-rate column when available, then -falls back to `1.0`. +If none of the household, person, or trip weight fields is set, prepare uses +the configured household sample-rate column when available and otherwise +starts with `1.0`. Setting any of those three fields disables automatic +household sample-rate expansion. Tables without their own source inherit a +related person or household weight when possible. + +`day_weight_col` defaults to `day_weight`. When that column exists on the day +table, each non-null value is authoritative; null values fall back to the +matching person weight, then household weight, then `1.0`. Set +`day_weight_col: null` to ignore a placeholder day-weight column and inherit +person or household weights for every day row. -If the same output tables contain an additional set of weights, add a named -column mode instead of duplicating the run or writing Python: +If the output tables contain other weights, add a named column mode instead of +duplicating the run or writing Python: ```yaml weighting: @@ -138,8 +184,10 @@ summarize: weighting_modes: [weighted, unweighted, calibrated] ``` -The named sources are validated and propagated to tours, days, vehicles, and -skimjoin sidecars as appropriate. See [43 - Weighting And Hosting Extensions](43-weighting-hosting-extensions.md#worked-example-add-a-weighting-mode) for the exact rules. +The visualizer validates the named sources and copies the weights to relevant +tours, days, vehicles, and skimjoin sidecar tables. See +[43 - Weighting And Hosting Extensions](43-weighting-hosting-extensions.md#worked-example-add-a-weighting-mode) +for the rules. ## Zones @@ -164,10 +212,10 @@ zones: ## Optional Features - For skim enrichment, read [Skimjoin](22-skimjoin.md). -- For custom geography aggregation, read the - [`summarize.geography` reference](13-configuration-reference.md#summarize). -- For segmentation, read the - [`segment` reference](13-configuration-reference.md#segment). +- To build the same summaries for configured subsets, read + [Segmentation](24-segmentation.md). +- To add district, county, or other zone mappings, read + [Geography](27-geography.md). - For every accepted key and default, use the [Configuration Reference](13-configuration-reference.md). diff --git a/wiki/12-running-workflows.md b/wiki/12-running-workflows.md index 670a1da..72f767f 100644 --- a/wiki/12-running-workflows.md +++ b/wiki/12-running-workflows.md @@ -1,39 +1,52 @@ # 12 - Running Workflows -The normal user experience is config-driven. Keep one launch command: +One configuration controls the standard workflow, and every workflow uses the +same start command: ```bash uv run activitysim-viz --config local_config.yaml ``` -The config decides which work runs, where artifacts are stored, and whether the -result is a live dashboard or an HTML file. Command-line flags are intended for -development and one-off diagnostics, not normal operation. +The configuration selects the work, artifact location, and dashboard mode. +Reserve command-line flags for development or one-time diagnostics. -## The Three Main Steps +## Workflow Order ```text -prepare -> summarize -> dashboard +prepare -> optional skimjoin -> optional segmentation -> summarize -> dashboard ``` - **Prepare** reads raw outputs and creates canonical prepared tables. - **Summarize** creates the smaller tables used by dashboard pages. -- **Dashboard** serves the live application or writes standalone HTML. +- **Dashboard** starts the live application or writes standalone HTML. -Skimjoin runs inside prepare when selected. Segmentation runs with summarize. +Skimjoin runs inside the prepare boundary when selected. Segmentation resolves +and slices prepared data inside the summarize boundary. Geography enrichment +runs during prepare and geography-aware aggregations run during summarize; it +is a feature, not a separate pipeline step. -These are requested workflow boundaries, not isolated commands. In particular, -`summarize` must have prepared data: it reuses a valid prepared cache or builds -prepared data from the configured raw/prepared inputs when the cache is missing -or stale. Adding `prepare` explicitly runs and persists that boundary first; -the summarize boundary then reuses the in-memory or cached result rather than -preparing a second time. +The written order of non-dashboard values in `pipeline.steps` does not control +runtime order. Those values select logical capabilities, and the runtime +resolves the fixed dependency order above. `dashboard`, when present, must be +the last listed value. Use the canonical order in every example and local +configuration because it makes intent clear: + +```yaml +pipeline: + steps: [prepare, skimjoin, segment, summarize, dashboard] +``` + +These steps are workflow boundaries, not independent commands. The `summarize` +step requires prepared data, so it reuses a valid prepared cache or builds the +data from the configured input. If the workflow includes `prepare`, the runtime +completes and stores that step first; `summarize` then uses the result from +memory or the cache without preparing it again. | Requested step | What it guarantees | Prerequisites resolved automatically | |---|---|---| -| `prepare` | Prepared tables are loaded/built and cached. | Raw files or `prepared_table_map`. | -| `summarize` | Default registered summaries are loaded/built and cached. | Prepared data is loaded/built as needed. | -| `dashboard` | Existing summary caches are loaded and displayed/exported. | No summaries are built; required caches must exist or come from `summary_table_map`. | +| `prepare` | The runtime loads or builds prepared tables and writes the cache. | Raw files or `prepared_table_map`. | +| `summarize` | The runtime loads or builds default registered summaries and writes the cache. | The runtime loads or builds prepared data as necessary. | +| `dashboard` | The runtime loads existing summary caches and shows or exports them. | The runtime does not build summaries. Required caches must exist or come from `summary_table_map`. | ## Configure A Live Workflow @@ -43,7 +56,7 @@ root: artifacts pipeline: steps: [prepare, summarize, dashboard] dashboard_mode: live - overwrite: false + refresh: [] dashboard: title: Regional Model Comparison @@ -56,8 +69,8 @@ dashboard: - trip_summaries ``` -This builds missing or stale artifacts, reuses valid caches, and starts the -dashboard. `dashboard.live.pages` controls which page groups are available. +This workflow reuses valid caches, builds any missing or stale artifacts, and +starts the dashboard. `dashboard.live.pages` selects the available page groups. ## Configure An HTML Export @@ -67,27 +80,27 @@ root: artifacts pipeline: steps: [prepare, summarize, dashboard] dashboard_mode: export - overwrite: false + refresh: [] dashboard: export: output_path: exports/dashboard.html ``` -The configured output is `artifacts/exports/dashboard.html`: relative export -paths resolve below `root`. Use an absolute path when the file must be written -elsewhere. Page and selector choices are covered in +This configuration writes `artifacts/exports/dashboard.html`. Relative export +paths start from `root`. Use an absolute path to write the file to a different +location. For page and selector choices, see [HTML Export](34-html-export.md). ## Configure A Processor-Only Workflow -Build prepared tables and summaries without opening or exporting a dashboard: +Build prepared tables and summaries without a dashboard: ```yaml pipeline: steps: [prepare, summarize] dashboard_mode: none - overwrite: false + refresh: [] ``` Other focused workflows use the same fields: @@ -99,32 +112,49 @@ Other focused workflows use the same fields: | Open a live dashboard from existing caches | `[dashboard]` | `live` | | Export HTML from existing caches | `[dashboard]` | `export` | -For loose dashboard-ready CSV or Parquet inputs, configure -`runs[*].summary_table_map`; do not treat them as cache directories. +For dashboard-ready CSV or Parquet files, configure +`runs[*].summary_table_map`. Do not use the files as cache directories. ## Pipeline Rules -Available logical steps are `prepare`, `skimjoin`, `segment`, `summarize`, and -`dashboard`. Dashboard must be last. `skimjoin` requires `prepare`; `segment` -requires `summarize`. +The logical steps are `prepare`, `skimjoin`, `segment`, `summarize`, and +`dashboard`. Values must be lowercase, unique, and valid. Put `dashboard` last. +`skimjoin` requires `prepare`, and `segment` requires `summarize`. The runtime +uses step membership, not the listed order, to form the prepare, summarize, +and dashboard boundaries. + +If you omit `pipeline.steps`, it defaults to `[summarize, dashboard]` and +prepares raw input when no valid prepared cache is available. Add `prepare` +when cache creation must be a visible step or when you enable `skimjoin`. + +Add `segment` when the summarize workflow must build configured subsets: + +```yaml +pipeline: + steps: [segment, summarize, dashboard] + dashboard_mode: live + refresh: [] +``` -The default when `pipeline.steps` is omitted is `[summarize, dashboard]`. -That default still prepares raw inputs when a valid prepared cache is not -available. Include `prepare` explicitly when prepared-cache creation is itself -an intended, visible stage or when `skimjoin` is enabled. +The `segment` step requires `summarize`. Its configuration alone does not +enable segmentation. Dashboard modes: - `live`: local Panel server; - `export`: standalone HTML; - `none`: no dashboard; and -- `host`: reserved extension point that currently logs a warning and executes - the normal live server; it does not publish to a hosting provider. +- `host`: reserved extension point. It writes a warning to the log and starts + the standard live server. It does not publish to a hosting provider. + +To host an exported dashboard as a public static file, use +[17 - Publish An Export With Posit Connect Cloud](17-posit-connect-cloud.md). +That workflow uses `dashboard_mode: export`, not `host`. ## Artifact And Cache Paths -Prepared and summary caches live under the configured `root`. Each run has a -manifest describing its inputs and config identity. +The visualizer stores prepared and summary caches under the configured `root`. +Each run has a manifest that describes its input and configuration identity. Set `root` once for the workflow: @@ -132,13 +162,14 @@ Set `root` once for the workflow: root: D:\activitysim_visualizer\regional_comparison ``` -For two runs labeled `Base` and `Build`, the normal layout is: +For two runs labeled `Base` and `Build`, the standard layout is: ```text regional_comparison/ base/ - manifest.json + manifest.json # summary-bundle manifest prepared_tables/ + manifest.json # final prepared/skimjoin identity households.parquet persons.parquet tours.parquet @@ -150,43 +181,165 @@ regional_comparison/ unweighted/ .csv build/ - manifest.json prepared_tables/ + manifest.json summary_tables/ + manifest.json ``` -The run-key directory is a filesystem-safe lowercase slug of the run label. -For example, `Build Scenario` becomes `build-scenario`. Colliding labels receive -ordered suffixes such as `build-1` and `build-2`; avoid duplicate labels because -reordering them changes which run receives each suffix. +The run-level `manifest.json` describes the summary bundle, while each prepared +cache has a manifest in its table directory. A prepare-only workflow therefore +writes `prepared_tables/manifest.json` but does not create the run-level summary +manifest. + +When you enable skimjoin, `base_prepared_tables/` contains a second prepared +manifest and the canonical tables before skim enrichment. The visualizer stores +enriched tables, skimjoin reports, and optional hypothetical sidecar tables +under `prepared_tables/`, giving summary and dashboard consumers one final path: + +```text +base/ + base_prepared_tables/ + manifest.json + trips.parquet + tours.parquet + ... + prepared_tables/ + manifest.json + trips.parquet + tours.parquet + trip_hypothetical_skims.parquet # only when enabled and populated + tour_hypothetical_skims.parquet # only when enabled and populated + skimjoin/ + config_normalized.yaml + .csv +``` + +The visualizer stores segmented summary CSV files in +`summary_tables//segments///`. The +run-level summary manifest describes these files. With `refresh: []`, summary +reuse is evaluated separately for the full run and for each configured segment. +Enabling a new segment therefore reuses compatible full summaries and builds +only the new segment summaries. Changing one segment rebuilds that segment; +removing one deletes its obsolete cached summary directory on the next summary +cache write. The prepared cache is loaded to resolve segment membership, but a +valid prepared or skimjoin cache is not recomputed. + +The run-key directory uses a lowercase, file-system-safe form of the run label; +for example, `Build Scenario` becomes `build-scenario`. Avoid duplicate labels, +which receive ordered suffixes such as `build-1` and `build-2`. Changing their +order also changes each suffix. Relative paths in `dashboard.export.output_path` resolve below this directory. Input paths follow the path rules documented in [Configuration Reference](13-configuration-reference.md#reading-this-reference). -Valid caches are reused automatically. To deliberately rebuild every cache -used by the configured steps, temporarily set: +The visualizer reuses valid caches automatically. To rebuild every stored stage +for the configured steps, temporarily set: ```yaml pipeline: steps: [prepare, summarize, dashboard] dashboard_mode: live - overwrite: true + refresh: all ``` -Return `overwrite` to `false` after the forced rebuild. Presentation-only -changes such as labels, colors, or enabled pages normally do not require cache -rebuilding. +Set `refresh` to `[]` after the rebuild. To rebuild summaries and keep prepared +and skimjoined data, use `refresh: [summarize]`. Changes to labels, colors, or +enabled pages do not usually require a cache rebuild. + +The refresh targets are stage-aware: + +| Refresh target | Reused | Rebuilt when enabled | +|---|---|---| +| `prepare` | nothing upstream of prepare | base prepared data, skimjoin output, summaries | +| `skimjoin` | `base_prepared_tables` | enriched `prepared_tables`, summaries | +| `summarize` | final `prepared_tables` | all default summaries for the full run and every configured segment | + +Cache reuse also depends on file metadata. Prepared manifests record the path, +size, and modification time of each raw input, along with the prepare, skimjoin, +and skim input identities. The summary manifest records the prepared-manifest +identity, summary configuration, and declaration digest for each summary. A +changed raw file invalidates prepare and all later output, while a changed skim +input can rebuild only skimjoin and its later output. A changed declaration can +rebuild only the affected summary table for each analysis unit. Segment +definitions are tracked separately from the full-summary configuration, so +compatible full and segment tables remain in the bundle. + +Use `--explain-cache` to print the cache decision for each run. The command +then exits without table loads, cache deletions, or artifact writes. The report +shows `REUSE`, `REBUILD`, `RUN`, or `DISABLED` for each workflow step. It also +shows the cache-validation reason when one is available. + +For annotated manifest examples, every stored field, and cache-recovery rules, +see [15 - Cache And Manifest Reference](15-cache-manifest-reference.md). ## CLI Overrides -CLI step, refresh, export-path, and port flags remain available for developers -and troubleshooting. They override the configured workflow for that one -invocation. Users should normally change the YAML and continue running the same -command so the intended workflow remains reproducible. +Command-line flags override the configured workflow for one run. For normal +operation, change the YAML so that the workflow remains reproducible. + +| Flag | Behavior | +|---|---| +| `--config PATH`, `-c PATH` | Load the named main config. The default is `config.yaml` next to `run.py`. | +| `--run DIR LABEL` | Replace configured `runs` with one CLI run. Repeat the flag for multiple runs. | +| `--run-skim PATH ...` | Supply one legacy prepare distance-skim path per `--run`, in order. Use `null` or an empty string to inherit `prepare.distance_skim.file`. | +| `--prepare` | Select the coarse prepare boundary for this invocation. | +| `--summarize` | Select the coarse summarize boundary for this invocation. | +| `--dashboard` | Select the dashboard boundary and force live mode unless `--export-html` is also present. | +| `--prepare-only` | Select only prepare; it cannot be combined with the three explicit step flags. | +| `--write-csvs` | Bypass reusable summary tables and force summary CSV/manifest writes; requires summarize. | +| `--from-csvs [CACHE_DIR ...]` | Run dashboard-only and load completed summary-cache directories explicitly. These are cache bundles with manifests, not loose CSV files. | +| `--skip-summary-cache-write` | Build summaries in memory without writing missing or stale summary cache entries; requires summarize. | +| `--refresh-prepared-cache` | Force prepared data and all affected downstream output to rebuild for selected runs. | +| `--refresh-summary-cache` | Preserve prepared directories and force summary output to rebuild. | +| `--refresh-caches` | Force both prepared and summary cache layers to rebuild. | +| `--export-html [PATH]` | Use export mode for a selected dashboard step. An omitted path uses `dashboard.export.output_path`, then `/exported_dashboard.html`. | +| `--port PORT` | Live-server port; default `5006`. | +| `--no-show` | Start the live server without opening a browser. | +| `--explain-cache` | Print the cache plan and exit without executing it. | +| `--log-path PATH` | Write this process's runtime log to an explicit file instead of the default shared log. | + +If you use `--prepare`, `--summarize`, or `--dashboard`, these flags replace +`pipeline.steps` with the selected main boundaries. They do not enable the +`skimjoin` or `segment` steps. Do not combine `--from-csvs` with processor steps +or `--write-csvs`. The `--write-csvs` and `--skip-summary-cache-write` flags +require summarize. Each refresh flag requires its corresponding processor +boundary. If the configuration omits dashboard, use `--export-html` with +`--dashboard`. + +Each executing visualizer workflow holds a lock in its configured output root. +A second executing process targeting the same root exits before reading or +writing caches. `--explain-cache` is the exception: it skips the lock, reads +cache manifests, prints the plan, and exits without workflow writes. + +### SIMOR scenario runner + +The SIMOR runner exports the Metro, LCOG, and SKATS dashboards with bounded +parallelism, then exports the comparison dashboard after all three succeed: + +```bash +uv run python scripts/run_simor_scenarios.py +``` + +It defaults to two concurrent area builds and writes separate runtime and +console logs under `simor_project_outputs/logs/scenario_runner/`. + +| Flag | Behavior | +|---|---| +| `--dry-run` | Print the staged area and comparison commands without running them. | +| `--max-parallel N` | Set concurrent area builds; the default is `2`. The comparison still starts only after every area succeeds. | +| `--log-root PATH` | Set the parent directory for timestamped per-scenario runtime and console logs. | +| `--refresh-caches` | Forward a full cache refresh to every dashboard build. Do not combine this with `--exports-only`, because refresh requires processor boundaries. | +| `--exports-only` | Run dashboard and HTML export from existing valid caches, without processor steps. This cannot be combined with `--refresh-caches` or `--no-skimjoin`. | +| `--no-skimjoin` | Run prepare, summarize, and HTML export while skipping skimjoin. This cannot be combined with `--exports-only`. | ## Related Chapters - [Getting Started](10-getting-started.md) - [Configuring Your Data](11-configuring-your-data.md) +- [Input Data Contract](14-input-data-contract.md) +- [Cache And Manifest Reference](15-cache-manifest-reference.md) +- [Segmentation](24-segmentation.md) +- [Geography](27-geography.md) - [Troubleshooting](90-troubleshooting.md) diff --git a/wiki/13-configuration-reference.md b/wiki/13-configuration-reference.md index cb091a9..7e7d433 100644 --- a/wiki/13-configuration-reference.md +++ b/wiki/13-configuration-reference.md @@ -1,16 +1,21 @@ # 13 - Configuration Reference -This page is the field-by-field reference for the main ActivitySim Visualizer -config file. For a shorter orientation, start with -[11 - Configuring Your Data](11-configuring-your-data.md). The canonical -example is [`config.yaml`](../config.yaml). - -This page documents the current canonical config layout. Unknown and removed -keys fail validation and, where possible, name their canonical replacement. +This page is a field-by-field reference for the main ActivitySim Visualizer +configuration. For an introduction, read +[11 - Configuring Your Data](11-configuring-your-data.md); for the canonical +example, see [`config.yaml`](../config.yaml). + +Unknown and removed keys are rejected at the canonical top level and in the +typed sections listed in this reference. Validation depth is not uniform: +intentional free-form mappings such as `extensions.settings` accept arbitrary +project keys, and some nested implementation mappings validate their values +rather than every possible key. Use documented fields, load the configuration +in a focused test, and do not rely on an unreported nested typo being accepted +or ignored. ## Reading This Reference -Path resolution depends on the field: +The field type controls the base directory for a relative path: | Field family | Relative to | Notes | |---|---|---| @@ -18,11 +23,12 @@ Path resolution depends on the field: | `runs[*].dir` | main config directory | Raw ActivitySim output directory. | | `files.*`, `runs[*].file_map.*` | the resolved run directory | File stems may omit `.parquet` or `.csv`; Parquet is tried before CSV. | | `fallback_files.*`, `prepared_table_map.*`, `summary_table_map.*` | main config directory | Values must include `.parquet` or `.csv`. | -| main-config skim, lookup, and skimjoin override paths | main config directory | Includes `prepare.distance_skim.file`, segmentation CSVs, and `skimjoin.defaults.*`. | -| paths inside the standalone skimjoin config | standalone skimjoin config directory | See chapter 25. | +| `prepare.distance_skim.file`, `runs[*].skim_file` | the resolved run directory | The loader resolves a relative legacy distance-skim path separately for each run. | +| other main-config enrichment, lookup, and skimjoin paths | main config directory | Includes `prepare.time_periods.network_los_file`, `prepare.non_motorized_distance_skim.file`, segmentation CSVs, and `skimjoin.defaults.*`. | +| paths inside the standalone skimjoin config | standalone skimjoin config directory | See chapter 23. | | `dashboard.export.output_path` | resolved `root` | Absolute output paths remain absolute. | -Cache impact uses these labels: +The Impact columns use these terms: | Impact | Meaning | |---|---| @@ -52,8 +58,8 @@ runs: ### Prepared-Table Workflow -Use `prepared_table_map` when a run should skip raw prepare and load canonical -prepared tables directly. +Use `prepared_table_map` to load canonical prepared tables and optional skim +sidecars directly. The run does not do the raw prepare step. ```yaml pipeline: @@ -66,6 +72,8 @@ runs: persons: C:\prepared\persons.parquet tours: C:\prepared\tours.parquet trips: C:\prepared\trips.parquet + trip_hypothetical_skims: C:\prepared\trip_hypothetical_skims.parquet + tour_hypothetical_skims: C:\prepared\tour_hypothetical_skims.parquet land_use: C:\prepared\land_use.parquet ``` @@ -100,13 +108,23 @@ skimjoin: runs: - dir: C:\models\base\output label: Base + skimjoin: + enabled: false - dir: C:\models\build\output label: Build skimjoin: skim_files: - C:\build_skims\*.omx + network_los_file: C:\skims\network_los.yaml ``` +Runs without a `runs[*].skimjoin` block use the fully resolved global defaults. +Set `runs[*].skimjoin.enabled: false` to skip enrichment for one run while +leaving skimjoin enabled for the other runs. +When a run has its own override block, put every path that varies or is required +for that run in the block. Omitted skim and network paths fall back to the +selected standalone skimjoin file, not to the other global path overrides. + ## Top-Level Fields | Field | Type | Default | Impact | Purpose | @@ -125,15 +143,15 @@ runs: | `segment` | mapping | disabled | Summary, Presentation | Optional segmented summaries and dashboard segment controls. | | `weighting` | mapping | `{}` | Summary, Presentation | Declarative named weighting modes backed by prepared source columns. | | `summarize` | mapping | weighted and unweighted summaries | Summary | Summary weighting, purpose grouping, geography, and PNR mode behavior. | -| `dashboard` | mapping | live dashboard defaults | Presentation | Dashboard title, page selection, MAZ geography toggle, and export settings. | +| `dashboard` | mapping | live dashboard defaults | Presentation | Dashboard title, page selection, calculation notes, MAZ geography toggle, and export settings. | | `display` | mapping | built-in labels and colors | Presentation | Dashboard labels, category order, and run colors. | | `extensions` | mapping | `{}` | Summary, Presentation | Advanced importable weighting calculation modules and their settings. Extension code is trusted. | -| `modes` | mapping | `{}` | Presentation | Optional mode ordering used when `display.labels.mode` is absent. | +| `modes` | mapping | `{}` | Summary, Presentation | Optional mode ordering and named summary mode groups. | ## `weighting` -`weighting.modes` defines named alternatives by pointing at columns already -present in prepared household, person, or trip tables. +`weighting.modes` defines named alternatives that use columns already present +in prepared household, person, or trip tables. | Field | Type | Default | Impact | Notes | |---|---|---|---|---| @@ -143,31 +161,19 @@ present in prepared household, person, or trip tables. | `modes..columns.persons` | string | none | Summary | Person source column; propagates to trips, tours, and days unless overridden. | | `modes..columns.trips` | string | none | Summary | Trip source column; tour weight becomes the mean selected trip weight by `tour_id`. | -```yaml -weighting: - modes: - calibrated: - label: Calibrated - columns: - households: calibrated_hh_weight - persons: calibrated_person_weight - trips: calibrated_trip_weight - -summarize: - weighting_modes: [weighted, unweighted, calibrated] -``` - -Each definition needs at least one supported source table. Named columns are -validated against every prepared run. See the [weighting cookbook](43-weighting-hosting-extensions.md#worked-example-add-a-weighting-mode). +Each definition requires at least one supported source table. The visualizer +validates named columns for each prepared run. For a complete configuration +example and its propagation rules, see the +[weighting cookbook](43-weighting-hosting-extensions.md#worked-example-add-a-weighting-mode). ## `extensions` -This is the advanced path for calculations that cannot be represented by -`weighting.modes` column selection. +Use this advanced method for calculations that `weighting.modes` column +selection cannot define. | Field | Type | Default | Impact | Notes | |---|---|---|---|---| -| `modules` | list of strings | `[]` | Summary | Importable modules that define `register_weighting_modes(registry)`. Installed weighting entry points are discovered separately. | +| `modules` | list of strings | `[]` | Summary | Importable modules that define `register_weighting_modes(registry)`. The loader finds installed weighting entry points separately. | | `settings` | mapping | `{}` | Summary | Arbitrary YAML settings available to transforms as `config.extension_settings`. Included in summary cache identity. | ```yaml @@ -186,49 +192,60 @@ See [Advanced: Custom Weight Calculations](43-weighting-hosting-extensions.md#ad | Field | Type | Default | Allowed values | Impact | Notes | |---|---|---|---|---|---| -| `steps` | non-empty list of strings | `[summarize, dashboard]` | `prepare`, `skimjoin`, `segment`, `summarize`, `dashboard` | Runtime | Steps must be lowercase, unique, and valid. `skimjoin` requires `prepare`; `segment` requires `summarize`; `dashboard` must be last when present. | -| `dashboard_mode` | string | `live` | `none`, `live`, `export`, `host` | Runtime, Presentation | Controls what the dashboard step does. `host` is reserved and currently warns, then falls back to the ordinary live server; it does not publish an application. | -| `overwrite` | boolean | `false` | `true`, `false` | Runtime | Bypasses reusable prepared/summary caches for configured processor steps and writes rebuilt artifacts. Return it to `false` after a forced rebuild. | +| `steps` | non-empty list of strings | `[summarize, dashboard]` | `prepare`, `skimjoin`, `segment`, `summarize`, `dashboard` | Runtime | Steps must be lowercase, unique, and valid. Membership selects fixed runtime boundaries; non-dashboard list order does not change execution order. `skimjoin` requires `prepare`; `segment` requires `summarize`; `dashboard` must be last when present. | +| `dashboard_mode` | string | `live` | `none`, `live`, `export`, `host` | Runtime, Presentation | Controls the dashboard step. `host` writes a warning and uses the standard live server. It does not publish an application. | +| `refresh` | list of strings or `all` | `[]` | `prepare`, `skimjoin`, `summarize`, `all` | Runtime | Forces only the named stored stages to rebuild. An upstream refresh invalidates enabled downstream stages. Leave empty for standard cache-aware operation. | ```yaml pipeline: steps: [prepare, skimjoin, segment, summarize, dashboard] dashboard_mode: export - overwrite: false + refresh: [] ``` +The visualizer stores `segment` output in summary bundles. Use +`refresh: [summarize]` to rebuild segmented output. Dashboard rendering does +not have a persistent processor cache, so dashboard is not a refresh target. + ## `runs` -Each run entry describes one scenario. `label` is strongly recommended because -it becomes the display name and helps cache/debug output remain understandable. +Each run entry describes one scenario. Set `label` whenever possible; it becomes +the display name and identifies the run in cache and debug output. | Field | Type | Default | Impact | Notes | |---|---|---|---|---| -| `dir` | path string | none | Prepare | Raw ActivitySim output folder. Required unless the run is supplied by `prepared_table_map`, by `summary_table_map` alone, or both. | +| `dir` | path string | none | Prepare | Raw ActivitySim output directory. Set this field unless `prepared_table_map` or `summary_table_map` supplies the run. | | `label` | string | folder name or `run` fallback | Summary, Presentation | Dashboard and cache-facing run name. Keep stable across reruns. | | `file_map` | mapping | inherits top-level `files` | Prepare | Per-run raw file stem overrides. Cannot be combined with `prepared_table_map`. | -| `prepared_table_map` | mapping | none | Prepare, Summary | Explicit `.parquet` or `.csv` canonical prepared tables. Skips raw prepare for that run. | -| `summary_table_map` | mapping | none | Summary, Presentation | Registered summary IDs mapped to dashboard-ready `.parquet` or `.csv` files. May be used alone or override generated summaries. | -| `skimjoin` | mapping | inherits global `skimjoin` | Prepare | Per-run skimjoin `config_path`, `skim_files`, and `network_los_file` overrides. | +| `prepared_table_map` | mapping | none | Prepare, Summary | Explicit `.parquet` or `.csv` canonical prepared tables and optional skim sidecars. Skips raw prepare for that run. | +| `summary_table_map` | mapping | none | Summary, Presentation | Maps registered summary IDs to dashboard-ready `.parquet` or `.csv` files. Use it alone or to replace generated summaries. | +| `skim_file` | path string | `prepare.distance_skim.file` | Prepare, Summary | Per-run legacy distance-skim override. Relative paths resolve from this run's `dir`. | +| `skimjoin` | mapping | inherits global `skimjoin` | Prepare | Per-run skimjoin path and hypothetical-sidecar overrides. | | `hh_weight_col` | string | none | Prepare, Summary | Household source for the run's primary `weighted` mode. | | `person_weight_col` | string | none | Prepare, Summary | Person source for the run's primary `weighted` mode. | | `trip_weight_col` | string | none | Prepare, Summary | Trip source for the run's primary `weighted` mode. | +| `day_weight_col` | string or null | `day_weight` | Prepare, Summary | Day source for the run's primary `weighted` mode. Non-null source values take precedence, then missing values inherit person, household, or `1.0`. Set null to ignore a placeholder day-weight column. | -Allowed `file_map` and `prepared_table_map` table ids are: +You can use these canonical table IDs in `file_map` and `prepared_table_map`: `households`, `persons`, `day`, `tours`, `trips`, `vehicles`, `joint_tour_participants`, `land_use`. -`prepared_table_map` paths must include `.parquet` or `.csv`. Relative paths are -resolved relative to the config file. +`prepared_table_map` additionally accepts the optional skim sidecar IDs +`trip_hypothetical_skims` and `tour_hypothetical_skims`. -`summary_table_map` uses registered IDs from the summary catalog. Its paths must -also end in `.parquet` or `.csv` and are resolved relative to the config file. +Each `prepared_table_map` path must include `.parquet` or `.csv`. A relative +path starts from the configuration file directory. + +`summary_table_map` uses registered IDs from the summary catalog. Each path must +end in `.parquet` or `.csv`. A relative path starts from the configuration file +directory. ### Run Labels And Run Keys -`label` is the dashboard name. Its filesystem-safe lowercase slug is the run -key used by cache directories, manifests, and settings such as +`label` is the dashboard name. The visualizer converts it to a lowercase, +file-system-safe run key. Cache directories, manifests, and settings use this +key. One example is `prepare.vot_bins.mappings`: | Label | Run key | @@ -237,9 +254,9 @@ key used by cache directories, manifests, and settings such as | `Build Scenario` | `build-scenario` | | `2026 / Toll Test` | `2026-toll-test` | -If normalized labels collide, every colliding key receives an ordered numeric -suffix (`build-1`, `build-2`). Keep labels unique and stable: changing their -order can change those suffixes and therefore cache/mapping identity. +If normalized labels are equal, each key gets an ordered numeric suffix +(`build-1`, `build-2`). Keep labels unique and stable. If you change their +order, you can change the suffixes and the cache or mapping identity. ```yaml runs: @@ -253,9 +270,9 @@ runs: ## `files` And `fallback_files` -`files` maps logical table ids to raw ActivitySim output file stems. If the value -has no extension, the reader tries `.parquet` first, then `.csv`, inside each -run directory. +`files` maps logical table IDs to raw ActivitySim output file names. If a value +has no extension, the reader searches each run directory for `.parquet` first +and then `.csv`. | Table id | Default stem | |---|---| @@ -268,9 +285,9 @@ run directory. | `joint_tour_participants` | `final_joint_tour_participants` | | `land_use` | `final_land_use` | -`fallback_files` supports optional table ids only: `day`, `vehicles`, -`joint_tour_participants`, and `land_use`. Values must be explicit `.parquet` or -`.csv` paths. +`fallback_files` supports only these optional table IDs: `day`, `vehicles`, +`joint_tour_participants`, and `land_use`. Each value must be an explicit +`.parquet` or `.csv` path. Use `fallback_files` when multiple runs share input files. ```yaml files: @@ -301,9 +318,12 @@ zones: ## `columns` -Most `columns` values may be a string or an ordered list of candidate source -column names. The first available candidate is used. The few scalar fields -listed first are read as single names. +Alias fields can be a single string or an ordered list of possible source +names; the visualizer uses the first available column. Exactly these fields are +scalar-only column names: `ptype`, `hhsize`, `auto_ownership`, `num_workers`, +`num_adults`, and `sample_rate`. Every other field in the table below is an +alias field and accepts a string or list. A list supplied to a scalar-only +field is not an alias search and must not be used. | Field | Default | Impact | Purpose | |---|---|---|---| @@ -381,14 +401,15 @@ columns: | `validation.relationship_checks` | string or `false` | `warn` | `off`, `warn`, `error`, or `false` for `off` | Prepare | Controls relationship-check failures during prepare. | | `distance_skim.file` | path string | none | OMX path | Prepare, Summary | Optional distance skim used by prepare. | | `distance_skim.matrix` | string | `SOV_DIST__MD` | matrix name | Prepare, Summary | Matrix read from `distance_skim.file`. | +| `category_mappings` | mapping | `{}` | run, table, and column mappings | Prepare, Skimjoin, Summary | Converts run-specific categorical input values into canonical prepared columns before enrichment. | | `auto_sufficiency_basis` | string | `licensed_drivers` | `licensed_drivers`, `workers`, `adults` | Prepare, Summary | Basis for household auto-sufficiency derivation. | | `student_types` | list of mappings | `[]` | student-type definitions | Prepare, Summary | School/university enrollment definitions used by prepared fields and shadow-pricing summaries. | -| `time_periods` | mapping | built-in periods | period definitions or ActivitySim config source | Prepare, Summary | Canonical time-period labels used by prepared tours and trips. | +| `time_periods` | mapping | disabled | ActivitySim `network_los.yaml` source | Prepare, Summary | Derives canonical period labels for prepared tours and trips. | | `non_motorized_distance_skim` | mapping | disabled | configured lookup | Prepare, Summary | Optional non-motorized distance enrichment. | | `vot_bins.source_column` | string | `income_segment` | any source column | Prepare, Skimjoin | Source value used to derive VOT bins. | | `vot_bins.output_column` | string | `vot_bin` | any output column | Prepare, Skimjoin | Prepared column written for skimjoin dimensions. | | `vot_bins.fallback_value` | scalar string | none | any value | Prepare, Skimjoin | Value used when no run-specific mapping applies. | -| `vot_bins.mappings` | mapping | `{}` | run key to value mapping | Prepare, Skimjoin | Enables VOT bin derivation. Run keys are normalized from run labels. | +| `vot_bins.mappings` | mapping | `{}` | run key to value mapping | Prepare, Skimjoin | Enables VOT bin calculation. The loader normalizes run keys from run labels. | ```yaml prepare: @@ -411,23 +432,93 @@ prepare: 3: H ``` +`prepare.category_mappings` is keyed by normalized run label, table id, and +target column. Each target requires a nonempty `mapping`; `source` defaults to +the target column. Unmapped values are preserved unless +`preserve_unmapped: false` is set. Output defaults to `string`; use +`output_type: boolean` for mappings whose values are `true`, `false`, or +`null`. Prepare records unmatched source values in its diagnostics. + +```yaml +prepare: + category_mappings: + raw_survey: + tours: + tour_mode: + mapping: + 1: WALK + 2: BIKE + persons: + has_license: + source: can_drive + output_type: boolean + preserve_unmapped: false + mapping: + 1: true + 3: false +``` + +`prepare.time_periods` accepts: + +| Field | Type | Default | Notes | +|---|---|---|---| +| `network_los_file` | path string | required | ActivitySim YAML containing `skim_time_periods.periods` and `skim_time_periods.labels`. Relative paths resolve from the main config. | +| `trip_period_number_column` | string | `depart` | Prepared trip source used to write `trip_period`. | +| `tour_start_period_number_column` | string | `start` | Prepared tour source used to write `start_period`. | +| `tour_end_period_number_column` | string | `end` | Prepared tour source used to write `end_period`. | + +The period breakpoint list must contain at least two integers. The label list +must contain one less entry. If trips contain `tour_id`, `outbound`, and the +derived `trip_period`, prepare also writes `first_inbound_trip_period` for each +tour. Prepare records missing configured source columns in the diagnostics. It +does not create values for missing columns. + +`prepare.non_motorized_distance_skim` accepts: + +| Field | Type | Default | Notes | +|---|---|---|---| +| `file` | path string | required | `.csv`, `.omx`, `.h5`, or `.hdf5` lookup. Relative paths resolve from the main config. | +| `matrix` | string or null | required for OMX/HDF5; `DISTWALK` for CSV | OMX matrix name. For CSV, names the value column; a `__` prefix is stripped when present. | + +CSV lookup files must contain `OMAZ`, `DMAZ`, and the selected value column. +Prepared trips must contain `o_maz` and `d_maz`. OMX and HDF5 lookups use the +prepared `OTAZ` and `DTAZ` columns. Both methods write +`prepared_non_motorized_distance`. They record diagnostics for unresolved +lookups. + ## `skimjoin` -The main config `skimjoin` section wires the visualizer runtime to a separate -skimjoin config file. See -[25 - Skimjoin Config Reference](25-skimjoin-config-reference.md) for the +The `skimjoin` section connects the visualizer runtime to a separate skimjoin +rules file. See +[23 - Skimjoin Config Reference](23-skimjoin-config-reference.md) for the lookup-rule schema. ```text main visualizer config pipeline.steps: enables the integrated skimjoin stage - skimjoin.defaults: selects the standalone config and optional path overrides + skimjoin.defaults: selects the rules file and can supply shared data paths + runs[*].skimjoin: can skip enrichment, select another rules file, or supply run-specific paths -> standalone skimjoin config - project/activitysim/defaults/modes: defines the actual lookup rules + activitysim/defaults/dimensions/modes: defines lookup behavior + project: supplies paths only when the main config does not ``` -Merely providing `skimjoin.defaults.config_path` does not run skimjoin; -`pipeline.steps` must also contain both `prepare` and `skimjoin`. +Setting `skimjoin.defaults.config_path` does not start skimjoin; the +`pipeline.steps` list must also contain `prepare` and `skimjoin`. + +For integrated use, `project` is optional in the standalone skimjoin file. The +effective configuration needs: + +| Requirement | Where to set it | +|---|---| +| Skimjoin rules file | `skimjoin.defaults.config_path` or `runs[*].skimjoin.config_path`. | +| At least one skim file | Main-config `skim_files`, or `project.skim_files` in the selected skimjoin file. | +| `network_los.yaml` | Only when `dimensions.PERIOD.values_from_network_los` is `true`; set it in the main config or as `project.network_los_file`. | +| Prepared trip and tour input | Supplied by the integrated prepare workflow. Do not set `project.trips_table`, `project.tours_table`, or `project.output_dir` for integrated use. | + +For path resolution, override precedence, and the standalone top-level +`skim_files` caveat, see +[Choose Where Paths Live](23-skimjoin-config-reference.md#choose-where-paths-live). | Field | Type | Default | Impact | Notes | |---|---|---|---|---| @@ -437,26 +528,41 @@ Merely providing `skimjoin.defaults.config_path` does not run skimjoin; | `failure_policy` | string | `record` | Runtime, Prepare | `record` keeps a failed enrichment as diagnostics; `error` stops the run. | | `create_hypothetical_skim_tables` | boolean | `false` | Prepare | Enables configured hypothetical skim tables. | -Run-level `runs[*].skimjoin` supports `config_path`, `skim_files`, and -`network_los_file`. Enable skimjoin by including it in `pipeline.steps`; -top-level `skimjoin.enabled` and `skimjoin.config_path` are removed keys. +Run-level `runs[*].skimjoin` supports `enabled`, `config_path`, `skim_files`, +`network_los_file`, and `create_hypothetical_skim_tables`. Set `enabled: false` +to skip skimjoin for that run. When omitted, it inherits the global pipeline +setting. `enabled: true` cannot enable skimjoin unless `pipeline.steps` also +contains `skimjoin`. The run-level +`config_path` replaces the global path. A supplied run-level skim or network +path replaces the corresponding value in the selected standalone file. +`create_hypothetical_skim_tables` inherits the global value when omitted. + +The run-reload and omitted-path rules are documented in +[Choose Where Paths Live](23-skimjoin-config-reference.md#choose-where-paths-live). + +To enable skimjoin globally, add it to `pipeline.steps`. Do not use the removed +top-level `skimjoin.enabled` or `skimjoin.config_path` keys; `enabled` is only +supported under `runs[*].skimjoin` as a per-run override. Integrated skim files must resolve to `.omx`, `.csv`, `.h5`, or `.hdf5`. ## `segment` -`segment` config is canonical in user YAML. Internally it is normalized to the -segmentation runtime settings. +Use `segment` as the canonical section in user YAML. The loader converts it to +the segmentation runtime settings. The section is active only when +`pipeline.steps` contains both `segment` and `summarize`. See +[24 - Segmentation](24-segmentation.md) for the runtime flow, relationship +slicing, output paths, and dashboard behavior. | Field | Type | Default | Allowed values | Impact | Notes | |---|---|---|---|---|---| | `dashboard.segmentation_type` | string | first configured definition | configured definition name | Presentation | Selected segment type shown in dashboard/export. | | `dashboard.visibility` | string | `full_and_segments` | `full_only`, `segments_only`, `full_and_segments` | Presentation | Whether the dashboard shows full-run outputs, segmented outputs, or both. | -| `definitions` | mapping | required when segment step is enabled | path-safe lowercase names | Summary | Segment definitions. | -| `definitions.*.include_full` | boolean | `true` | `true`, `false` | Summary | Also build full-run summaries. | -| `definitions.*.persist_segmented_prepared_tables` | boolean | `false` | `true`, `false` | Prepare, Summary | Persist segment-specific prepared tables. | +| `definitions` | mapping | required when you enable the segment step | path-safe lowercase names | Summary | Segment definitions. | +| `definitions.*.include_full` | boolean | `true` | `true`, `false` | Summary | Accepted setting. The current runtime always builds one full-run analysis unit. | +| `definitions.*.persist_segmented_prepared_tables` | boolean | `false` | `true`, `false` | Prepare, Summary | Accepted setting. The current runtime keeps slices in memory and writes segmented summaries, not segmented prepared directories. | | `definitions.*.allow_overlapping` | boolean | `false` | `true`, `false` | Summary | Allows one source value to appear in multiple segments. | -| `definitions.*.on_empty_segment` | string | `warn` | `error`, `warn`, `skip` | Summary | Behavior when a segment has no rows. | +| `definitions.*.on_empty_segment` | string | `warn` | `error`, `warn`, `skip` | Summary | `error` stops, `skip` omits the unit, and `warn` keeps an empty unit. | | `definitions.*.source` | mapping | required | `prepared_column` or `csv_lookup` | Summary | Source of segment values. | | `definitions.*.segments` | list | required | list of segment mappings | Summary, Presentation | Segment ids, labels, and matched values. | @@ -482,7 +588,13 @@ segment: values: [1] ``` -`source_table` may be `hh`, `per`, `tours`, `trips`, or `land_use`. +Prepared-column source fields: + +| Field | Type | Default | Notes | +|---|---|---|---| +| `type` | string | `prepared_column` | Must be `prepared_column`. | +| `source_table` | string | auto-detect | Use `households`, `persons`, `day`, `tours`, `trips`, `vehicles`, `joint_tour_participants`, or `land_use`, or the runtime aliases `hh`, `per`, and `joint_participants`. Auto-detection requires the column to occur in exactly one table. | +| `column` | string | required | Prepared column containing the values matched by `segments[*].values`. | CSV lookup source: @@ -504,6 +616,24 @@ segment: values: [north] ``` +CSV lookup source fields: + +| Field | Type | Notes | +|---|---|---| +| `type` | string | Must be `csv_lookup`. | +| `file` | path string | CSV path relative to the main config directory. | +| `join.source_table` | string | Prepared anchor table using the names above. | +| `join.source_key_column` | string | Join key on the prepared table. | +| `join.csv_key_column` | string | Join key in the CSV. One key cannot map to different segment values. | +| `segment_value_column` | string | CSV column matched by `segments[*].values`. | + +Each segment requires a path-safe lowercase `id`, a nonblank `label`, and one +or more `values`. A path-safe name matches `[a-z0-9._-]+` and cannot start or +end with `.`, `_`, or `-`. Prepared-column values must match the prepared +column's type; CSV lookup segment values must be strings. Definitions and +segments are cached independently. The full run uses standard summary paths; segment output uses +`summary_tables//segments///`. + ## `summarize` | Field | Type | Default | Allowed values | Impact | Notes | @@ -515,12 +645,12 @@ segment: | `group_joint_tour_purposes` | boolean | `true` | `true`, `false` | Summary | Group joint tour purposes in summaries. | | `group_atwork_tour_purposes` | boolean | `true` | `true`, `false` | Summary | Group at-work tour purposes in summaries. | | `group_school_tour_purposes` | boolean | `true` | `true`, `false` | Summary | Group school tour purposes in summaries. | -| `geography.enabled` | boolean | `false` | `true`, `false` | Summary, Presentation | Enables custom geography mapping and aggregations. | -| `geography.landuse_col` | string | none | land-use column | Summary | Existing land-use geography column used for geography summaries. | -| `geography.mapping` | mapping | none | raw value to label | Summary, Presentation | Label mapping for geography values. | -| `geography.aggregations` | mapping | none | aggregation definitions | Summary, Presentation | Additional zone-to-geography lookup definitions. | +| `geography.enabled` | boolean | `false` | `true`, `false` | Prepare, Summary, Presentation | Enables the legacy mapping and named geography aggregations. Disabled definitions are ignored. | +| `geography.landuse_col` | string | none | land-use column | Prepare, Summary | Existing land-use column used to create compatibility `HGEO` and `WGEO` fields. | +| `geography.mapping` | mapping | none | raw value to label | Prepare, Summary, Presentation | Optional normalization for values from `landuse_col`. | +| `geography.aggregations` | mapping | `{}` | aggregation definitions | Prepare, Summary, Presentation | Named zone-to-geography lookups that create role-specific prepared columns. | -Each `geography.aggregations.*` entry requires: +Each `geography.aggregations.*` entry requires these fields: | Field | Type | Notes | |---|---|---| @@ -530,6 +660,10 @@ Each `geography.aggregations.*` entry requires: | `zone_id_col` | string | Required with `file`. | | `geography_col` | string | Required with `file`. | +For a complete explanation of source zones, generated columns, summary fields, +dashboard labels, cache behavior, and lookup validation, see +[27 - Geography](27-geography.md). + ```yaml summarize: weighting_modes: [weighted, unweighted] @@ -549,6 +683,8 @@ summarize: | Field | Type | Default | Allowed values | Impact | Notes | |---|---|---|---|---|---| | `title` | string | `ActivitySim Visualizer` | any string | Presentation | Dashboard title. | +| `logo` | path string | none | recognized image file | Presentation | Optional logo shown in the live dashboard and embedded in standalone HTML exports. Relative paths resolve from the config file. | +| `include_notes` | boolean | `true` | `true`, `false` | Presentation | Show expandable calculation notes beneath annotated charts and tables. | | `enable_maz_geographies` | boolean | `false` | `true`, `false` | Presentation | Enables MAZ geography options in dashboard pages that support them. | | `live.pages` | list | all/default page registry behavior | page or group ids | Presentation | Live dashboard page selection. | | `export.output_path` | path string | none | HTML path | Presentation | Relative paths resolve under `root`. | @@ -560,6 +696,12 @@ summarize: | `export.exclude_pages` | list of strings | `[]` | page ids | Presentation | Pages excluded from export. | | `export.exclude_groups` | list of strings | `[]` | group ids | Presentation | Groups excluded from export. | +`dashboard.host` is a reserved configuration block. The schema accepts +`account`, `app_id`, `title`, and `verify`. The runtime does not normalize or +use these values. `pipeline.dashboard_mode: host` writes a warning to the log +and starts the standard live server. See the hosting extension procedure in +chapter 43. + `live.pages` entries may be strings or group mappings: ```yaml @@ -573,15 +715,13 @@ dashboard: - trip_stop_distance ``` -Export page overrides are keyed by page id or by nested group/page id. Selector -keys depend on the page. Selector values may be `default`, `all`, a single -string, or a list of strings. `parts.*.enabled` can hide named export parts. +Use a page ID or a nested group and page ID as an export page override key. +Selector keys depend on the page. A selector value can be `default`, `all`, one +string, or a list of strings. Set `parts.*.enabled` to hide named export parts. -Export inherits the page set resolved by `dashboard.live.pages`. -`dashboard.export.pages` is an override mapping, not an allow-list: mentioning -one page does not remove the others. A page override with `enabled: false`, or -`exclude_pages` / `exclude_groups`, can narrow the inherited set. Export cannot -add a page that live configuration did not select. Find valid IDs in: +Export page selection, overrides, and exclusions follow the rules in +[Enabling Pages](31-dashboard-pages.md#enabling-pages). Find valid IDs in these +locations: - page and group IDs: the generated catalog in chapter 31; - selector IDs: `self.select(...)` and `self.selector(...)` calls on the page; @@ -634,11 +774,29 @@ display: - "#a00000" ``` +## `modes` + +`modes` supplies legacy mode order and summary groups. Use +`display.labels.mode.mapping` to define labels and order together. + +| Field | Type | Default | Impact | Notes | +|---|---|---|---|---| +| `order` | list of strings | none | Presentation | Raw mode order used only when `display.labels.mode` is absent. | +| `groups` | mapping of lists | none | Summary | Named mode groups included in summary cache identity. The `Auto` group explicitly selects auto modes for `auto_vmt_totals` and segmented auto-VMT summaries; without it, those summaries use built-in name matching. | + +```yaml +modes: + order: [SOV, HOV2, HOV3, WALK, BIKE, WALK_TRANSIT] + groups: + Auto: [SOV, HOV2, HOV3, TAXI, TNC_SINGLE, TNC_SHARED] +``` + ## Advanced Category Config -`summarize.category_normalization` uses the same category shape as -`display.labels`, but changes normalized values written into summary outputs. -Use it for summary-affecting normalization or grouping, not cosmetic relabeling. +`summarize.category_normalization` uses the same category format as +`display.labels`. It changes normalized values in summary output. Use it for +normalization or groups that change summaries. Do not use it only to change +display labels. ```yaml summarize: @@ -678,19 +836,19 @@ prepare: is_university: true ``` -Matching rules are deterministic: +The visualizer applies these rules in sequence: -1. When `prepare.student_types` is empty, prepare infers `School` from available +1. When `prepare.student_types` is empty, prepare gets `School` from available `ENROLLGRADEKto8`/`ENROLLGRADE9to12` columns and `University` from `COLLEGEENROLL`. -2. When a configured entry omits `person`, labels or land-use column names - containing `univ` or `college` match `is_university`; other entries match - `is_student` and exclude university students. -3. With more than two configured entries, every non-university-defaulting entry - must provide `person`; otherwise config validation fails. -4. A `person` mapping combines all supplied conditions with AND. Scalar and - list values are both accepted for `school_segment`, `SCHG`, and `pstudent`. -5. If multiple entries match one person, the first configured entry wins. +2. When an entry omits `person`, prepare examines labels and land-use column + names. Names that contain `univ` or `college` match `is_university`. Other + entries match `is_student` and exclude university students. +3. If there are more than two entries, each non-university default entry must + provide `person`. If it does not, configuration validation fails. +4. A `person` mapping combines all conditions with AND. You can use scalar or + list values for `school_segment`, `SCHG`, and `pstudent`. +5. If multiple entries match one person, the visualizer uses the first entry. For example, three school levels must select their person rows explicitly: @@ -715,5 +873,9 @@ prepare: - [11 - Configuring Your Data](11-configuring-your-data.md) - [12 - Running Workflows](12-running-workflows.md) +- [14 - Input Data Contract](14-input-data-contract.md) +- [15 - Cache And Manifest Reference](15-cache-manifest-reference.md) - [21 - Prepared Tables](21-prepared-tables.md) -- [25 - Skimjoin Config Reference](25-skimjoin-config-reference.md) +- [23 - Skimjoin Config Reference](23-skimjoin-config-reference.md) +- [24 - Segmentation](24-segmentation.md) +- [27 - Geography](27-geography.md) diff --git a/wiki/14-input-data-contract.md b/wiki/14-input-data-contract.md new file mode 100644 index 0000000..ff42edd --- /dev/null +++ b/wiki/14-input-data-contract.md @@ -0,0 +1,203 @@ +# 14 - Input Data Contract + +This chapter defines the boundary between ActivitySim output, canonical +prepared tables, summaries, and dashboard pages. Use it when you create a new +configuration, generate tables outside the visualizer, or diagnose an +unavailable summary. + +The visualizer accepts three input levels: + +```text +raw ActivitySim tables -> prepare -> canonical prepared tables -> summarize +canonical prepared tables -------------------------------> summarize +registered summary tables --------------------------------> dashboard +``` + +The level you supply determines which checks and transformations the visualizer +can perform. + +## Raw Table Inventory + +The raw reader recognizes these logical table IDs. The default file stems are +defined by `files` and can be changed globally or with `runs[*].file_map`. + +| Table ID | Default stem | Coverage role | Main keys and relationships | +|---|---|---|---| +| `households` | `final_households` | Core | One row per `household_id`. Supplies household attributes and home zone. | +| `persons` | `final_persons` | Core | One row per `person_id`; `household_id` refers to households. | +| `tours` | `final_tours` | Core | One row per `tour_id`; normally contains `person_id` and `household_id`. | +| `trips` | `final_trips` | Core | One row per `trip_id`; normally contains `tour_id`, `person_id`, and `household_id`. | +| `day` | `final_day` | Optional | Person-day or household-day rows. Uses `day_id`, `person_id`, and/or `household_id`. | +| `vehicles` | `final_vehicles` | Optional | Vehicle rows related to households by `household_id`. | +| `joint_tour_participants` | `final_joint_tour_participants` | Optional | Participation rows related by `tour_id` and `person_id`. | +| `land_use` | `final_land_use` | Optional | One row per MAZ or TAZ. Supplies employment, enrollment, parking, and geography data. | + +“Core” means the table is part of the standard household-person-tour-trip +model and is needed for broad default-page coverage. The reader does not stop +when one core file is absent. It marks that table `unavailable`, continues with +the tables it can load, and lets summary contracts identify the affected +outputs. A run is skipped only when none of the four core tables is usable. + +Optional tables can still be required by individual summaries or pages. For +example, vehicle-characteristic summaries need `vehicles`, and several +geography or parking outputs need `land_use`. + +## Raw File And Column Rules + +For a configured stem without an extension, the reader first looks for +`.parquet` and then `.csv` in the run directory. An explicit `.csv` +or `.parquet` name selects only that file. A configured `fallback_files` path +is tried after the run-local file is absent; fallbacks are supported for +`day`, `vehicles`, `joint_tour_participants`, and `land_use`. + +Raw column names are not a fixed schema. The `columns` and `zones` settings +select source names and prepare copies the first available alias into a +canonical column. These are the minimum relationship concepts for a fully +connected standard run: + +| Concept | Canonical name | Expected tables | +|---|---|---| +| Household key | `household_id` | households, persons, tours, trips; optional day and vehicles | +| Person key | `person_id` | persons, tours, trips; optional day and joint participants | +| Tour key | `tour_id` | tours, trips, joint participants | +| Trip key | `trip_id` | trips | +| Home zone | `home_zone_id` | households and/or persons | +| Work/school zone | `workplace_zone_id`, `school_zone_id` | persons | +| Origin/destination | `origin`, `destination` | tours and trips | +| Land-use zone | `MAZ`, `TAZ` | land use | + +The [configuration reference](13-configuration-reference.md#columns) lists all +configurable aliases. A missing concept does not necessarily invalidate its +whole table. It makes calculations that declare that column unavailable. + +## Canonical Prepared Contract + +Prepare preserves source columns and adds or normalizes canonical fields. The +portable contract is therefore a set of stable concepts, not one exhaustive +column list for every regional model. + +| Prepared table | Stable identifiers | Common normalized or derived fields | +|---|---|---| +| `hh` | `household_id` (`Int64`) | `home_zone_id`, `HHVEH`, `HHSIZE`, `WORKERS`, `ADULTS`, `AUTOSUFF`, `HGEO`, `finalweight` | +| `per` | `person_id`, `household_id` (`Int64`) | `person_type`, home/work/school zones and geographies, worker/student fields, work/school distance, `finalweight` | +| `day` | `day_id`, `person_id`, `household_id` (`Int64` when present) | activity pattern, date/day fields, `finalweight` | +| `tours` | `tour_id`, `person_id`, `household_id` (`Int64`) | purpose, mode, category, time, stops, zones, distance, geography, `finalweight` | +| `trips` | `trip_id`, `tour_id`, `person_id`, `household_id` (`Int64`) | purpose, mode, departure, direction, stops, zones, distance, geography, `finalweight` | +| `vehicles` | `vehicle_id`, `household_id` (`Int64`) | number, type, body, fuel, age, `finalweight` | +| `joint_participants` | `tour_id`, `person_id` (`Int64`) | participant attributes retained from the source | +| `land_use` | `MAZ`, `TAZ` (`Int64` when present) | employment/enrollment, parking, and named geography fields | + +The table-specific finalizer casts known canonical numeric fields to integer or +`Float64`, categorical fields to strings, and `finalweight` to `Float64`. +Columns not owned by the canonical contract keep their source types. See +[21 - Prepared Tables](21-prepared-tables.md) for the enrichment stages and +[26 - Summary Catalog](26-summary-catalog.md) for the exact fields required by +each summary. The standalone +`reference/processor-output-table-reference.md` documents the canonical and +processor-derived prepared fields for incorporation into user-facing output +documentation. + +## Prepared Relationship Checks + +After prepare or `prepared_table_map` loading, the runtime can check these +foreign-key relationships: + +| Source | Source key | Target | Target key | +|---|---|---|---| +| persons | `household_id` | households | `household_id` | +| day | `household_id` | households | `household_id` | +| day | `person_id` | persons | `person_id` | +| tours | `household_id` | households | `household_id` | +| tours | `person_id` | persons | `person_id` | +| trips | `household_id` | households | `household_id` | +| trips | `person_id` | persons | `person_id` | +| trips | `tour_id` | tours | `tour_id` | +| vehicles | `household_id` | households | `household_id` | +| joint participants | `person_id` | persons | `person_id` | +| joint participants | `tour_id` | tours | `tour_id` | + +A check is skipped when a table or key column is unavailable. With +`prepare.validation.relationship_checks: warn`, orphan rows produce warnings. With +`error`, they stop the workflow. Direct aggregations can still count an orphan +row, while an aggregation that joins to the parent can drop it. Fixing keys is +therefore preferable to suppressing the check. + +## Using `prepared_table_map` + +`prepared_table_map` loads CSV or Parquet files directly into `RunData`. It +does not run canonicalization, enrichment, weighting, geography mapping, or +integrated skimjoin. Supply canonical fields and types yourself. + +The accepted base keys are the eight config/file table IDs in the inventory +above. The map also accepts `trip_hypothetical_skims` and +`tour_hypothetical_skims`; their long-form contracts are documented in +[Adding A Skim Output](22-skimjoin.md#adding-a-skim-output). Omitted optional +base tables are marked `unavailable`. Omitted core tables are represented as +empty tables. Files that do not exist are `unavailable`; files that cannot be +read are `failed`. These states and their details flow into summary and page +diagnostics. + +Before using custom prepared tables, verify: + +1. IDs and foreign keys use compatible types and values. +2. Every requested summary has its required columns from chapter 26. +3. Every weighted table has the intended `finalweight`. +4. Geography and skimjoin columns already exist if the corresponding outputs + depend on them. +5. Named weighting source columns are present when configured. + +## Using `summary_table_map` + +A mapped summary file is already at the final aggregation boundary. Its key +must be a registered summary ID, and its columns, order, and Polars-compatible +types must match that summary's declared schema. + +The visualizer cannot derive prepared rows, alternate weights, or segment +membership from an aggregated summary file. Built-in weighted and unweighted +modes copy the same mapped table into both modes. Declarative or custom modes +normally reject mapped summaries unless their registered external-summary +policy explicitly permits copying. + +If a run also has raw or prepared input, mapped summary tables replace the same +generated IDs and leave other generated summaries unchanged. During +segmentation, the same mapped table overlays every analysis unit; it does not +change by segment. + +## Availability States + +Tables and summaries use four stored states: + +| State | Meaning | +|---|---| +| `available` | The source loaded or the calculation returned rows. | +| `empty` | The source or valid result contains no rows. | +| `unavailable` | A file, table, or declared prerequisite is absent. | +| `failed` | Reading or calculation raised an error under the recording policy. | + +An empty cache file uses an internal `__empty__` sentinel column so CSV and +Parquet can store an otherwise zero-column frame. The loader converts it back +to an empty `DataFrame`; user-created input tables should not use this sentinel +as application data. + +## Contract Checklist + +When adding or exchanging data across the boundary: + +1. Use config table IDs for file mappings and `RunData` names in Python + contracts. +2. Preserve unique IDs and valid relationships. +3. Materialize canonical fields before bypassing prepare. +4. Declare exact summary inputs and output schema. +5. Treat units and weighting as part of the data contract, even when the file + format cannot encode them. +6. Test unavailable, empty, and partial-run behavior, not only the complete + case. + +## Related Chapters + +- [11 - Configuring Your Data](11-configuring-your-data.md) +- [13 - Configuration Reference](13-configuration-reference.md) +- [15 - Cache And Manifest Reference](15-cache-manifest-reference.md) +- [21 - Prepared Tables](21-prepared-tables.md) +- [26 - Summary Catalog](26-summary-catalog.md) +- [90 - Troubleshooting](90-troubleshooting.md) diff --git a/wiki/15-cache-manifest-reference.md b/wiki/15-cache-manifest-reference.md new file mode 100644 index 0000000..de0072d --- /dev/null +++ b/wiki/15-cache-manifest-reference.md @@ -0,0 +1,236 @@ +# 15 - Cache And Manifest Reference + +Prepared and summary caches are stored contracts, not temporary copies of +arbitrary files. Each cache directory has a `manifest.json` that records its +schema, inputs, configuration identity, table inventory, and diagnostics. + +Do not edit a manifest by hand. Change the source data or configuration and let +the workflow rebuild the affected stage. + +## Directory Layout + +For run key `base`, the standard layout is: + +```text +/base/ + manifest.json # summary manifest + summary_tables/ + weighted/.csv + unweighted/.csv + weighted/segments///.csv + base_prepared_tables/ # present when skimjoin is enabled + manifest.json + + prepared_tables/ + manifest.json # final prepared/skimjoin manifest + + + skimjoin/ +``` + +Without skimjoin, prepare writes only `prepared_tables`. With skimjoin, +`base_prepared_tables` stores the reusable pre-skim boundary and +`prepared_tables` stores the enriched result. + +## Prepared Manifest + +The current prepared schema version is an implementation value. The loader can +read a limited set of earlier versions, but users should rely on the fields, +not a hard-coded version number. A shortened example is: + +```json +{ + "schema_version": 11, + "source": "activitysim-visualizer-prepared-cache", + "label": "Base", + "run_key": "base", + "source_run_dir": "C:\\models\\base\\output", + "prepare_config_digest": "...", + "table_format": "parquet", + "table_root": "prepared_tables", + "table_files": { + "households": "households.parquet", + "persons": "persons.parquet", + "tours": "tours.parquet", + "trips": "trips.parquet" + }, + "table_states": { + "households": "available", + "day": "unavailable" + }, + "table_diagnostics": { + "day": "Cannot find 'final_day.parquet' or 'final_day.csv' ..." + }, + "run_fingerprint": {}, + "identity": { + "raw_inputs": {}, + "prepare_config": "...", + "skimjoin_config": null, + "skim_inputs": [] + } +} +``` + +Important fields: + +| Field | Meaning | +|---|---| +| `schema_version`, `source`, `generated_at_utc` | Storage format, producer, and write time. | +| `label`, `run_key`, `source_run_dir` | Display identity, cache identity, and raw source location. | +| `config_path`, `prepare_config_digest` | Configuration source and normalized prepare identity. | +| `table_format`, `table_root`, `table_files` | How to find each prepared table. | +| `sidecar_root`, `sidecar_files` | Optional hypothetical skim sidecar files. | +| `table_states`, `table_diagnostics` | Per-table `available`, `empty`, `unavailable`, or `failed` state and reason. | +| `unavailable_tables`, `failed_tables` | Compatibility views of the same table diagnostics. | +| `source_file_map`, `run_fingerprint` | Resolved raw input mapping, file identities, skims, run weights, and run-level overrides. | +| `identity` | Compact upstream identity used by later stages. | +| `hh_weight_col`, `person_weight_col`, `trip_weight_col`, `day_weight_col` | Primary run-level source weight fields. | +| `prepare_diagnostics` | Recorded preparation warnings and relationship results. | +| `skimjoin_*` | Enabled/status/config/input identity, applied outputs, skipped rules, warning/fallback counts, failure detail, and sidecar row counts. | + +Every configured prepared table has an entry in `table_files`, including empty +or unavailable tables. Those tables are stored with the internal empty +sentinel and restored according to `table_states`. + +## Summary Manifest + +The run-level summary manifest covers the full run and every segment. A +shortened example is: + +```json +{ + "schema_version": 15, + "source": "activitysim-visualizer-summary-cache", + "label": "Base", + "run_key": "base", + "summary_config_digest": "...", + "weighting_modes": ["weighted", "unweighted"], + "summary_ids": ["population_totals", "trip_mode_by_tour_purpose_and_tour_mode"], + "summary_files": { + "population_totals": "population_totals.csv" + }, + "summary_states": { + "weighted": {"population_totals": "available"} + }, + "summary_diagnostics": {"weighted": {}}, + "summary_digests": { + "weighted": {"population_totals": "..."} + }, + "prepared_manifest_identity": {}, + "segmentation_enabled": true, + "segmentation_types": [] +} +``` + +Important fields: + +| Field | Meaning | +|---|---| +| `schema_version`, `source`, `generated_at_utc` | Storage format, producer, and write time. | +| `label`, `run_key`, `source_run_dir` | Run identity and source location. | +| `summary_config_digest` | Normalized configuration that can affect summaries. | +| `weighting_modes` | Stored mode IDs in dashboard order. | +| `summary_ids`, `summary_files` | Registered IDs and their cache filenames. | +| `empty_summaries`, `summary_states` | Per-mode empty/state inventory. | +| `unavailable_summaries`, `failed_summaries`, `summary_diagnostics` | Per-mode problem inventory and explanations. | +| `summary_digests` | Per-mode declaration/implementation identity. It allows one changed builder to rebuild without discarding unrelated tables. | +| `run_fingerprint` | Run and external-summary input identity. | +| `prepared_manifest_identity` | Exact prepared source/config identity used by the summaries. | +| `identity` | Compact upstream-prepared and summary-config identity. | +| `segmentation_enabled`, `segmentation_types` | Stored analysis-unit definitions and segment metadata. | + +Each entry in `segmentation_types` contains the definition name and source, +plus a `segments` list. Each segment records its ID, label, matched values, +source columns or CSV join, summary roots, states, diagnostics, and digests. + +## What Makes A Cache Stale + +The cache identity is intentionally stage-specific: + +| Change | Earliest affected stage | +|---|---| +| Raw file path, size, or modification time | prepare | +| Raw file mapping, run weight fields, prepare enrichment settings | prepare | +| Skimjoin rules or resolved skim inputs | skimjoin | +| Geography mapping rows | prepare, then summarize | +| Segmentation definition or values | affected summary analysis units | +| Weighting definition or summary configuration | summarize | +| One summary declaration, builder location, schema, or requirements | that summary in each affected analysis unit | +| Dashboard labels, page selection, or layout | presentation only; no processor rebuild | + +File identity uses resolved path, byte size, and nanosecond modification time. +It does not hash the full file contents. Replacing content while preserving all +three values can defeat automatic detection; use an explicit refresh in that +unusual case. + +## Read `--explain-cache` Output + +Run: + +```bash +uv run activitysim-viz --config local_config.yaml --explain-cache +``` + +The command does not load tables, execute builders, create the cache root, or +start the dashboard. It prints one plan per run: + +```text +Pipeline plan - Base + prepare REUSE + skimjoin DISABLED + summarize REBUILD - 1 analysis-unit summary tables are stale; 0 analysis units are obsolete + dashboard RUN +``` + +| Action | Meaning | +|---|---| +| `REUSE` | The stored manifest agrees with current identity and requirements. | +| `REBUILD` | The cache is missing, explicitly refreshed, stale, incompatible, or downstream of a rebuilding stage. The rest of the line gives the reason. | +| `RUN` | The non-persistent dashboard action will execute. | +| `DISABLED` | The logical step is not selected. | + +A summarize `REBUILD` decision does not always mean every summary will run. +The summarize cache can reuse compatible tables and rebuild only stale summary +IDs or analysis units. + +## Refresh Boundaries + +Prefer the narrowest repeatable refresh: + +```yaml +pipeline: + refresh: [skimjoin] +``` + +| Refresh | Rebuilds | Keeps | +|---|---|---| +| `prepare` | raw prepare, enabled skimjoin, and summaries | nothing downstream | +| `skimjoin` | final skimjoined prepared cache and summaries | `base_prepared_tables` | +| `summarize` | full and segmented summaries | final prepared cache | +| `all` | every enabled stored stage | dashboard has no stored stage to refresh | + +After a one-time diagnostic refresh, remove it or set `refresh: []` so normal +reuse resumes. + +## Safe Inspection And Recovery + +1. Run `--explain-cache` before deleting anything. +2. Read the manifest state and diagnostic fields. +3. Confirm the run key; duplicate normalized labels can add `-1`, `-2`, and so + on. +4. Use `pipeline.refresh` when the cache is valid but you intentionally need a + rebuild. +5. If a cache is corrupt, remove only the exact run/stage directory and rerun + the selected workflow. A removed cache is recoverable only by rebuilding it. + +Do not move one run's manifest into another run directory, copy tables without +their manifest, or change digests to force reuse. These actions bypass the +identity checks that protect cross-run comparisons. + +## Related Chapters + +- [12 - Running Workflows](12-running-workflows.md) +- [14 - Input Data Contract](14-input-data-contract.md) +- [21 - Prepared Tables](21-prepared-tables.md) +- [24 - Segmentation](24-segmentation.md) +- [90 - Troubleshooting](90-troubleshooting.md) diff --git a/wiki/16-dashboard-user-guide.md b/wiki/16-dashboard-user-guide.md new file mode 100644 index 0000000..5862276 --- /dev/null +++ b/wiki/16-dashboard-user-guide.md @@ -0,0 +1,58 @@ +# 16 - Dashboard User Guide + +Use this guide to choose a dashboard page and interpret its controls and +results. For general instructions about selecting runs, changing global +controls, reading partial results, and exporting the dashboard, see +[30 - Output Visualizer](30-output-visualizer.md). + +The visualizer currently registers 27 pages. Page-local controls are +data-driven, so a selector can be absent or have fewer choices when its source +summary is unavailable. Pages that are disabled by default must be enabled in +the dashboard configuration before they appear. + +## Page-By-Page Guide + +The Group column gives the navigation item that contains the page. Open that +group, then select the page shown in the Page column. A `Standalone` page is a +top-level navigation item and does not belong to a group. + +| Group | Page | What it answers | Main interpretation or controls | +|---|---|---|---| +| N/A | Overview | How large is each run, and what is its basic household/person and auto-VMT profile? | Population cards and core distributions. Difference cards use the first run as the base. The underlying population totals count people and households even though the current KPI labels say person-days and HH-days. | +| N/A | Prepared Trip Demo | How can a page aggregate disaggregate prepared trips directly? | Disabled by default and intended as a developer example. It requires prepared trips in live mode and is omitted from standalone HTML export. | +| Daily Travel | Daily Activity Pattern | What daily activity patterns and mandatory/non-mandatory tour frequencies occur by person type? | Select person type. Tour/trip rates are per person-day measures and do not become shares with the global Values control. Distribution axes currently say persons even when the underlying observations are person-days. | +| Daily Travel | Escorted Tours | How much school escorting occurs, who chauffeurs, and what are the tour-leg/stop/distance patterns? | Direction, escort category, student-count, and person-type views depend on available required and optional escort summaries. | +| N/A | Joint Travel | How common are joint-tour patterns, party sizes, compositions, and household participation? | Household size and party size provide different denominators; read each axis and note before comparing percentages. Multi-day inputs can use household-day or person-day observations, although the current chart axes still say households or people. | +| Long-Term Choices | Individual Choices | How do license holding, bicycle comfort, transit-pass ownership, and transit subsidy vary by person type? | Person-type selectors apply to individual features. Each feature can be unavailable independently. | +| Long-Term Choices | Vehicle Ownership and Type | How many vehicles do households own, and what are the modeled age, fuel, and body-type distributions? | Household-size and vehicle-characteristic views use different source tables and populations. Allocated tour vehicle outputs are on Tour Mode, not this page. | +| Long-Term Choices | Mandatory Location Choice | Where are workers and students located, how far do they travel, and how common are work-from-home/telecommute choices? | Geography, location subject, and distance views use home, work, or school roles. “All geographies” and a selected geography can use different display logic. | +| Long-Term Choices | Employment/Enrollment Match By Geography | How closely do modeled workers/students match employment/enrollment targets? | Choose geography and, for school results, student type. Residual is modeled minus target; percent error needs a nonzero target. | +| Skim Summaries | Tour Skims | What are the distributions and statistics of observed or hypothetical tour skim components? | Select skim scenario/family, direction, component, and mode from available data. Prepared tour data adds detailed views in live mode; export remains summary-based. | +| Skim Summaries | Trip Skims | What are the distributions and statistics of observed or hypothetical trip skim components? | Select skim scenario/family, component, and trip mode. Units come from each skim component and are not converted. | +| Tour Summaries | Tour Purpose | What shares or counts of tours occur by category and purpose? | Category and purpose are separate summaries. Percent mode compares distributions within each plotted total. | +| Tour Summaries | Tour Mode | How do tour modes vary by purpose and auto sufficiency, and what vehicle characteristics are allocated to auto tours? | Purpose, auto-sufficiency, occupancy, and vehicle-characteristic controls use distinct summary features. | +| Tour Summaries | Tour Time | When do tours start and end, and how long do they last? | Select tour purpose. Time bins follow prepared/configured time values. | +| Tour Summaries | Tour Distance | What are tour distance distributions and average distances by purpose and home geography? | Purpose and geography controls affect different views. Existing labels assume prepared distance is in miles. | +| Tour Summaries | Tour Stop Frequency | How many outbound, inbound, and total stops occur, and how frequent are at-work subtours? | Select purpose where offered; stop-frequency codes and derived counts are different measures. | +| Tour Summaries | Internal vs. External Tours | How often do non-mandatory tours cross the model boundary, and where are external destinations? | Select home or destination geography from available summary rows. Do not add overlapping geography totals. | +| Tour Summaries | Park-and-Ride Location | How do modeled PNR tour counts compare with lot capacity? | Select geography. Residuals require valid PNR modes, zones, and capacity data; MAZ output can be hidden by dashboard config. | +| Trip Summaries | Trip and Stop Purpose | What trip purposes occur, and what purposes occur at intermediate stops within each tour purpose? | Select tour purpose for the stop view. Trips and stops use different count fields and denominators. | +| Trip Summaries | Trip Mode | How does trip mode vary by tour purpose and tour mode? | Tour-purpose and tour-mode selectors filter the registered three-dimensional summary. | +| Trip Summaries | Trip and Stop Time | When do trips and stops depart? | Select tour purpose. Departure trip count and departure stop count are separate series. | +| Trip Summaries | Trip and Stop Distance | What are direct trip distances and stop out-of-direction distances? | Select tour purpose and distance range where available. The two charts use different prepared distance fields. | +| Trip Summaries | Parking Location | How do trips parked by zone compare with parking capacity? | Disabled by default and requires live prepared `land_use`. Current summary geography is the base parking MAZ/TAZ, not every named aggregation. | +| Validation Summaries | Traffic Validation | How closely do modeled link/count-location/screenline volumes match observations? | Select period and facility type. RMSE, RMSPE, R-squared, scatter, fit, and screenline outputs have distinct valid-data rules. | +| Validation Summaries | Transit Validation | How do boardings and transfer rates vary by operator, technology, and access mode? | Uses supplied validation summary contracts. A missing operator/technology field can remove only the affected feature. | +| Validation Summaries | VMT Validation | How does personal-auto and non-motorized VMT vary by home geography, income, household size, period, and mode? | Many selectors are dependent. Optional outside tables add external, commercial, and bicycle outputs independently. | +| Validation Summaries | Regional Validation | How do modeled district/county commute flows compare with observed matrices? | Disabled by default. Select flow type and metric: modeled, observed, difference, percent difference, or absolute percent difference. Percent difference needs nonzero observed flow. | + +For page IDs, default-enabled status, data prerequisites, and extension +contracts, see [31 - Dashboard Page Contract](31-dashboard-pages.md). + +## Related Chapters + +- [10 - Getting Started](10-getting-started.md) +- [11 - Configuring Your Data](11-configuring-your-data.md) +- [12 - Running Workflows](12-running-workflows.md) +- [30 - Output Visualizer](30-output-visualizer.md) +- [34 - HTML Export](34-html-export.md) diff --git a/wiki/17-posit-connect-cloud.md b/wiki/17-posit-connect-cloud.md new file mode 100644 index 0000000..40c3bff --- /dev/null +++ b/wiki/17-posit-connect-cloud.md @@ -0,0 +1,219 @@ +# 17 - Publish An Export With Posit Connect Cloud + +This guide publishes an ActivitySim Visualizer HTML export to Posit Connect +Cloud. Posit Connect Cloud has a free plan for public content and supports +publishing from Positron or Visual Studio Code (VS Code). + +This procedure hosts the standalone HTML export. It does not run the live +Python/Panel dashboard, and it does not use `pipeline.dashboard_mode: host` or +the reserved `dashboard.host` configuration. The published dashboard has the +same pages, selectors, and limitations as the local HTML export. + +> **Privacy:** Content on the free plan is public. Do not publish model results +> that contain confidential, licensed, or otherwise restricted information. +> Check the current [Connect Cloud plans](https://connect.posit.cloud/plans) +> before you publish because plan features and limits can change. + +## Before You Start + +You need: + +- a completed standalone HTML export; +- a free Posit Connect Cloud account; +- Positron, or VS Code with the Posit Publisher extension; and +- permission to publish the dashboard publicly. + +Positron includes Posit Publisher. In VS Code, install +[Posit Publisher from the Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=posit.publisher). + +If you have not created an export, follow +[34 - HTML Export](34-html-export.md). Open the HTML file locally and check its +pages and selectors before you publish it. The adjacent +`.diagnostics.json` file is useful for debugging, but the HTML file does +not depend on it and you do not need to publish it. + +## 1. Create A Publishing Workspace + +Create a small folder for the deployment and copy the finished HTML file into +it. For example: + +```text +activitysim_visualizer_publish/ +├── dashboard.html +└── .posit/ + └── publish/ + └── .toml +``` + +The `.posit/` directory does not exist at first. Posit Publisher creates it +when you configure the deployment. + +![VS Code publishing workspace containing dashboard.html and Posit Publisher metadata](images/publishing-workspace.png) + +*A small publishing workspace keeps the exported dashboard separate from the +development repository.* + +A separate workspace is optional, but it makes the deployment contents clear. +It also reduces the chance that you publish source data, caches, configuration +files, or other project files by mistake. If the publishing folder is inside +the repository, open that folder (not the repository root) as the IDE workspace. + +## 2. Add A Connect Cloud Credential + +1. Open **Posit Publisher** from the Activity Bar. +2. Expand **CREDENTIALS**, then select **+**. +3. Select **Posit Connect Cloud**. +4. Sign in or create an account in the browser window. +5. Confirm that the authorization code in the browser matches the code in the + IDE. +6. Select **Continue**, then **Authorize**. +7. Return to the IDE and confirm or enter a credential nickname. + +The credential now appears in Posit Publisher. If you start a deployment +without a credential, Publisher can also guide you through this process. + +## 3. Create The Deployment + +1. Open the publishing workspace in Positron or VS Code. +2. Open **Posit Publisher**. +3. Select **+** to create a deployment. +4. Select `dashboard.html` as the entrypoint. +5. Select **New deployment**. +6. Enter a title, such as `ActivitySim Visualizer`. +7. Select the Connect Cloud credential. +8. Review the generated TOML configuration. +9. Under **PROJECT FILES**, include only `dashboard.html`. +10. Select **Deploy Your Project**. + +The ActivitySim Visualizer HTML export is self-contained. A static deployment +does not need the source repository, visualizer configuration, summary caches, +Python environment, `requirements.txt`, or diagnostics sidecar. + +Publisher displays a success notification and a **View Content** button after +a successful deployment. If deployment fails, select **View Publishing Log** +and inspect the revision history linked from the log. + +## 4. Review The Publisher Configuration + +Publisher stores its deployment configuration in a TOML file under `.posit/`. +A focused configuration for this deployment looks like this: + +![Posit Publisher TOML configuration for the ActivitySim Visualizer HTML export](images/posit-publisher-config.png) + +*The generated configuration identifies the HTML entrypoint and limits the +deployment to that file.* + +```toml +"$schema" = "https://cdn.posit.co/publisher/schemas/posit-publishing-schema-v3.json" +type = "html" +entrypoint = "dashboard.html" +title = "ActivitySim Visualizer" +product_type = "connect_cloud" + +files = [ + "/dashboard.html", +] +``` + +| Setting | Meaning | +|---|---| +| `type` | The content type. Use `html` for the standalone export. | +| `entrypoint` | The file that Connect Cloud opens. | +| `title` | The title shown in Connect Cloud. | +| `product_type` | The publishing target. Use `connect_cloud`. | +| `files` | Project-relative files included in the deployment. | + +The `files` setting uses `.gitignore`-style include patterns. A leading `/` +selects a file at the publishing-workspace root. Listing only +`/dashboard.html` prevents Publisher from including unrelated files. + +Publisher owns the configuration format and can add fields as the extension +changes. Start with the generated file, then narrow its `files` list. See the +[Posit Publisher configuration reference](https://github.com/posit-dev/publisher/blob/main/docs/configuration.md) +for the current schema. + +## 5. Check And Share The Published Dashboard + +Open **View Content** and check the same items that you checked locally: + +- every intended dashboard page appears; +- page and global selectors change the displayed content; +- charts, tables, and labels render correctly; and +- no data that must remain private is present. + +Connect Cloud gives the content a public URL similar to: + +```text +https://[content-id].share.connect.posit.cloud +``` + +Use **Share** or **Standalone View** on the content page to copy the viewer +link. Standalone View removes the Connect Cloud management interface and is +usually the clearest link to give dashboard users. + +## 6. Update The Dashboard + +After you create a new ActivitySim Visualizer export: + +1. replace `dashboard.html` in the publishing workspace; +2. open Posit Publisher; +3. select the existing deployment; +4. confirm that only the intended project files are included; and +5. select **Deploy Your Project**. + +Keep the `.posit/` directory. Its TOML files identify the existing deployment. +If you lose them, Publisher cannot update that content item from the same local +configuration. You can create a new deployment, but it will be a different +content item. + +## 7. Set A Readable URL + +The default content ID is difficult to remember. To set a readable address: + +1. open the published content's administration page; +2. select **Edit settings**; +3. open the **URL** settings; +4. enter a unique custom name; and +5. save the change. + +The resulting URL follows this pattern: + +```text +https://[account-name]-[custom-name].share.connect.posit.cloud +``` + +This customizable Connect Cloud URL is available separately from paid custom +domain features. See the official +[content settings documentation](https://docs.posit.co/connect-cloud/user/manage/content_settings.html) +for current options. + +## Common Problems + +| Problem | Check | +|---|---| +| Publisher does not offer Connect Cloud | Update Posit Publisher and confirm that you selected a Connect Cloud credential. | +| Publisher selects the wrong files | Open the publishing folder as the workspace and restrict `files` to `/dashboard.html`. | +| The deployment creates a new content item | Select the existing deployment and keep its `.posit/` configuration. | +| The deployed page differs from the live dashboard | Confirm the behavior in the local HTML export. Prepared-data sections and live-only callbacks are not part of export mode. | +| A selector value is absent | Add the value to the export configuration, rebuild the HTML, and publish it again. | +| The HTML file is unexpectedly large | Review the export diagnostics sidecar and reduce exported pages, selector values, or regions. | +| Deployment fails without a clear message | Open **View Publishing Log**, then inspect the linked Connect Cloud revision. | + +For export-specific diagnosis, see +[34 - HTML Export](34-html-export.md#debugging-exports) and +[90 - Troubleshooting](90-troubleshooting.md#export-problems). + +## Official References + +- [Publish from Positron or VS Code](https://docs.posit.co/connect-cloud/user/publish/ide.html) +- [Connect Cloud plans](https://connect.posit.cloud/plans) +- [Posit Publisher configuration](https://github.com/posit-dev/publisher/blob/main/docs/configuration.md) +- [Connect Cloud content settings](https://docs.posit.co/connect-cloud/user/manage/content_settings.html) + +## Related Chapters + +- [12 - Running Workflows](12-running-workflows.md) +- [16 - Dashboard User Guide](16-dashboard-user-guide.md) +- [30 - Output Visualizer](30-output-visualizer.md) +- [34 - HTML Export](34-html-export.md) +- [43 - Weighting And Hosting Extensions](43-weighting-hosting-extensions.md) diff --git a/wiki/20-output-processor.md b/wiki/20-output-processor.md index da5be12..8c45e74 100644 --- a/wiki/20-output-processor.md +++ b/wiki/20-output-processor.md @@ -1,7 +1,7 @@ # 20 - Output Processor -The Output Processor turns ActivitySim model outputs into stable data products -for the dashboard. +The Output Processor converts ActivitySim model output to stable data for the +dashboard. ```text raw ActivitySim tables or prepared table inputs @@ -25,12 +25,12 @@ The processor is responsible for: - applying weights - adding geography and zone fields - optionally joining skim values to trips and tours -- optionally slicing outputs into configured segments +- optionally dividing output into configured segments - writing prepared and summary caches -- recording manifests and diagnostics so stale outputs can be detected +- recording manifests and diagnostics to identify stale output -The dashboard should not re-read raw ActivitySim files. It should consume -summary caches and, only for pages that explicitly ask for them, prepared tables. +The dashboard reads summary caches instead of reopening raw ActivitySim files. +Only pages that require prepared data read the prepared tables. ## Runtime Data Contract @@ -48,10 +48,12 @@ The key runtime object is `RunData` in | `joint_participants` | Joint tour participants. | | `land_use` | Prepared land-use/geography table. | | `skim_matrix` | Optional distance skim support. | +| `trip_hypothetical_skims` | Optional long-form hypothetical trip skim values. | +| `tour_hypothetical_skims` | Optional long-form hypothetical directional tour skim values. | | `skimjoin_artifacts` | Optional skimjoin manifest and QA reports. | -Summary builders and prepared-data dashboard pages should depend on this -prepared contract rather than raw model-specific table layouts. +Summary builders and prepared-data dashboard pages must use this prepared +contract. They must not use raw, model-specific table layouts. ## Processor Subsystems @@ -59,37 +61,38 @@ prepared contract rather than raw model-specific table layouts. |---|---|---| | Prepare | [21 - Prepared Tables](21-prepared-tables.md) | Normalize raw outputs and add derived fields. | | Skimjoin | [22 - Skimjoin](22-skimjoin.md) | Add skim-derived trip and tour columns. | -| Summaries | [23 - Summary Functions](23-summary-functions.md) | Build dashboard-ready tables. | -| Summary catalog | [24 - Summary Catalog](24-summary-catalog.md) | Inspect registered summary outputs. | - -The former static prepared-cache schema document recorded one -`estimation-output` dataset, including its row counts and model-specific -columns. It was not a portable runtime contract and became stale as inputs -changed. Use [Prepared Table Names and Fields](21-prepared-tables.md) for the -stable contract and inspect the manifest and table schema of the actual cache -when exact model-specific columns are needed. +| Segmentation | [24 - Segmentation](24-segmentation.md) | Slice related prepared tables and repeat summaries for configured subsets. | +| Summaries | [25 - Summary Functions](25-summary-functions.md) | Build dashboard-ready tables. | +| Summary catalog | [26 - Summary Catalog](26-summary-catalog.md) | Inspect registered summary outputs. | +| Geography | [27 - Geography](27-geography.md) | Add consistent MAZ-, TAZ-, and custom geography fields. | + +The former static prepared-cache schema described one `estimation-output` data +set, including its row counts and model-specific columns. Because those details +became incorrect when the input changed, the schema was not a portable runtime +contract. Use [Prepared Table Names and Fields](21-prepared-tables.md) for the +stable contract, and inspect the relevant cache manifest and table schema for +exact model-specific columns. ## Where Processor Output Goes -Prepared caches are reusable canonical data. Summary caches are smaller, -dashboard-ready CSVs. The summary cache is the normal dashboard input. +Prepared caches contain reusable canonical data, while summary caches contain +the smaller CSV files that serve as the standard dashboard input. -The processor also carries diagnostic state. A table or summary can be: +The processor also keeps diagnostic status. A table or summary can be: - available and populated - available but empty - unavailable because an optional input is missing - failed, with a recorded diagnostic -This is intentional. The dashboard can show partial results instead of failing -the entire workflow when one optional table or summary is unavailable. +This status information lets the dashboard show partial results instead of +stopping the entire workflow when an optional table or summary is unavailable. -“Empty” and “unavailable” are different contracts. Empty means the input and -calculation were valid but produced zero rows. Unavailable means a prerequisite -table/column was absent or a declared operation could not run. Failed means an -exception was recorded under the configured failure policy. Preserve the -availability metadata when copying `RunData`; checking only -`DataFrame.is_empty()` loses that distinction. +"Empty" and "unavailable" have different meanings. An empty result is valid but +has zero rows. An unavailable result is missing a required table or column, or +its declared operation could not run. A failed result means that the configured +failure policy recorded an exception. Preserve the availability metadata when +you copy `RunData`; checking only `DataFrame.is_empty()` loses this distinction. ### Example: Follow One Metric @@ -104,20 +107,19 @@ final_trips.csv -> page reads the table through self.data.summary(...) ``` -Each boundary has one owner. Prepare resolves source filenames and aliases; -the summary defines the aggregate; the cache validates the persisted contract; -the page handles presentation. This separation is why a page should not open a -raw file or reproduce a weighted aggregation. Chapter 44 works through this -example in code. +Each boundary has one owner: prepare resolves source file names and aliases, +the summary defines the aggregate, the cache validates the stored contract, and +the page controls presentation. A page should therefore neither open a raw file +nor repeat a weighted aggregation. Chapter 44 gives the code for this example. ## Extension Checklist -When adding new processor-visible behavior: +To add processor behavior: -1. Decide whether the new data belongs in prepared tables, skimjoin outputs, or +1. Decide whether the new data belongs in prepared tables, skimjoin output, or a summary table. 2. Add or update the smallest processor subsystem that owns that behavior. -3. Preserve stable output schemas and use typed empty fallbacks where possible. +3. Keep stable output schemas and use typed empty fallback results when possible. 4. Update dashboard page requirements if a page depends on the new output. 5. Add focused tests for the new behavior. 6. Regenerate wiki catalogs if summary declarations or page definitions changed. @@ -126,6 +128,8 @@ When adding new processor-visible behavior: - [21 - Prepared Tables](21-prepared-tables.md) - [22 - Skimjoin](22-skimjoin.md) -- [23 - Summary Functions](23-summary-functions.md) +- [24 - Segmentation](24-segmentation.md) +- [25 - Summary Functions](25-summary-functions.md) +- [27 - Geography](27-geography.md) - [44 - Summary Function Cookbook](44-summary-function-cookbook.md) - [40 - Developer Workflows](40-developer-workflows.md) diff --git a/wiki/21-prepared-tables.md b/wiki/21-prepared-tables.md index e8e70ba..30f3672 100644 --- a/wiki/21-prepared-tables.md +++ b/wiki/21-prepared-tables.md @@ -1,9 +1,14 @@ # 21 - Prepared Tables -Prepared tables are the processor's canonical form of ActivitySim output. They -hide raw file naming differences and expose stable fields for summaries and +Prepared tables are the canonical form of ActivitySim output. They remove +differences in raw file names and provide stable fields for summaries and dashboard pages. +[14 - Input Data Contract](14-input-data-contract.md) defines the exact input +inventory, canonical identifiers and types, relationship checks, availability +states, and requirements for bypassing prepare. This chapter explains how the +processor creates and extends that contract. + ## Prepare Data Flow ```text @@ -13,9 +18,10 @@ reader -> zone context -> household/person enrichment -> student enrollment - -> day and vehicle preparation + -> day preparation -> tour enrichment -> trip enrichment + -> vehicle preparation -> non-motorized distance and time-period enrichment -> VOT bins -> final casts @@ -45,7 +51,7 @@ with domain boundaries in `processor/prepare/enrichment/domains.py`. ## Prepared Table Names -Runtime table names are defined in `processor.models.PreparedTableName`: +`processor.models.PreparedTableName` defines the runtime table names: | Config/file table ID | `RunData`/summary-contract name | Meaning | |---|---|---| @@ -59,65 +65,135 @@ Runtime table names are defined in `processor.models.PreparedTableName`: | `land_use` | `land_use` | Land use and geography lookup data. | | no file-map ID | `skim` | Optional `skim_matrix` support exposed as a special prepared requirement. | -Use config/file IDs in `files`, `file_map`, and `prepared_table_map`. Use the -runtime names in `RunData` access and `@summary(required_columns=...)`; for -example, `run.per` and `required_columns={"per": ("person_type",)}`. +Use configuration and file IDs in `files`, `file_map`, and +`prepared_table_map`. The prepared map additionally accepts +`trip_hypothetical_skims` and `tour_hypothetical_skims`; these load the +matching `RunData` sidecar attributes and are not base prepared-table names for +summary requirements. Use runtime names to access `RunData` and in +`@summary(required_columns=...)`. Examples are `run.per` and +`required_columns={"per": ("person_type",)}`. ## Common Prepared Fields -The exact schema can differ by model and optional inputs, but summaries commonly -rely on: +The exact schema can differ for each model and optional input. Summaries +frequently use these fields: - canonical IDs: `household_id`, `person_id`, `tour_id`, `trip_id` - purpose and mode fields: `tour_purpose`, `trip_purpose`, `tour_mode`, `trip_mode` - time fields: `start_hour`, `end_hour`, `depart_hour` - stop fields: `num_ob_stops`, `num_ib_stops`, `num_tot_stops`, `stops` -- distance/geography fields: `SKIMDIST`, `OTAZ`, `DTAZ`, `HGEO`, `WGEO` +- distance/geography fields: `tour_distance`, `SKIMDIST`, `OTAZ`, `DTAZ`, `HGEO`, `WGEO` - household/person aliases: `HHVEH`, `HHSIZE`, `AUTOSUFF`, `NUMBER_HH` - aggregation weight: `finalweight` -Use the prepared field when it exists rather than probing raw names in a summary +Use the prepared field when it exists. Do not search for raw names in a summary or page. -This list is orientation, not a guarantee that every table has every field. -For a specific summary, the generated catalog in chapter 24 is the authoritative -list of required prepared columns. At runtime, `@summary` prerequisites and -prepared-table availability metadata determine whether a calculation can run. +For tours, `tour_distance` preserves a supplied `tour_distance` value and fills +only its missing values from `SKIMDIST`. +`SKIMDIST` remains the configured network-skim distance used by summaries that +specifically require skim-based travel. Generic `distance_miles` is not +automatically treated as a primary-destination tour distance because it may +describe the full traveled tour. The Raw Statewide Survey normalizer is an +explicit exception: it prepares that survey's reported linked-tour +`distance_miles` as `SKIMDIST` because no scenario skim join is available. +Existing tour-distance comparison summaries use `SKIMDIST` explicitly rather +than switching metrics when `tour_distance` happens to be present. Prepared +caches created before schema version 11 must be refreshed for the current +weight, endpoint, duration, and joint-composition preparation rules. + +This introductory list does not imply that every table has every field. For a +specific summary, the generated catalog in chapter 26 lists the declared +mechanical prerequisites. A builder can also resolve alternative column names +or apply conditional requirements that the declaration does not express. At +runtime, `@summary` requirements, builder checks, and prepared-table +availability metadata determine whether a calculation can run. + +The standalone `reference/processor-output-table-reference.md` provides the +user-facing field dictionary for canonical and processor-derived prepared +outputs, summary outputs, and optional hypothetical-skim sidecars. + +### Schema 11 Tour Rules + +Current tour and trip preparation applies these canonical rules: + +- When tours already contain `origin`, a home-based tour uses its household + `home_zone_id` there when that value is available. An `atwork` or `at_work` + tour instead uses its parent tour's `destination` when the parent identifiers + resolve. Otherwise the supplied tour origin remains the fallback. This stage + does not create `origin` when that column is absent. +- When trips contain `tour_id`, `origin`, `destination`, `outbound`, and + `trip_num`, and tours contain `tour_id` and `origin`, the first outbound trip + begins at the prepared tour origin and the last inbound trip ends there. + Other trip endpoints are left unchanged. +- When `start_hour` and `end_hour` are available, `tourdur` is the inclusive + difference, `end_hour - start_hour + 1`, clipped to 1 through 48 periods. +- Missing joint-tour `composition` is derived as `adults`, `children`, or + `mixed` from unique participant records. Person types 1 through 5 are adults + and 6 through 8 are children; age 18 is the fallback boundary. The age + fallback accepts any numeric value from 0 through 994 and does not detect age + category codes, so use it only when the source stores age in years. A + supplied composition takes precedence. +- `NUMBER_HH` uses the joint-participant row count when available, then + `number_of_participants`, then a supplied `NUMBER_HH`, and finally 1. + +### Raw Statewide Survey Adapter + +Prepare recognizes the included raw statewide-survey layout by its columns, +not by the run label. The adapter runs only when all of these signature fields +are present: + +| Raw table | Required signature fields | +|---|---| +| households | `num_people`, `num_vehicles` | +| persons | `can_drive`, `person_type` | +| tours | `tour_start_hour`, `tour_start_minute`, `tour_end_hour`, `tour_end_minute`, `tour_mode`, `tour_purpose`, `duration_minutes`, `distance_miles` | +| trips | `linked_trip_mode`, `d_purpose_category`, `depart_hour`, `depart_minute`, `distance_miles` | + +For that layout, prepare maps household size and vehicle count, converts clock +times to the 48 half-hour model periods that start at 3:00 a.m., writes tour +`distance_miles` to `SKIMDIST`, and writes trip `distance_miles` to `od_dist` +and `prepared_non_motorized_distance`. When present, +`joint_num_participants` becomes `number_of_participants`. + +The adapter does not define the survey's category meanings. Use +`prepare.category_mappings`, keyed by normalized run label, to map license, +mode, purpose, category, or other raw codes into canonical prepared values. ## Inspecting An Exact Prepared Schema -There is intentionally no repository-wide dump of every column from one sample -prepared cache. Raw model extensions and optional inputs make such a snapshot -model-specific and quickly stale. +The repository does not treat the columns in one sample prepared cache as a +fixed schema. Raw model extensions, optional inputs, and other input changes +make that list model-specific. -For the cache you are actually using: +To inspect a cache: -1. Read the run's `manifest.json` to find the prepared-table files and recorded - availability state. -2. Inspect the Parquet or CSV schema for the relevant table. +1. Read the run's `manifest.json`. Find the prepared-table files and the + recorded availability status. +2. Examine the Parquet or CSV schema for the relevant table. 3. Use `processor.models.RunData` names at runtime and the file/config names in [Prepared Table Names](#prepared-table-names). -4. Use the generated [Summary Catalog](24-summary-catalog.md) to find the exact - prepared columns required by each registered summary. +4. Use the generated [Summary Catalog](26-summary-catalog.md) to find each + summary's declared prerequisite columns, then read its description for + builder-level alternatives or conditional requirements. -Stable additions belong in the owning prepare enrichment module and should be -covered by a prepare test. A row count or a column found only in one regional -model output is evidence about that dataset, not part of the visualizer's -portable contract. +Add stable fields to the relevant prepare enrichment module, with a prepare +test for each field. A row count or column found in only one regional model +describes that data set, not the portable visualizer contract. ## Adding A Prepared Column -For an end-to-end worked example, see +For a complete example, see [Add A Column To An Existing Prepared Table](41-data-extension-cookbook.md#worked-example-add-a-column-to-an-existing-prepared-table). -Use this path when many summaries/pages need the same derived field or when the -field is part of canonical model-output normalization. +Use this approach when many summaries or pages need the same derived field, or +when the field is part of canonical model-output normalization. Checklist: 1. Choose the owning enrichment module. -2. Add the Polars expression or transformation in the appropriate stage. -3. Keep missing source columns graceful when the input is optional. +2. Add the Polars expression or transformation in the relevant stage. +3. If the input is optional, keep the table usable when source columns are missing. 4. Add final type/cast behavior if the field must be stable. 5. Add or update tests that prepare a minimal run and assert the new column. 6. If a summary depends on the column, add it to that summary's contract @@ -133,12 +209,12 @@ if "source_column" in state.trips.columns: ) ``` -Do not add page-only formatting columns to prepared tables. Prefer page helpers -or summary output columns for presentation concerns. +Do not add page formatting columns to prepared tables. Use page helpers or +summary output columns for presentation. ## Using Prepared Tables As Inputs -`prepared_table_map` lets a config bypass raw prepare for a run: +Use `prepared_table_map` to omit raw prepare for a run: ```yaml runs: @@ -151,15 +227,19 @@ runs: land_use: C:\prepared\land_use.parquet ``` -This path assumes the supplied tables already match the prepared contract. +The supplied tables must agree with the prepared contract. -Adding a new prepared table type is a larger change covering config, `RunData`, -reader, availability, cache IO, pruning, and possibly segmentation. Follow the -[complete worked example](41-data-extension-cookbook.md#worked-example-add-a-prepared-table). +A new prepared table type changes the configuration, `RunData`, reader, +availability, cache I/O, and pruning. It can also change segmentation. Follow +the [complete example](41-data-extension-cookbook.md#worked-example-add-a-prepared-table). ## Related Chapters - [11 - Configuring Your Data](11-configuring-your-data.md#already-prepared-tables) -- [23 - Summary Functions](23-summary-functions.md) -- [31 - Dashboard Pages](31-dashboard-pages.md) +- [14 - Input Data Contract](14-input-data-contract.md) +- [15 - Cache And Manifest Reference](15-cache-manifest-reference.md) +- [25 - Summary Functions](25-summary-functions.md) +- [24 - Segmentation](24-segmentation.md) +- [27 - Geography](27-geography.md) +- [31 - Dashboard Page Contract](31-dashboard-pages.md) - [01 - Architecture](01-architecture.md) diff --git a/wiki/22-skimjoin.md b/wiki/22-skimjoin.md index dda6607..961501e 100644 --- a/wiki/22-skimjoin.md +++ b/wiki/22-skimjoin.md @@ -1,26 +1,130 @@ # 22 - Skimjoin -Skimjoin enriches prepared trips and tours with skim-derived columns. It runs as -an optional late-prepare step after raw outputs have been normalized. +Skimjoin adds skim-derived columns to prepared trips and tours. This optional +final part of prepare runs after the raw output has been normalized. -Use skimjoin when summaries or dashboard pages need values from OMX skims or -sidecar lookup files, such as time, cost, distance, walk access, or composed -tour-level attributes. +Use skimjoin when summaries or dashboard pages require values from OMX skims or +sidecar lookup files. Examples are time, cost, distance, walk access, and direct +trip or tour attributes. -For full field-by-field skimjoin config options, lookup-rule grammar, defaults, -and examples, see -[25 - Skimjoin Config Reference](25-skimjoin-config-reference.md). +For each skimjoin field, lookup rule, default, and example, see +[23 - Skimjoin Config Reference](23-skimjoin-config-reference.md). -Two YAML files participate in integrated use: +Integrated skimjoin uses two YAML files: -- the **main visualizer config** enables the stage with `pipeline.steps` and - points at files through `skimjoin.defaults` or per-run overrides; and -- the **standalone skimjoin config** defines `project`, `activitysim`, - dimensions, mode/component lookup rules, fallbacks, and tour aggregation. +- the **main visualizer config** enables the step with `pipeline.steps` and + selects the rules file. It can also supply shared or run-specific data paths. +- the **skimjoin rules file** defines prepared column names, defaults, + dimensions, mode/component lookup rules, and fallbacks. Its optional + `project` block supplies data paths for the standalone CLI or as integrated + defaults. -Paths in the first file resolve from the main config; paths owned by the second -resolve from the standalone skimjoin config. Supplying a config path alone does -not enable the stage—`pipeline.steps` must contain `prepare` and `skimjoin`. +Paths in the first file start from the main configuration file, while paths in +the second start from the standalone skimjoin configuration file. Providing a +configuration path does not enable the step; `pipeline.steps` must contain both +`prepare` and `skimjoin`. + +## Where Path Settings Belong + +For integrated use, the main config can own the skim paths while the skimjoin +file contains only reusable lookup rules. This is useful when several model +runs share rules but use different skim files. + +Main visualizer config: + +```yaml +pipeline: + steps: [prepare, skimjoin, summarize, dashboard] + +skimjoin: + defaults: + config_path: configs\skimjoin_rules.yaml + skim_files: + - skims\shared\*.omx + network_los_file: skims\shared\network_los.yaml + +runs: + - dir: C:\models\base\output + label: Base + skimjoin: + enabled: false + - dir: C:\models\build\output + label: Build + skimjoin: + skim_files: + - skims\build\*.omx + network_los_file: skims\build\network_los.yaml +``` + +`configs\skimjoin_rules.yaml`: + +```yaml +activitysim: + trip_mode_column: trip_mode + tour_mode_column: tour_mode + trip_id_column: trip_id + tour_id_column: tour_id + outbound_column: outbound + +defaults: + origin: OTAZ + destination: DTAZ + output_prefix: skim_ + +dimensions: + PERIOD: + source_columns: + trip_source_column: depart + outbound_tour_source_column: start + inbound_tour_source_column: first_inbound_trip_depart + values_from_network_los: true + +modes: + SOV: + time: SOV_TIME__{PERIOD} + distance: SOV_DIST__{PERIOD} +``` + +In this pattern, `project` is not required. The integrated prepare workflow +already supplies trip and tour tables, and the main config supplies the skims. + +Use `project` when the skimjoin file must be self-contained, especially for the +standalone CLI: + +```yaml +project: + skim_files: + - C:\skims\*.omx + network_los_file: C:\skims\network_los.yaml + trips_table: C:\prepared\trips.parquet + tours_table: C:\prepared\tours.parquet + output_dir: C:\skimjoin_output +``` + +For integrated use, only `project.skim_files` and, when needed, +`project.network_los_file` are relevant. `project.trips_table`, +`project.tours_table`, and `project.output_dir` belong to the standalone CLI. + +The effective integrated settings follow these rules: + +1. `runs[*].skimjoin.config_path` replaces `skimjoin.defaults.config_path` for + that run. +2. Main-config `skim_files` and `network_los_file` replace the corresponding + `project` values in the selected rules file. +3. `runs[*].skimjoin.enabled: false` skips enrichment for that run. When + omitted, the run inherits the global pipeline setting; `true` cannot enable + skimjoin when the global pipeline omits the step. +4. A run with no override block uses the fully resolved global defaults. +5. When a run override causes the rules file to be reloaded, any omitted skim + or network path comes from that file. Repeat a required global path in the + run block if the rules file does not contain it. +6. `network_los_file` is required only when + `dimensions.PERIOD.values_from_network_los` is `true`. + +For predictable overrides, keep integrated skim paths either in `project` or +in the main config, not in the skimjoin file's top-level `skim_files` field. +The top-level form is supported but takes precedence during config promotion +and can prevent a main-config skim override from replacing it. ## Runtime Placement @@ -36,9 +140,73 @@ prepare raw outputs The runtime adapter is [`processor/skimjoin/pipeline.py`](../processor/skimjoin/pipeline.py). -## Config Anatomy +## How The Implementation Works -A skimjoin config describes: +Integrated skimjoin has six main stages: + +1. **Resolve one config per run.** The runtime selects the global or run-level + rules file, applies path overrides, resolves paths, and validates the typed + schema. +2. **Normalize rules.** Mode, segment, component, dimension, target-table, + missing-data, and fallback settings become ordered trip and tour lookup + rules. The standalone strict validator can report output collisions and + invalid fallback chains before annotation. +3. **Inventory skim inputs.** The runtime scans OMX, HDF5, and CSV inputs and + records matrix names, qualified source names, shapes, lookup types, and key + columns. CSV schema is inferred lazily, and row count is collected with the + streaming engine. Duplicate file-qualified references are invalid. +4. **Select rows for each rule.** A rule matches the configured trip or tour + mode, then applies `when`, `segment_on`, target, and dimension conditions. + Missing source columns can make a rule unusable and appear in diagnostics. +5. **Resolve and execute lookups.** Dimension values fill matrix-name + placeholders. OD rules read a selected OMX/HDF5 matrix or CSV OD table; key + rules read keyed CSV sidecars. Sentinel values and failed lookups remain + unresolved for a later fallback or the missing-data report. +6. **Package enriched data and diagnostics.** Successful output columns replace + the prepared `RunData.trips` and `RunData.tours` tables. The runtime stores a + manifest, lookup reports, and optional hypothetical sidecars in the final + prepared cache. + +Fallback rules run in order only for rows that still lack a valid value. The +standalone strict validator rejects overlapping outputs unless every affected +rule uses `combine: sum`. Integrated loading does not currently run that +table-aware collision check; overlapping `replace` rules can proceed and keep +the first resolved value. + +Tour lookups use two directional contexts derived from each prepared tour. The +inbound context swaps origin and destination fields, and dimension settings can +name separate outbound and inbound source columns. This produces direct tour +lookup outputs with `_outbound` and `_inbound` suffixes. + +## CSV Join And Memory Behavior + +Before integrated annotation starts, skimjoin expands the configured matrix +templates against the CSV inventory and plans only the referenced numeric value +columns. It also collects one union of non-null origin/destination pairs from +every configured trip OD column pair present in prepared trips and from both +directional tour OD column pairs present in prepared tours. Each OD CSV is then +scanned lazily, semi-joined to that demand union, grouped, and collected with +the streaming engine. Polars frame joins apply the cached values to lookup work +rows, and the same skim store is reused for trip, tour, and +hypothetical-sidecar lookups in that run. + +This optimization has explicit limits. A keyed CSV has no OD pair by which to +filter demand, so skimjoin reads every non-null key while still selecting only +the planned value columns. OMX, HDF5, and H5 inputs load the complete selected +two-dimensional dataset on first use and cache it for later lookups. + +CSV key, origin, and destination identifiers must be numeric because both +source and lookup identifiers are cast to `Float64` for joins. When a CSV +contains duplicate rows for one key or OD pair, each value column uses the last +non-null value in source order. A pair with no usable value follows the +fallback chain and remains unresolved; observed annotation records it, while +hypothetical-sidecar lookup deliberately suppresses reports. See +[Missing Policy Execution](23-skimjoin-config-reference.md#missing-policy-execution) +for the current policy limitation. + +## Configuration sections + +A skimjoin configuration contains these sections: | Section | Purpose | |---|---| @@ -47,47 +215,104 @@ A skimjoin config describes: | `defaults` | Default origin, destination, output prefix, and missing-data policies. | | `zone_mapping` | Optional zone lookup behavior. | | `dimensions` | Time period or other dimensions used to resolve matrix names. | +| `ignore_modes` | Trip modes allowed to have no lookup rules. | | `modes` | Mode-specific lookup rules. | -| `tour_aggregation` | How trip skim values roll up to tours. | +| `tour_aggregation` | Accepted settings that current execution does not use. | -Per-run overrides in the main visualizer config can change selected skim files, -`network_los_file`, or the whole skimjoin config path. +The optional `project` section contains paths, not lookup behavior. Main-config +defaults or run overrides can supply the integrated skim paths instead. ## Adding A Skim Output -Start with the [Basic OD Lookup](25-skimjoin-config-reference.md#basic-od-lookup) -for a complete mode rule, then add dimensions, fallbacks, or tour aggregation -only when the new output requires them. +Start with the [Basic OD Lookup](23-skimjoin-config-reference.md#basic-od-lookup) +for a complete mode rule. Add dimensions or fallback rules only when the new +output requires them. Checklist: -1. Confirm the prepared trips/tours contain the source columns needed for lookup. +1. Make sure prepared trips or tours contain the required lookup columns. 2. Add or update a lookup rule in the skimjoin config. -3. Choose the output name and keep the `skim_` prefix convention unless there is - a strong reason not to. -4. Set missing matrix and missing OD policies deliberately. -5. Add fallback lookup rules only when a real fallback is meaningful. -6. If tours need the value, configure tour aggregation or directional outputs. -7. Add/update a summary in `processor/summarize/summaries/skimjoin.py` if the +3. Select an output name. Use the `skim_` prefix unless the interface requires a different prefix. +4. Review the missing-matrix and missing-OD settings and the current + [execution limitation](23-skimjoin-config-reference.md#missing-policy-execution). +5. Add fallback lookup rules only when a valid fallback value is available. +6. Set `apply_to` when the component belongs only on trips or tours. +7. Add or update a summary in `processor/summarize/summaries/skimjoin.py` if the dashboard needs aggregate reporting. 8. Regenerate wiki catalogs if summary declarations or dashboard requirements changed. -Set `skimjoin.create_hypothetical_skim_tables: true` (globally or in a run -override) when the configured lookups should also produce hypothetical skim -sidecar tables. This is opt-in because it adds output work and artifacts. +Set `skimjoin.create_hypothetical_skim_tables: true` globally or in a run +override to create hypothetical skim sidecar tables. The default is `false` +because this option creates more output and artifacts. + +Hypothetical sidecars rerun each configured mode's lookup rules against every +eligible observed row. They do not change the observed trip or tour mode and +do not replace the annotated prepared tables. They provide long-form values +for comparisons such as “what would this trip's auto time be under each +configured mode?” Sidecars process one hypothetical mode at a time and reuse +the observed annotation's skim cache. + +| Trip sidecar field | Type | Meaning | +|---|---|---| +| `trip_id` | `Int64` | Prepared trip identifier. | +| `observed_mode` | string | Original configured trip mode. | +| `hypothetical_mode` | string | Mode whose rules produced the value. | +| `component` | string | Skim output column name. | +| `value` | `Float64` | Looked-up component value, or null. | +| `finalweight` | `Float64` | Prepared trip weight. | + +The tour sidecar has the same structure with `tour_id` and one additional +`direction` field. `direction` is `outbound` or `inbound` when the component +ends with the corresponding suffix; it is null for unsuffixed outputs. +Sidecars are empty unless the prepared source contains the configured ID and +mode columns plus `finalweight`. Those fields are necessary but do not +guarantee rows: a component enters the sidecar only when that hypothetical +mode resolves it for at least one source row. Other rows for an included +component can be null, and a sidecar can remain empty when no component +resolves. + +## Standalone Skimjoin CLI + +The integrated pipeline is the standard approach. The standalone command-line +interface is useful for inspecting or validating a skimjoin configuration, or +for creating annotated tables without the full visualizer: + +```bash +uv run python -m processor.skimjoin.cli COMMAND --config skimjoin.yaml +``` + +| Command | Additional flags | Output | +|---|---|---| +| `inventory` | `--preview` | Writes `skim_inventory.csv` and `inventory_debug.log` under `project.output_dir`. Preview also writes trip/tour column inventories and ActivitySim value counts when the configured tables are available. | +| `validate` | none | Strictly validates config, inventory, and configured ActivitySim tables; writes `config_normalized.yaml` and `validation_report.txt`. Returns exit code 1 and writes a failure report when validation fails. | +| `annotate-trips` | `--out PATH`, `--preview` | Writes annotated trips plus validation, lookup-summary, and missing-lookup artifacts. The default table is `/trips_with_skims.parquet`. | +| `annotate-tours` | `--out PATH`, `--preview` | Writes annotated tours and lookup diagnostics. The default table is `/tours_with_skims.parquet`. | +| `run` | `--out-trips PATH`, `--out-tours PATH`, `--preview` | Executes both annotations and writes their validation and QA reports. Uses the two file names above by default. | + +Each command requires `--config`; output flags are optional only when +`project.output_dir` is configured. Standalone input tables come from +`activitysim.trips_table` and `activitysim.tours_table`. Chapter 23 describes +the legacy `project.trips_table` and `project.tours_table` fallback. Input and +output tables must be CSV or Parquet. For annotation commands, `--preview` adds +a short output-column inventory but does not limit rows or prevent writes. ## Debugging Skimjoin -Start with the skimjoin artifacts on the prepared run: +First, examine these skimjoin artifacts for the prepared run: - `skim_lookup_summary` - `missing_lookup_report` - `fallback_lookup_report` - `skipped_rule_report` -- `tour_aggregation_summary` - `failure_report` +Integrated artifacts are stored under +`//prepared_tables/skimjoin/`. The prepared manifest records the +status, resolved rules/input identity, applied outputs, skipped rules, +warning/fallback counts, failure detail, and hypothetical sidecar row counts. +Chapter 23 gives the exact report schemas and concrete skim file layouts. + Common causes: | Symptom | Check | @@ -95,24 +320,38 @@ Common causes: | No skim columns appear | `pipeline.steps`, resolved config path, run overrides, and skim file glob resolution. | | Rule skipped | Source mode, `when` clause, ignored modes, and required dimensions. | | Missing matrix | Matrix naming pattern, dimensions, network LOS periods, and OMX contents. | -| Missing OD values | Origin/destination columns, zone mapping, sentinel values, and missing OD policy. | -| Tours missing values | Tour aggregation config and outbound/inbound source columns. | +| Missing OD values | Origin/destination columns, zone mapping, and sentinel values. Current annotation records unresolved rows regardless of the configured missing-OD policy. | +| Tours missing values | `apply_to`, tour mode, outbound/inbound source columns, dimensions, and OD columns. | + +With `failure_policy: record`, an integrated failure keeps the original +prepared trips and tours, writes empty skim sidecars, and records a +`failure_report`. With `failure_policy: error`, the exception stops the run. + +The prepared-cache identity includes the normalized skimjoin rules and resolved +skim inputs. A changed rules file or skim file invalidates skimjoin and later +summaries without requiring raw preparation to run again. Use +`refresh: [skimjoin]` to force that boundary while retaining +`base_prepared_tables`. ## Where To Change Code | Task | Start here | |---|---| | Config shape or validation | `processor/skimjoin/config/schema.py` | +| Main/run override resolution | `runtime/config/normalize_skimjoin.py` | | Config normalization | `processor/skimjoin/config/normalize.py` | +| Skim inventory | `processor/skimjoin/inventory.py` | +| Integrated CSV OD demand planning | `processor/skimjoin/csv_demand.py` | | Skim store behavior | `processor/skimjoin/skimstore/` | | Trip annotation | `processor/skimjoin/annotate/trips.py` | | Tour annotation | `processor/skimjoin/annotate/tours.py` | | Runtime reports | `processor/skimjoin/runtime_reports.py` | +| Integrated execution | `processor/skimjoin/runtime_execution.py` | | Skim summary tables | `processor/summarize/summaries/skimjoin.py` | ## Related Chapters - [13 - Configuration Reference](13-configuration-reference.md#skimjoin) -- [25 - Skimjoin Config Reference](25-skimjoin-config-reference.md) -- [23 - Summary Functions](23-summary-functions.md) +- [23 - Skimjoin Config Reference](23-skimjoin-config-reference.md) +- [25 - Summary Functions](25-summary-functions.md) - [90 - Troubleshooting](90-troubleshooting.md) diff --git a/wiki/23-skimjoin-config-reference.md b/wiki/23-skimjoin-config-reference.md new file mode 100644 index 0000000..f6e537b --- /dev/null +++ b/wiki/23-skimjoin-config-reference.md @@ -0,0 +1,758 @@ +# 23 - Skimjoin Config Reference + +This page is a field-by-field reference for the standalone skimjoin +configuration used by the main visualizer. For a workflow introduction, read +[22 - Skimjoin](22-skimjoin.md). For a complete self-contained example that +also supports the standalone CLI, see +[`example_skimjoin_config.yaml`](../example_skimjoin_config.yaml). + +The skimjoin configuration answers four questions: + +1. Which skim files and optional `network_los.yaml` must skimjoin use? +2. Which prepared trip and tour columns supply modes, IDs, dimensions, and OD + lookup columns? +3. Which matrix or sidecar table must skimjoin read for each mode and component? +4. Which missing-data settings are inherited, and what current execution + limitation applies? + +## Choose Where Paths Live + +The standalone skimjoin file always defines lookup behavior. Its `project` +section is optional for integrated visualizer use because the main config can +supply the data paths. + +| Use case | Put paths here | +|---|---| +| Integrated workflow with shared paths | `skimjoin.defaults` in the main visualizer config. | +| Integrated workflow with paths that differ by run | `runs[*].skimjoin` in the main visualizer config. | +| Self-contained skimjoin file or standalone CLI | `project` in the skimjoin config. | + +For integrated use, the effective config must have at least one skim file. A +`network_los_file` is required only when +`dimensions.PERIOD.values_from_network_los` is `true`. The integrated workflow +supplies prepared trip and tour tables, so it does not need +`project.trips_table`, `project.tours_table`, or `project.output_dir`. + +Main visualizer config with shared paths: + +```yaml +pipeline: + steps: [prepare, skimjoin, summarize, dashboard] + +skimjoin: + defaults: + config_path: configs\skimjoin_rules.yaml + skim_files: + - skims\*.omx + network_los_file: skims\network_los.yaml +``` + +The referenced rules file can then omit `project`: + +```yaml +activitysim: + trip_mode_column: trip_mode + tour_mode_column: tour_mode + trip_id_column: trip_id + tour_id_column: tour_id + outbound_column: outbound + +defaults: + origin: OTAZ + destination: DTAZ + +modes: + SOV: + time: SOV_TIME + distance: SOV_DIST +``` + +Main-config paths resolve from the main config directory. Paths in `project` +resolve from the skimjoin file's directory. + +Main-config skim and network paths replace the corresponding `project` values. +Avoid the standalone top-level `skim_files` field when you need this override +behavior. If that top-level field is present, config promotion keeps it instead +of the injected `project.skim_files` value. + +Run overrides have one additional rule: a run with no override block uses the +complete global resolution, but a run-specific override reloads the selected +skimjoin file. Any skim or network path omitted from that run block then comes +from the selected skimjoin file. Repeat a required global path in the run block +when the file does not contain it. + +## Common Recipes + +### Basic OD Lookup + +```yaml +activitysim: + trip_mode_column: trip_mode + tour_mode_column: tour_mode + trip_id_column: trip_id + tour_id_column: tour_id + outbound_column: outbound + +defaults: + origin: OTAZ + destination: DTAZ + output_prefix: skim_ + +zone_mapping: + lookup_name: taz + +modes: + SOV: + time: SOV_TIME + distance: SOV_DIST +``` + +`time: SOV_TIME` is equivalent to: + +```yaml +time: + matrix: SOV_TIME +``` + +When multiple skim files contain the same matrix name, qualify the reference +with its source filename. For example, set the files in the main config: + +```yaml +skimjoin: + defaults: + skim_files: + - C:\skims\bike_commute.omx + - C:\skims\bike_noncommute.omx +``` + +In the skimjoin rules file: + +```yaml +modes: + BIKE: + distance: + matrix: "bike_commute.omx::distance" +``` + +Unqualified references continue to work when the matrix name is unique across +the configured skim files. An ambiguous unqualified reference fails validation +and lists the available qualified names. Filename-qualified references may also +contain dimension placeholders. + +### Period Dimension Lookup + +The main config can supply both paths: + +```yaml +skimjoin: + defaults: + skim_files: + - C:\skims\auto.omx + network_los_file: C:\skims\network_los.yaml +``` + +The rules file defines how to use `network_los.yaml`: + +```yaml +activitysim: + trip_mode_column: trip_mode + tour_mode_column: tour_mode + trip_id_column: trip_id + tour_id_column: tour_id + outbound_column: outbound + +defaults: + origin: OTAZ + destination: DTAZ + +dimensions: + PERIOD: + source_columns: + trip_source_column: depart + outbound_tour_source_column: start + inbound_tour_source_column: first_inbound_trip_depart + values_from_network_los: true + +modes: + SOV: + time: SOV_TIME__{PERIOD} +``` + +### Segmented PNR Lookup + +```yaml +modes: + PNR_TRANSIT: + output_prefix: skim_ + segment_on: outbound + segments: + true: + auto_time: + matrix: SOV_TIME__{PERIOD} + origin: OTAZ + destination: pnr_taz + transit_time: + matrix: WTW_TIV__{PERIOD} + origin: pnr_taz + destination: DTAZ + false: + auto_time: + matrix: SOV_TIME__{PERIOD} + origin: pnr_taz + destination: DTAZ + transit_time: + matrix: WTW_TIV__{PERIOD} + origin: OTAZ + destination: pnr_taz +``` + +### Fallback Lookup + +```yaml +modes: + SOV: + time: + output: skim_auto_time + matrix: SOV_TIME__{PERIOD} + fallbacks: + - matrix: SOV_TIME__MD +``` + +Fallbacks run after the primary lookup and apply only to rows that do not yet +have a valid value. Every fallback step uses the same final output column. + +## Top-Level Sections + +| Section | Type | Default | Purpose | +|---|---|---|---| +| `project` | mapping | optional | Skim paths and standalone CLI paths. | +| `skim_files` | list | required after path resolution | Compatibility input promoted from `project.skim_files`. Prefer main-config paths for integrated use and `project.skim_files` for standalone use. | +| `activitysim` | mapping | required | Prepared trip/tour source column names. | +| `defaults` | mapping | built-in lookup defaults | Origin, destination, output prefix, missing-data policy, and sentinels. | +| `zone_mapping` | mapping | no mapping name | OMX zone lookup name behavior. | +| `dimensions` | mapping | `{}` | Placeholder definitions for matrix names. | +| `ignore_modes` | list | `[]` | Trip modes allowed to have no lookup rules. | +| `modes` | mapping | required | Mode-specific lookup rules. | +| `tour_aggregation` | mapping | built-in defaults | Accepted by the typed schema but unused. See [Unsupported Tour Aggregation](#unsupported-tour-aggregation). | + +The Pydantic schema rejects unknown keys in typed sections. + +## `project` + +`project` is a path container. It is not required when the main visualizer +config supplies all paths needed by the integrated workflow. + +| Field | Type | Default | Notes | +|---|---|---|---| +| `skim_files` | list of path strings | `[]` | OMX, CSV, HDF5, or H5 skim inputs. Required when the main visualizer config does not supply them, and required by the standalone `inventory` command. | +| `network_los_file` | path string | none | ActivitySim `network_los.yaml`, used when `dimensions.PERIOD.values_from_network_los` is true. | +| `trips_table` | path string | none | Standalone CLI input fallback. The integrated workflow ignores it. Prefer `activitysim.trips_table` for standalone use. | +| `tours_table` | path string | none | Optional standalone CLI input fallback. The integrated workflow ignores it. | +| `output_dir` | path string | none | Required by standalone `inventory`; used as the default output location for other CLI commands. The integrated workflow ignores it. | + +```yaml +project: + skim_files: + - C:\skims\*.omx + - C:\skims\maz_stop_walk.csv + network_los_file: C:\skims\network_los.yaml + trips_table: C:\prepared\trips.parquet + tours_table: C:\prepared\tours.parquet + output_dir: C:\skimjoin_output +``` + +For the standalone `validate`, `annotate-trips`, `annotate-tours`, and `run` +commands, a trips table must resolve through `activitysim.trips_table` or the +legacy `project.trips_table`. A tours table is optional unless the requested +operation needs tour input. Output flags can replace `project.output_dir` for +annotation commands. + +## `activitysim` + +`activitysim` names columns in prepared trip and tour tables. + +| Field | Type | Default | Notes | +|---|---|---|---| +| `trips_table` | path string | none | Standalone CLI trips table. May also be promoted from `project.trips_table`. | +| `tours_table` | path string | none | Standalone CLI tours table. May also be promoted from `project.tours_table`. | +| `trip_mode_column` | string | `trip_mode` | Mode column in prepared trips. | +| `tour_mode_column` | string | `tour_mode` | Mode column in prepared tours. | +| `trip_id_column` | string | `trip_id` | Trip id column. | +| `tour_id_column` | string | `tour_id` | Tour id column. | +| `outbound_column` | string | `outbound` | Trip outbound/inbound flag. | + +Column names cannot be blank. + +## `defaults` + +Each mode, segment, and component inherits these defaults. A value closer to +the rule takes precedence. + +| Field | Type | Default | Allowed values | Notes | +|---|---|---|---|---| +| `origin` | string | `origin` | source column | Origin column for OD lookups. | +| `destination` | string | `destination` | source column | Destination column for OD lookups. | +| `output_prefix` | string | `skim_` | any string | Prefix used when a component does not set `output`. | +| `missing_matrix_policy` | string | `error` | `error`, `warn`, `set_null` | Accepted and inherited for absent matrices but currently unenforced. See [Missing Policy Execution](#missing-policy-execution). | +| `missing_od_policy` | string | `error` | `error`, `warn`, `set_null` | Accepted and inherited for missing/out-of-bounds OD values but currently unenforced during annotation. See [Missing Policy Execution](#missing-policy-execution). | +| `sentinel_values` | list of numbers | `[]` | numeric list | Skimjoin treats lookup results equal to these values as missing. | + +```yaml +defaults: + origin: OTAZ + destination: DTAZ + output_prefix: skim_auto_ + missing_matrix_policy: warn + missing_od_policy: set_null + sentinel_values: [9999, 999999] +``` + +## Context Inheritance + +You can set these keys in the top-level `defaults`, a mode, a mode `defaults` +block, a segment, or a component: + +`origin`, `destination`, `output_prefix`, `missing_matrix_policy`, +`missing_od_policy`, `sentinel_values`, `when`, and `dimensions`. + +Settings nearer to a rule override or merge with parent settings: + +| Key | Merge behavior | +|---|---| +| `origin`, `destination`, `output_prefix`, policies | Override parent value. | +| `sentinel_values` | Override parent list. | +| `when` | Merge by source column; child values replace same-column parent values. | +| `dimensions` | Merge by dimension name; child dimension replaces same-name parent dimension. | + +```yaml +modes: + SOV: + output_prefix: skim_auto_ + defaults: + missing_od_policy: warn + time: + matrix: SOV_TIME + missing_od_policy: set_null +``` + +## `zone_mapping` + +`zone_mapping` controls the selection of an OMX lookup name. + +| Field | Type | Default | Allowed values | Notes | +|---|---|---|---|---| +| `lookup_name` | string or null | `null` | OMX mapping name | Default mapping name used for OMX OD matrices. | +| `file_lookup_names` | mapping | `{}` | file pattern to lookup name | Overrides `lookup_name` for matching file paths or file names. Patterns use shell-style matching. | +| `missing_zone_policy` | string | `error` | `error`, `warn`, `set_null` | Accepted but currently unenforced. | + +```yaml +zone_mapping: + lookup_name: taz + file_lookup_names: + fares.omx: zone_number + "*maz*.omx": maz + missing_zone_policy: error +``` + +## `dimensions` + +Dimensions supply placeholder values for matrix names such as +`SOV_TIME__{PERIOD}`. + +Each dimension entry has these fields: + +| Field | Type | Default | Notes | +|---|---|---|---| +| `source_columns.trip_source_column` | string | required | Source column used for trip lookup rules. | +| `source_columns.outbound_tour_source_column` | string | required | Source column used for outbound tour lookup rules. | +| `source_columns.inbound_tour_source_column` | string | required | Source column used for inbound tour lookup rules. | +| `values_from_network_los` | boolean | `false` | Only supported for `PERIOD`. Requires an effective `network_los_file` from the main config or `project`. | +| `values` | mapping | `{}` | Raw source value to matrix-name token. The loader normalizes keys and values to strings. | + +If `values` is empty, skimjoin converts the raw source value to a string and +inserts it into the matrix name. If `values` is present, every observed value +must have a mapping. + +```yaml +dimensions: + PERIOD: + source_columns: + trip_source_column: depart + outbound_tour_source_column: start + inbound_tour_source_column: first_inbound_trip_depart + values_from_network_los: true + + VOT: + source_columns: + trip_source_column: vot_bin + outbound_tour_source_column: vot_bin + inbound_tour_source_column: vot_bin + values: + L: L + M: M + H: H +``` + +## `ignore_modes` + +`ignore_modes` lists trip modes that do not need skim enrichment and therefore +do not require a matching `modes` rule. + +```yaml +ignore_modes: + - BIKE + - WALK + - OTHER +``` + +## `modes` + +`modes` contains the main skimjoin rules. Each key is a prepared trip or tour +mode. A mode block can contain context keys and component lookup rules. + +Reserved keys inside a mode block: + +| Key | Purpose | +|---|---| +| `output_prefix` | Overrides inherited output prefix. | +| `origin` | Overrides inherited origin column. | +| `destination` | Overrides inherited destination column. | +| `dimensions` | Overrides or adds dimension definitions for this mode. | +| `when` | Adds row filters for this mode. | +| `segment_on` | Splits the mode into segment-specific rule blocks. | +| `segments` | Segment value to component block mapping. | +| `defaults` | Nested context defaults for this mode. | +| `missing_matrix_policy` | Mode-level missing matrix policy. | +| `missing_od_policy` | Mode-level missing OD policy. | +| `sentinel_values` | Mode-level sentinel list. | +| `skip` | If `true`, skip the mode block. | +| `apply_to` | Reserved for component rules. | +| `combine` | Reserved for component rules. | +| `fallbacks` | Reserved for component rules. | +| `tour_origin` | Reserved for future/compatibility context. | +| `tour_destination` | Reserved for future/compatibility context. | + +Skimjoin uses each non-reserved key in a mode or segment block as a component +name. + +```yaml +modes: + HOV2: + output_prefix: skim_auto_ + time: SR2_TIME + distance: SR2_DIST +``` + +This example creates `skim_auto_time` and `skim_auto_distance`. + +## Component Rules + +A component rule can be a matrix-name string or a mapping. + +| Field | Type | Default | Allowed values | Notes | +|---|---|---|---|---| +| `matrix` | string | required | matrix/table value name | Matrix name, `filename::matrix` reference, or template using `{DIMENSION}` placeholders. | +| `output` | string | `output_prefix` + component name | output column | Final output column. Tour lookup outputs also receive `_outbound` or `_inbound`. | +| `lookup` | string | `od` | `od`, `key` | Lookup type. | +| `key_column` | string | none | source column | Required when `lookup: key`. | +| `origin` | string | inherited | source column | Origin column for OD lookup. | +| `destination` | string | inherited | source column | Destination column for OD lookup. Ignored for `lookup: key`. | +| `when` | mapping | inherited/merged | equality or `in` filters | Additional row filters. | +| `dimensions` | mapping | inherited/merged | dimension definitions | Component-specific placeholder definitions. | +| `missing_matrix_policy` | string | inherited | `error`, `warn`, `set_null` | Accepted and inherited but currently unenforced during annotation. | +| `missing_od_policy` | string | inherited | `error`, `warn`, `set_null` | Accepted and inherited but currently unenforced during annotation. | +| `sentinel_values` | list of numbers | inherited | numeric list | Component sentinel values. | +| `combine` | string | `replace` | `replace`, `sum` | How to combine overlapping outputs for a row. | +| `apply_to` | string | `both` | `trips`, `tours`, `both` | Whether the component creates trip rules, tour rules, or both. | +| `fallbacks` | list | `[]` | list of component mappings or strings | Lookup chain attempted after the primary step. | + +```yaml +modes: + WALK_TRANSIT: + output_prefix: skim_ + access_walk_time: + output: skim_walk_time + combine: sum + matrix: WTW_ACC__{PERIOD} + egress_walk_time: + output: skim_walk_time + combine: sum + matrix: WTW_EGR__{PERIOD} +``` + +If multiple rules write the same output for the same rows, set `combine: sum` +on all affected rules. The standalone strict validator reports an output +collision without this setting. Integrated loading does not run that check; +overlapping `replace` rules can proceed and keep the first resolved value. + +## `when` Filters + +`when` applies a rule only to rows that agree with source column conditions. A +condition can be scalar equality or an `in` list. + +```yaml +modes: + DRIVE: + time: + matrix: SOV_TIME + when: + income_segment: + in: [1, 2] + outbound: true +``` + +`when` filters merge through context inheritance. A mode-level filter applies +to all its components. A child filter can replace the same column key. + +## `segment_on` And `segments` + +Use `segment_on` when one mode requires different lookup rules for different +source values. Each key under `segments` is a value from the `segment_on` +column. Skimjoin adds the corresponding `when` filter for each segment. + +```yaml +modes: + PNR_TRANSIT: + segment_on: outbound + segments: + true: + auto_time: + matrix: SOV_TIME + origin: OTAZ + destination: pnr_taz + false: + auto_time: + matrix: SOV_TIME + origin: pnr_taz + destination: DTAZ +``` + +Validation checks that every observed value for a covered mode has a segment +block. + +## `fallbacks` + +Fallback entries use the same string or mapping format as primary component +rules. Skimjoin tries them in list order after a previous step fails. A fallback +uses the parent component output unless it sets its own, and all steps in the +chain must use the same final output. + +```yaml +modes: + SOV: + time: + matrix: SOV_TIME__{PERIOD} + fallbacks: + - matrix: SOV_TIME__MD + - matrix: SOV_TIME + missing_matrix_policy: set_null +``` + +Skimjoin writes fallback reports to `fallback_lookup_report`. + +## Lookup Types + +| `lookup` | Required fields | Behavior | +|---|---|---| +| `od` | `matrix`, `origin`, `destination` | Reads an OMX OD matrix or CSV OD table by origin and destination. | +| `key` | `matrix`, `key_column` | Reads a keyed sidecar table by one source column. | + +For CSV skim files, the inventory code finds key and value columns from the file +structure and can also identify origin and destination columns. For OMX files, +OD lookups use the configured `zone_mapping` lookup name. + +```yaml +modes: + WALK: + terminal_walk: + lookup: key + key_column: MAZ + matrix: walk_dist_local_bus + output: skim_walk_dist +``` + +### Concrete CSV Layouts + +A keyed CSV uses its first column as the key and every later numeric column as +a separate inventory value: + +```csv +MAZ,terminal_walk,parking_cost +101,2.5,4.00 +102,1.8,6.50 +``` + +If the file is `maz_access.csv`, its inventory names are +`maz_access__terminal_walk` and `maz_access__parking_cost`. A key rule can use: + +```yaml +terminal_walk: + lookup: key + key_column: origin + matrix: maz_access__terminal_walk +``` + +An OD CSV is recognized only when its first two normalized headers form one of +these pairs: `origin`/`destination`, `otaz`/`dtaz`, `omaz`/`dmaz`, +`orig`/`dest`, or `from`/`to`. Every later numeric column becomes a separate OD +table: + +```csv +OTAZ,DTAZ,time,distance +1,1,0.0,0.0 +1,2,12.5,8.1 +2,1,13.0,8.1 +2,2,0.0,0.0 +``` + +For `auto_md.csv`, refer to these values as `auto_md__time` and +`auto_md__distance`. CSV rows need not form a complete square matrix; an +unlisted pair remains unresolved and is eligible for fallback. Non-numeric +columns after the key or OD pair are ignored by the inventory. + +Keys and OD identifiers must be numeric or castable to `Float64`; null keys +and pairs are discarded. If a keyed CSV repeats one key, or an OD CSV repeats +one pair, each value column independently uses the last non-null value in file +order. + +Any other first-two-header combination is inventoried as a keyed CSV on its +first column, even when the rows otherwise look like OD data. Rename an +OD-shaped file's first two headers to a recognized pair before configuring an +OD lookup. + +The integrated workflow plans CSV reads from all configured matrix templates. +It selects only referenced value columns and restricts OD CSVs to a +conservative union of non-null pairs from every configured trip OD column pair +present in prepared trips and both directional-tour OD column pairs present in +prepared tours. Mode and `when` filters do not narrow this read-time union. +Keyed CSVs still retain every non-null key. The loaded physical CSV layouts are +shared across trip, tour, and hypothetical-sidecar lookups for that run. +Standalone annotation selects planned value columns but does not have the +integrated run-wide OD demand plan. + +### OMX, HDF5, And H5 Layouts + +Skimjoin inventories every two-dimensional dataset in `.omx`, `.h5`, and +`.hdf5` files. The inventory records the full dataset path but uses the final +path component as its unqualified matrix name. For example, dataset +`/data/SOV_TIME` is referred to as `SOV_TIME` when unique. If more than one +file exposes that name, use `filename.omx::SOV_TIME`. + +OD matrix row and column positions are resolved with the selected OMX mapping. +Set `zone_mapping.lookup_name`, or use `file_lookup_names` when files use +different mappings. Prepared origin and destination identifiers are cast to +`Int64`; use integer zone IDs. An identifier absent from an existing selected +mapping produces an unresolved lookup. A configured mapping name absent from +the OMX file instead raises an error and can fail the skimjoin stage. + +Without an OMX mapping, a lookup batch is treated as zero-based only when at +least one origin or destination is zero and all identifiers fit the matrix. +Otherwise identifiers are treated as one-based. Configure a mapping to avoid +this heuristic. Each selected OMX/HDF5/H5 dataset is loaded in full on first +use and cached by file and dataset path. + +## Trip And Tour Rules + +By default, each component creates trip and tour lookup rules: + +| Target | Source mode column | Dimension source | Output name | +|---|---|---|---| +| Trips | `activitysim.trip_mode_column` | `trip_source_column` | `output` | +| Outbound tours | `activitysim.tour_mode_column` | `outbound_tour_source_column` | `output_outbound` | +| Inbound tours | `activitysim.tour_mode_column` | `inbound_tour_source_column` | `output_inbound` | + +Set `apply_to: trips` or `apply_to: tours` to run a component on only one target +table. + +## Missing Data And Reports + +During integrated prepare, skimjoin writes these report artifacts: + +| Report | Purpose | +|---|---| +| `skim_lookup_summary` | Successful lookup counts and output summaries. | +| `missing_lookup_report` | Missing matrix, missing OD, missing dimension, and skipped lookup details. | +| `fallback_lookup_report` | Fallback attempts and outcomes. | +| `skipped_rule_report` | Rules skipped because a trip source column or mode column is missing. | +| `tour_aggregation_summary` | On a successful run, currently contains the ordinary direct-tour lookup summary rather than a trip-to-tour aggregation. | +| `failure_report` | Runtime failure detail when skimjoin cannot complete. | + +The files are under +`//prepared_tables/skimjoin/`. They are CSV except for the +resolved `config_normalized.yaml`. Empty reports are still written with their +declared headers when the integrated run reaches report packaging. + +### Report Schemas + +| Report | Columns | +|---|---| +| `skim_lookup_summary` | `rule_name`, `mode`, `component`, `output`, `matrix_name`, `n_trips`, `origin_column`, `destination_column`, `mean_value`, `min_value`, `max_value`, `n_missing` | +| `missing_lookup_report` | `rule_name`, `trip_id`, `origin`, `destination`, `matrix_name`, `reason` | +| `skipped_rule_report` | `rule_name`, `reason`, `n_rows` | +| `fallback_lookup_report` | `table_name`, `rule_name`, `output`, `logical_id`, `direction`, `primary_matrix_name`, `fallback_matrix_name`, `fallback_step_index`, `fallback_reason`, `fallback_eligible`, `fallback_attempted`, `fallback_succeeded`, `fallback_exhausted` | +| `failure_report` | `stage`, `error_type`, `detail` | + +`n_trips` is the number of lookup rows covered by a rule/matrix combination, +including invalid results; `n_missing` is the invalid subset. In the fallback +report, `logical_id` is the trip or tour ID named by `table_name`, and +`direction` is populated for directional tour work. The `reason` and +`fallback_reason` strings are diagnostic codes/details; treat them as +diagnostics rather than a stable category enumeration for downstream data +exchange. + +The final prepared manifest also stores compact run-level fields: + +| Manifest field | Meaning | +|---|---| +| `skimjoin_enabled` | Whether integrated skimjoin was enabled for the run. | +| `skimjoin_status` | Completed, recorded failure, or other packaged execution state. | +| `skimjoin_config_digest` | Identity of normalized lookup behavior. | +| `skimjoin_resolved_network_los_file` | Effective network LOS path, if used. | +| `skimjoin_applied_outputs` | Every final trip/tour column whose name contains `skim_`. This can include a pre-existing column and omits a new custom-named output without that text. | +| `skimjoin_skipped_rules` | Compact skipped-rule records. | +| `skimjoin_warning_count`, `skimjoin_fallback_count` | Aggregate diagnostic counts. | +| `skimjoin_fallback_outputs` | Outputs that used fallback values. | +| `skimjoin_hypothetical_sidecars_enabled` | Whether hypothetical sidecars were requested. | +| `skimjoin_trip_hypothetical_rows`, `skimjoin_tour_hypothetical_rows` | Written sidecar row counts. | +| `skimjoin_failure_detail` | Recorded exception detail under record policy. | + +The `skimjoin_applied_outputs` name filter can leave +`skimjoin_status: no_outputs` after a custom-named output was successfully +written, or produce `applied` from only a pre-existing `skim_` column. This is +a current manifest implementation gap; inspect the prepared table and lookup +summary rather than treating the status as proof that a new lookup resolved. + +## Missing Policy Execution + +The schema accepts and inherits `missing_matrix_policy`, `missing_od_policy`, +and `zone_mapping.missing_zone_policy`, but current annotation does not branch +on those values. During integrated and standalone annotation, `error`, `warn`, +and `set_null` therefore behave alike: an unresolved lookup can proceed to a +fallback, otherwise it is reported and remains null or absent. + +For a viable rule that matches at least one row, the standalone `validate` +command treats a referenced missing matrix as a validation failure. It consults +`missing_od_policy` only for direct OMX/HDF5/H5 matrix-bound checks: `error` +adds a failure, `warn` adds a warning, and `set_null` adds neither. That bounds +check compares raw origin and destination values with matrix shape without +applying the configured OMX mapping, so mapped external zone IDs can be +reported out of bounds even when the mapping is valid. No current execution +path consults `missing_zone_policy`. These are implementation gaps, not +supported policy differences. They do not affect the main visualizer +`skimjoin.failure_policy`, which separately controls whole-stage exceptions. + +## Unsupported Tour Aggregation + +The typed rules schema accepts an unused `tour_aggregation` block with `method`, +`aggregations`, and `directional_outputs`, but current annotation ignores it. +Direct tours are annotated through the normal tour rules, and the legacy +trip-to-tour aggregation helper raises an unsupported-operation error. Do not +configure this block as an active feature. The successful +`tour_aggregation_summary` report currently duplicates the ordinary direct-tour +lookup summary; its empty/failure schema is still a different legacy schema. + +## Related Chapters + +- [13 - Configuration Reference](13-configuration-reference.md#skimjoin) +- [22 - Skimjoin](22-skimjoin.md), including the standalone CLI +- [90 - Troubleshooting](90-troubleshooting.md) diff --git a/wiki/23-summary-functions.md b/wiki/23-summary-functions.md deleted file mode 100644 index eb68960..0000000 --- a/wiki/23-summary-functions.md +++ /dev/null @@ -1,153 +0,0 @@ -# 23 - Summary Functions - -Summary functions turn prepared `RunData` into dashboard-ready Polars -`DataFrame`s. A summary's identity, prerequisites, output schema, cache name, -and builder are declared together. - -## Mental Model - -```text -RunData + Config - -> @summary declaration and builder - -> validated Polars DataFrame - -> weighted/unweighted summary cache - -> dashboard page -``` - -Builders live under [`processor/summarize/summaries`](../processor/summarize/summaries). -`processor.summarize.catalog` explicitly imports those owning modules and -discovers their declarations. There is no separate summary-spec registry to -edit. - -## Summary Declaration - -Use `@summary(...)` from `processor.summarize`. The declaration provides: - -- the stable summary ID and optional cache filename -- an ordered Polars output schema -- required prepared tables and columns -- a typed empty result -- strict result validation -- whether the summary is built by default - -```python -import polars as pl - -from processor.models import RunData -from processor.summarize import summary -from runtime.config import Config - - -@summary( - id="trip_distance_by_mode", - schema={ - "trip_mode": pl.Utf8, - "trip_count": pl.Float64, - "average_distance": pl.Float64, - }, - required_columns={ - "trips": ("trip_mode", "od_dist", "finalweight"), - }, -) -def trip_distance_by_mode(run: RunData, config: Config) -> pl.DataFrame: - return ( - run.trips.group_by("trip_mode") - .agg( - trip_count=pl.col("finalweight").sum(), - average_distance=( - (pl.col("od_dist") * pl.col("finalweight")).sum() - / pl.col("finalweight").sum() - ), - ) - .with_columns( - pl.col("trip_mode").cast(pl.Utf8), - pl.col("trip_count").cast(pl.Float64), - pl.col("average_distance").cast(pl.Float64), - ) - .select("trip_mode", "trip_count", "average_distance") - ) -``` - -Successful builders must return exactly the declared columns, in the declared -order and with the declared dtypes. Missing declared inputs are handled before -the builder runs and produce its typed empty result. - -Use `required_tables` only when the presence of an entire table or `skim` is -enough to express the prerequisite. Use `required_columns` for ordinary table -dependencies; it also implies that the named runtime table must exist. Table -names here are `RunData` names (`hh`, `per`, `tours`, `trips`, -`joint_participants`, `land_use`), not config IDs such as `households` or -`persons`. - -## Weighting - -Builders aggregate `finalweight`; they do not branch on weighting mode. The -summary workflow supplies the appropriate prepared data for weighted and -unweighted builds. - -## Adding A Summary Function - -For a complete calculation, contract test, catalog, and page-wiring example, -follow the [Summary Function Cookbook](44-summary-function-cookbook.md). - -1. Put the builder in the domain module that owns the calculation. -2. Decorate it with `@summary(...)` and declare identity, ordered schema, and - mechanical prerequisites. -3. Read prepared `RunData` tables, not raw files. -4. Aggregate `finalweight` and return one long-form `pl.DataFrame`. -5. Cast and select explicitly at the end of the builder. -6. Use `builder.empty()` only for domain-specific empty conditions that the - declared prerequisites cannot express. -7. Add focused calculation and contract tests. -8. Add the summary ID to a page's required or optional summaries when needed. -9. Run `uv run python scripts/generate_wiki_catalogs.py`. - -The catalog import rejects duplicate IDs. Ordinary summarize workflows build -every declaration with `build_by_default=True`; enabled page requirements do -not narrow or expand that build set. `build_by_default=False` registers a -contract without adding it to ordinary generated builds. In the current public -workflow this is the external-table pattern: provide the table through -`summary_table_map`. Merely listing a non-default ID in a page declaration does -not cause its builder to run. - -## Summary CSV Boundary - -Summary caches are the dashboard input and their registered tables are already -stored as CSV files under each run and weighting mode. Normal summarize runs -write missing or stale cache tables unless `--skip-summary-cache-write` is used. - -For a developer diagnostic, this command bypasses reusable summary caches, -rebuilds the configured summaries, and forces the cache CSVs/manifests to be -written: - -```bash -uv run activitysim-viz --config local_config.yaml --summarize --write-csvs -``` - -It does not create a second export format or a separate calibration directory. -`processor.summarize.csv_export.write_summary_csvs()` is the shared low-level -writer used by cache storage. Dashboard pages load registered summaries through -`self.data`; they do not open those CSVs directly. - -To register a new dashboard-ready table produced outside the visualizer, use -the [outside summary table recipe](41-data-extension-cookbook.md#worked-example-add-an-outside-summary-table). - -## Segmentation - -Segmentation runs inside the summarize workflow and builds the same declarations -for configured slices of the prepared data. Segment sources may be a prepared -column or a CSV lookup. Dashboard visibility is controlled by -`segment.dashboard`. - -## Summary Catalog - -The generated [24 - Summary Catalog](24-summary-catalog.md) lists every current -declaration, output filename, builder, schema, and prerequisite. Regenerate it -after summary declarations change. - -## Related Chapters - -- [20 - Output Processor](20-output-processor.md) -- [21 - Prepared Tables](21-prepared-tables.md) -- [31 - Dashboard Pages](31-dashboard-pages.md) -- [44 - Summary Function Cookbook](44-summary-function-cookbook.md) diff --git a/wiki/24-segmentation.md b/wiki/24-segmentation.md new file mode 100644 index 0000000..3d340a4 --- /dev/null +++ b/wiki/24-segmentation.md @@ -0,0 +1,319 @@ +# 24 - Segmentation + +Segmentation runs the standard summary catalog for selected subsets of a model +run. For example, it can produce the same summaries for urban and rural +households, for income groups, or for people in different survey samples. + +Segmentation does not add a grouping column to one summary. It creates a +related `RunData` slice for each configured segment, then runs every registered +default summary against that slice. + +## Runtime Placement + +```text +raw or prepared input + -> prepare + -> optional skimjoin + -> resolve segment membership + -> slice related prepared tables in memory + -> summarize the full run and each segment + -> write full and segmented summary caches + -> show the selected segmentation in the dashboard +``` + +Segmentation is part of the summarize workflow. Enable it by adding `segment` +to `pipeline.steps`; a `segment` configuration block does not enable the step +by itself. The step also requires `summarize`. + +```yaml +pipeline: + steps: [segment, summarize, dashboard] + dashboard_mode: live + refresh: [] +``` + +Add `prepare` when you want cache creation to be explicit. If the workflow also +uses skimjoin, include both `prepare` and `skimjoin` before `segment`. + +## Complete Prepared-Column Example + +This example divides each run by a canonical person column: + +```yaml +pipeline: + steps: [segment, summarize, dashboard] + dashboard_mode: live + refresh: [] + +segment: + dashboard: + segmentation_type: person_sex + visibility: full_and_segments + definitions: + person_sex: + source: + type: prepared_column + source_table: per + column: sex + allow_overlapping: false + on_empty_segment: warn + segments: + - id: female + label: Female + values: [2] + - id: male + label: Male + values: [1] +``` + +`person_sex` is the segmentation type. Each segment selects source rows whose +`sex` value appears in its `values` list. The dashboard presents the results as +series such as `Base (Female)` and `Base (Male)`. + +## How Table Slicing Works + +The source table is the anchor for membership. After matching its rows, the +runtime follows canonical IDs to create a consistent set of related tables: + +| Source table | Membership starts with | Related data retained | +|---|---|---| +| `hh` | matching households | Their people, days, vehicles, tours, trips, and joint tours. | +| `per` | matching people | Their households, days, tours, trips, vehicles, and joint participation rows. | +| `day` | matching day rows | Related people or households, then their tours, trips, vehicles, and joint tours. | +| `tours` | matching tours | Their people, households, trips, participants, days, and vehicles. | +| `trips` | matching trips | Their tours, people, households, participants, days, and vehicles. | +| `vehicles` | matching vehicles | Their households and all related household records. | +| `joint_participants` | matching participation rows | Their joint tours, tour owners and participants, households, trips, days, and vehicles. | +| `land_use` | matching MAZ or TAZ rows | Households whose home zone matches, then their related records. | + +This relationship expansion is important when interpreting totals. A +trip-based segment contains only matching trips, but its household total counts +households associated with those trips. It is not a household classification +unless the source itself is a household field. + +The relationship keys must be present. Vehicle sources need `household_id`; +joint-participant sources need `tour_id` and `person_id`; day sources need +`person_id` or `household_id`; and land-use sources need a resolved `MAZ` or +`TAZ` key. + +## Source Types + +### Prepared Column + +Use `prepared_column` when the segment value already exists in a prepared +table: + +```yaml +source: + type: prepared_column + source_table: hh + column: income_segment +``` + +`source_table` accepts `households`, `persons`, `day`, `tours`, `trips`, +`vehicles`, `joint_tour_participants`, and `land_use`, along with their runtime +aliases `hh`, `per`, and `joint_participants`. + +You can omit `source_table` if the column occurs in exactly one segmentable +table. Set it explicitly when the name is absent or appears in more than one +table. + +### CSV Lookup + +Use `csv_lookup` when membership comes from an external classification: + +```yaml +segment: + definitions: + district: + source: + type: csv_lookup + file: lookups\household_district.csv + join: + source_table: hh + source_key_column: household_id + csv_key_column: household_id + segment_value_column: district + segments: + - id: north + label: North + values: [North] + - id: south + label: South + values: [South] +``` + +Relative lookup paths start from the main configuration directory. The CSV +must contain the join key and segment-value columns. Keys and values cannot be +blank, and one CSV key cannot map to multiple segment values. The join must not +duplicate rows in the anchor table. + +## Definition And Segment Settings + +| Field | Default | Behavior | +|---|---|---| +| `source` | required | Selects a prepared column or CSV lookup and its anchor table. | +| `segments` | required | Defines the path-safe lowercase `id`, display `label`, and matched `values` for each segment. | +| `allow_overlapping` | `false` | When `false`, one source value cannot appear in more than one segment in the same definition. When `true`, the same row can contribute to multiple segments. | +| `on_empty_segment` | `warn` | `error` stops the run, `skip` omits the analysis unit, and `warn` keeps an empty analysis unit so its summaries can report empty or unavailable results. | +| `include_full` | `true` | Accepted by the schema. The current runtime always builds one full-run analysis unit, regardless of this value. Control dashboard visibility with `segment.dashboard.visibility`. | +| `persist_segmented_prepared_tables` | `false` | Accepted by the schema. The current runtime keeps segment slices in memory and does not write separate prepared-table directories. | + +Segment IDs and definition names become path components, so they must already +be lowercase and path-safe. The exact accepted form is one or more lowercase +letters, digits, periods, underscores, or hyphens (`[a-z0-9._-]+`), with no +leading or trailing period, underscore, or hyphen. Names can start with a +digit. Spaces, slashes, uppercase letters, and characters outside that set are +rejected rather than normalized for you. + +Value typing is source-specific: + +- `prepared_column` compares each YAML value to the prepared column using its + existing type. Use numbers for numeric columns, booleans for Boolean columns, + and quoted strings when a numeric-looking code is stored as text. +- `csv_lookup` trims and stores lookup segment values as strings. The + corresponding `segments[*].values` should therefore be strings too. For + example, use `values: ["1"]`, not `values: [1]`, for CSV value `1`. +- CSV join keys are trimmed as strings during config normalization, then cast + to the prepared anchor key's type for the join. Values that cannot be cast do + not match. + +A segment can combine several source values: + +```yaml +- id: low_and_medium + label: Low and Medium Income + values: [low, medium] +``` + +Segments do not have to cover every source value. Unmatched rows remain in the +full-run summaries but do not appear in any configured segment. If overlapping +is enabled, do not add segment totals together unless double counting is +intentional. + +## Dashboard And Export Settings + +`segment.dashboard` selects which stored series the live dashboard shows: + +| Field | Default | Behavior | +|---|---|---| +| `segmentation_type` | first definition by name | Selects one configured definition for presentation. Other definitions can still exist in the cache. | +| `visibility` | `full_and_segments` | `full_only`, `segments_only`, or `full_and_segments`. | + +HTML export inherits these values. Override them for one export with: + +```yaml +dashboard: + export: + dashboard: + segmentation_type: district + segmentation_visibility: segments_only +``` + +An export can only use segmentation types and segments already present in the +summary cache. + +## Outputs And Cache Behavior + +Full-run summaries keep their standard paths: + +```text +//summary_tables//.csv +``` + +Segmented summaries use: + +```text +//summary_tables//segments/ + //.csv +``` + +The run-level summary manifest records each type and segment, including its +label, source, matched values, summary states, diagnostics, and digests. The +prepared cache remains at the normal run path. + +One shortened manifest entry looks like this: + +```json +{ + "segmentation_type": "district", + "source_type": "csv_lookup", + "segment_column": "district", + "source_table": "hh", + "source_key_column": "household_id", + "csv_file": "C:\\lookups\\household_district.csv", + "csv_key_column": "household_id", + "csv_segment_value_column": "district", + "include_full": false, + "segments": [ + { + "segmentation_type": "district", + "segment_id": "north", + "segment_label": "North", + "is_full": false, + "source_type": "csv_lookup", + "segment_column": "district", + "segment_values": ["North"], + "summary_roots": { + "weighted": "summary_tables/weighted/segments/district/north" + }, + "summary_states": {}, + "summary_diagnostics": {}, + "summary_digests": {} + } + ] +} +``` + +The stored `include_full` value on the type entry describes the segmented tree, +not the accepted config field. The full run remains at the standard summary +root and is always built by the current runtime. + +With `refresh: []`, cache validation is independent for the full run and each +segment. Adding or changing one segment rebuilds that segment while compatible +full-run and other segment summaries remain reusable. Removing a segment prunes +its obsolete summary directory on the next cache write. Use +`refresh: [summarize]` to rebuild all full and segmented summaries while keeping +prepared data. + +Segmentation requires raw or prepared rows. A run supplied only through +`summary_table_map` cannot be segmented because its tables are already +aggregated. If a run combines raw or prepared input with `summary_table_map`, +the mapped external table is overlaid unchanged on every full and segmented +analysis unit. Do not use that pattern for a measure that must vary by segment. + +## Implementation And Extension Points + +| Task | Start here | +|---|---| +| Config validation and normalization | `runtime/config/normalize_segmentation.py` | +| Relationship slicing and analysis units | `processor/segmentation.py` | +| Segment identity and metadata | `processor/analysis_units.py` | +| Summary workflow integration | `runtime/workflows/summarize.py` | +| Cache paths and manifests | `processor/summarize/cache.py` and `cache_storage.py` | +| Dashboard series selection | `dashboard/state.py` | + +Summary builders normally need no segment-specific code. They receive a sliced +`RunData` object and use the same declaration, schema, and weighting logic as +the full run. + +## Troubleshooting + +| Symptom | Check | +|---|---| +| No segmented output | Make sure `pipeline.steps` contains both `segment` and `summarize`. | +| Source column not found | Set the correct runtime `source_table` and inspect the prepared schema. | +| CSV lookup fails | Check path resolution, required columns, blank values, duplicate keys, and join-key types. | +| A segment is empty | Compare its `values` with the prepared or lookup values and review `on_empty_segment`. | +| Totals overlap | Check `allow_overlapping` and whether the selected anchor represents the population being counted. | +| Dashboard shows only full or only segmented series | Check `segment.dashboard.visibility` and the selected `segmentation_type`. | +| Old segment remains on disk | Run summarize so the next cache write can prune obsolete units. | + +## Related Chapters + +- [12 - Running Workflows](12-running-workflows.md) +- [13 - Configuration Reference](13-configuration-reference.md#segment) +- [21 - Prepared Tables](21-prepared-tables.md) +- [25 - Summary Functions](25-summary-functions.md) +- [42 - Config, Columns, And Labels](42-config-column-label-cookbook.md) +- [90 - Troubleshooting](90-troubleshooting.md) diff --git a/wiki/24-summary-catalog.md b/wiki/24-summary-catalog.md deleted file mode 100644 index a64deb1..0000000 --- a/wiki/24-summary-catalog.md +++ /dev/null @@ -1,119 +0,0 @@ -# 24 - Summary Catalog - -This page is generated from the `@summary(...)` declarations collected by -`processor.summarize.catalog`. - -Regenerate it with: - -```bash -uv run python scripts/generate_wiki_catalogs.py -``` - - -_Generated from `processor.summarize.catalog.SUMMARY_DEFINITIONS`._ - -Total registered summaries: **100** - -| Summary ID | Filename | Builder | Output schema | Required inputs | -|---|---|---|---|---| -| `adult_escort_event_stop_distribution` | `adult_escort_event_stop_distribution.csv` | `processor.summarize.summaries.daily_travel_escort_distributions.adult_escort_event_stop_distribution` | `segment: String`
    `stop_count: Int32`
    `tour_count: Float64` | tours: `tour_id`, `school_esc_outbound`, `school_esc_inbound`
    trips: `tour_id`, `escort_event_role`, `escort_stops_before_event`, `escort_stops_after_event`, `finalweight` | -| `adult_escort_trip_stop_frequency` | `adult_escort_trip_stop_frequency.csv` | `processor.summarize.summaries.daily_travel_escort_distributions.adult_escort_trip_stop_frequency` | `tour_purpose: String`
    `outbound_stop_count: Int32`
    `inbound_stop_count: Int32`
    `total_stop_count: Int32`
    `tour_count: Float64` | tours: `tour_purpose`, `school_esc_outbound`, `school_esc_inbound`, `num_ob_stops`, `num_ib_stops`, `num_tot_stops`, `finalweight` | -| `adult_escorted_tour_distance_distribution_by_direction` | `adult_escorted_tour_distance_distribution_by_direction.csv` | `processor.summarize.summaries.daily_travel_escort_distributions.adult_escorted_tour_distance_distribution_by_direction` | `distance_bin: String`
    `direction: String`
    `tour_count: Float64` | tours: `SKIMDIST`, `school_esc_outbound`, `school_esc_inbound`, `finalweight` | -| `adult_escorted_tour_purposes_by_direction` | `adult_escorted_tour_purposes_by_direction.csv` | `processor.summarize.summaries.daily_travel_escort_counts.adult_escorted_tour_purposes_by_direction` | `tour_purpose: String`
    `direction: String`
    `tour_count: Float64` | tours: `tour_purpose`, `school_esc_outbound`, `school_esc_inbound`, `finalweight` | -| `adult_escorted_tours_by_person_type_and_direction` | `adult_escorted_tours_by_person_type_and_direction.csv` | `processor.summarize.summaries.daily_travel_escort_counts.adult_escorted_tours_by_person_type_and_direction` | `person_type: String`
    `direction: String`
    `tour_count: Float64` | tours: `school_esc_outbound`, `school_esc_inbound`, `finalweight` | -| `adult_escorted_trip_distance_distribution_by_direction` | `adult_escorted_trip_distance_distribution_by_direction.csv` | `processor.summarize.summaries.daily_travel_escort_distributions.adult_escorted_trip_distance_distribution_by_direction` | `distance_bin: String`
    `direction: String`
    `trip_count: Float64` | tours: `tour_id`, `school_esc_outbound`, `school_esc_inbound`
    trips: `tour_id`, `od_dist`, `finalweight` | -| `allocated_vehicle_age_by_occupancy` | `allocated_vehicle_age_by_occupancy.csv` | `processor.summarize.summaries.tour_vehicles.allocated_vehicle_age` | `age: String`
    `occupancy: String`
    `vehicle_count: Float64` | tours: `vehicle_occup_1`, `vehicle_occup_2`, `vehicle_occup_3.5`, `finalweight` | -| `allocated_vehicle_body_type_by_occupancy` | `allocated_vehicle_body_type_by_occupancy.csv` | `processor.summarize.summaries.tour_vehicles.allocated_vehicle_body` | `body_type: String`
    `occupancy: String`
    `vehicle_count: Float64` | tours: `vehicle_occup_1`, `vehicle_occup_2`, `vehicle_occup_3.5`, `finalweight` | -| `allocated_vehicle_fuel_type_by_occupancy` | `allocated_vehicle_fuel_type_by_occupancy.csv` | `processor.summarize.summaries.tour_vehicles.allocated_vehicle_fuel` | `fuel_type: String`
    `occupancy: String`
    `vehicle_count: Float64` | tours: `vehicle_occup_1`, `vehicle_occup_2`, `vehicle_occup_3.5`, `finalweight` | -| `atwork_subtour_frequency_distribution` | `atwork_subtour_frequency_distribution.csv` | `processor.summarize.summaries.tour_profiles.at_work_sub_tour_freq` | `atwork_subtour_frequency_category: String`
    `atwork_subtour_count: Float64` | tours: `tour_purpose`, `tour_category`, `atwork_subtour_frequency`, `finalweight` | -| `auto_ownership_distribution` | `auto_ownership_distribution.csv` | `processor.summarize.summaries.long_term_vehicle.auto_ownership` | `household_size: String`
    `household_vehicle_count: Int64`
    `household_count: Float64` | hh: `HHSIZE`, `HHVEH`, `finalweight` | -| `auto_vmt_by_home_geography_income_hhsize_time_period` | `auto_vmt_by_home_geography_income_hhsize_time_period.csv` | `processor.summarize.summaries.validation.auto_vmt_by_home_geography_income_hhsize_time_period` | `geography_type: String`
    `geography_id: String`
    `income_segment: String`
    `household_size: String`
    `time_period: String`
    `mode: String`
    `auto_vmt: Float64`
    `trip_count: Float64`
    `distance_source: String`
    `time_period_source: String` | trips: `finalweight` | -| `auto_vmt_totals` | `auto_vmt_totals.csv` | `processor.summarize.summaries.validation.auto_vmt_totals` | `auto_vmt: Float64` | trips: `trip_mode`, `od_dist`, `finalweight` | -| `auto_vmt_validation_summary` | `auto_vmt_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.auto_vmt_validation_summary` | `TOD: String`
    `SOV: Float64`
    `HOV2: Float64`
    `HOV3: Float64`
    `Truck: Float64`
    `Total: Float64` | - | -| `autonomous_vehicle_ownership_totals` | `autonomous_vehicle_ownership_totals.csv` | `processor.summarize.summaries.long_term_vehicle.av_ownership` | `household_with_autonomous_vehicle_count: Float64` | hh: `av_ownership`, `finalweight` | -| `average_mandatory_tour_distance_by_purpose_and_geography` | `average_mandatory_tour_distance_by_purpose_and_geography.csv` | `processor.summarize.summaries.tour_geography.avg_mand_tour_distance` | `mandatory_tour_purpose: String`
    `geography_type: String`
    `geography_id: String`
    `average_tour_distance: Float64`
    `person_count: Float64` | per: `finalweight` | -| `average_nonmandatory_tour_distance_by_purpose_and_geography` | `average_nonmandatory_tour_distance_by_purpose_and_geography.csv` | `processor.summarize.summaries.tour_geography.avg_non_mand_tour_distance` | `nonmandatory_tour_purpose: String`
    `geography_type: String`
    `geography_id: String`
    `average_tour_distance: Float64`
    `tour_count: Float64` | per: `person_id`, `home_zone_id`
    tours: `person_id`, `tour_category`, `tour_purpose`, `SKIMDIST`, `finalweight` | -| `bicycle_comfort_level_distribution` | `bicycle_comfort_level_distribution.csv` | `processor.summarize.summaries.long_term_person.bicycle_comfort_level` | `person_type: String`
    `bicycle_comfort_level: String`
    `person_type_label: String`
    `person_count: Float64` | per: `person_type`, `bike_comfort`, `finalweight` | -| `bicycle_vmt_by_facility_type` | `bicycle_vmt_by_facility_type.csv` | `processor.summarize.summaries.validation.bicycle_vmt_by_facility` | `facility_type: String`
    `bicycle_vmt: Float64` | - | -| `commercial_vehicle_validation_summary` | `commercial_vehicle_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.commercial_vehicle_validation_summary` | `tod: String`
    `car: Float64`
    `mu: Float64`
    `su: Float64`
    `Total: Float64` | - | -| `commercial_vehicle_vmt_validation_summary` | `commercial_vehicle_vmt_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.commercial_vehicle_vmt_validation_summary` | `tod: String`
    `car: Float64`
    `mu: Float64`
    `su: Float64`
    `Total: Float64` | - | -| `commercial_vmt_totals` | `commercial_vmt_totals.csv` | `processor.summarize.summaries.validation.commercial_vehicle_vmt` | `commercial_vehicle_type: String`
    `external_vmt: Float64`
    `internal_vmt: Float64` | - | -| `commuting_flows` | `commuting_flows.csv` | `processor.summarize.summaries.long_term_geography.commuting_flows` | `origin_geography_type: String`
    `origin_geography_id: String`
    `destination_geography_type: String`
    `destination_geography_id: String`
    `commuter_count: Float64` | per: `home_zone_id`, `workplace_zone_id`, `is_worker`, `finalweight` | -| `count_location_counts_validation_summary` | `count_location_counts_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.count_location_counts_validation_summary` | `id: Int64`
    `FACTYPE: Int64`
    `am_vol: Float64`
    `md_vol: Float64`
    `pm_vol: Float64`
    `day_vol: Float64` | - | -| `count_location_fit_validation_summary` | `count_location_fit_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.count_location_fit_validation_summary` | `facility_type: String`
    `period: String`
    `slope: Float64`
    `intercept: Float64`
    `r_squared: Float64`
    `n_locations: Int64`
    `observed_min: Float64`
    `observed_max: Float64`
    `equation_label: String`
    `r_squared_label: String` | - | -| `count_location_scatter_validation_summary` | `count_location_scatter_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.count_location_scatter_validation_summary` | `id: Int64`
    `facility_type: String`
    `period: String`
    `observed_volume: Float64`
    `modeled_volume: Float64` | - | -| `count_location_volumes_validation_summary` | `count_location_volumes_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.count_location_volumes_validation_summary` | `id: Int64`
    `FACTYPE: Int64`
    `am_vol: Float64`
    `md_vol: Float64`
    `pm_vol: Float64`
    `day_vol: Float64` | - | -| `county_flows_joja_validation_summary` | `county_flows_joja_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.county_flows_joja_validation_summary` | `: String`
    `Benton: Float64`
    `Linn: Float64`
    `Marion: Float64`
    `Total: Float64` | - | -| `county_flows_validation_summary` | `county_flows_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.county_flows_validation_summary` | `: String`
    `Albany: Float64`
    `Corvallis: Float64`
    `Lebanon: Float64`
    `Philomath: Float64`
    `Total: Float64` | - | -| `daily_activity_pattern_by_person_type` | `daily_activity_pattern_by_person_type.csv` | `processor.summarize.summaries.daily_travel_activity.dap_summary` | `person_type: String`
    `daily_activity_pattern: String`
    `person_count: Float64` | per: `person_type`, `cdap_activity`, `finalweight` | -| `escorted_tour_totals` | `escorted_tour_totals.csv` | `processor.summarize.summaries.daily_travel_escort_counts.total_escorted_tours` | `tour_count: Float64` | tours: `school_esc_outbound`, `school_esc_inbound`, `finalweight` | -| `external_nonmandatory_tour_locations` | `external_nonmandatory_tour_locations.csv` | `processor.summarize.summaries.tour_geography.ext_non_mand_tour_loc` | `geography_type: String`
    `geography_id: String`
    `external_nonmandatory_tour_count: Float64` | tours: `tour_category`, `is_external_tour`, `destination`, `finalweight` | -| `external_trip_validation_summary` | `external_trip_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.external_trip_validation_summary` | `tod: String`
    `hbcoll: Float64`
    `hbo: Float64`
    `hbr: Float64`
    `hbs: Float64`
    `hbsch: Float64`
    `hbw: Float64`
    `nhbnw: Float64`
    `nhbw: Float64`
    `truck: Float64`
    `Total: Float64` | - | -| `external_vmt_validation_summary` | `external_vmt_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.external_vmt_validation_summary` | `tod: String`
    `hbcoll: Float64`
    `hbo: Float64`
    `hbr: Float64`
    `hbs: Float64`
    `hbsch: Float64`
    `hbw: Float64`
    `nhbnw: Float64`
    `nhbw: Float64`
    `truck: Float64`
    `Total: Float64` | - | -| `external_worker_workplace_locations` | `external_worker_workplace_locations.csv` | `processor.summarize.summaries.long_term_geography.external_workplace_loc` | `geography_type: String`
    `geography_id: String`
    `external_worker_count: Float64`
    `all_worker_count: Float64` | per: `is_worker`, `is_external_worker`, `external_workplace_zone_id`, `finalweight` | -| `free_parking_eligibility_by_workplace_geography` | `free_parking_eligibility_by_workplace_geography.csv` | `processor.summarize.summaries.long_term_geography.free_parking` | `geography_type: String`
    `geography_id: String`
    `workers_without_free_parking_count: Float64`
    `workers_with_free_parking_count: Float64` | per: `is_worker`, `free_parking_at_work`, `workplace_zone_id`, `finalweight` | -| `household_jtp_by_household_size_and_jtf` | `household_jtp_by_household_size_and_jtf.csv` | `processor.summarize.summaries.joint_travel.jtf_by_hhsize` | `jtf: String`
    `household_size: String`
    `household_percent: Float64` | hh: `household_id`, `HHSIZE`, `finalweight`
    tours: `tour_category`, `household_id` | -| `household_size_distribution` | `household_size_distribution.csv` | `processor.summarize.summaries.demographics.hh_size` | `household_size: Int64`
    `household_count: Float64` | hh: `HHSIZE`, `finalweight` | -| `households_with_school_escorting_by_student_count_and_direction` | `households_with_school_escorting_by_student_count_and_direction.csv` | `processor.summarize.summaries.daily_travel_escort_counts.households_with_school_escorting_by_student_count_and_direction` | `student_count: Int64`
    `direction: String`
    `household_count: Float64` | hh: `household_id`, `finalweight`
    per: `household_id`, `person_type`
    tours: `tour_purpose`, `school_esc_outbound`, `school_esc_inbound`, `finalweight` | -| `internal_external_nonmandatory_tour_frequency_by_home_geography` | `internal_external_nonmandatory_tour_frequency_by_home_geography.csv` | `processor.summarize.summaries.tour_geography.int_vs_ext_non_mand_tour_freq` | `geography_type: String`
    `geography_id: String`
    `internal_nonmandatory_tour_count: Float64`
    `external_nonmandatory_tour_count: Float64` | per: `person_id`, `home_zone_id`
    tours: `person_id`, `tour_category`, `is_external_tour`, `finalweight` | -| `internal_external_worker_by_geography` | `internal_external_worker_by_geography.csv` | `processor.summarize.summaries.long_term_geography.internal_vs_external` | `geography_type: String`
    `geography_id: String`
    `internal_worker_count: Float64`
    `external_worker_count: Float64` | per: `is_worker`, `is_external_worker`, `home_zone_id`, `finalweight` | -| `joint_tour_composition_by_party_size` | `joint_tour_composition_by_party_size.csv` | `processor.summarize.summaries.joint_travel.joint_composition_by_party_size` | `tour_composition: String`
    `party_size: Int64`
    `joint_tour_count: Float64` | tours: `tour_category`, `composition`, `number_of_participants`, `finalweight` | -| `joint_tour_composition_distribution` | `joint_tour_composition_distribution.csv` | `processor.summarize.summaries.joint_travel.joint_composition` | `tour_composition: String`
    `joint_tour_count: Float64` | tours: `tour_category`, `finalweight` | -| `joint_tour_party_size_distribution` | `joint_tour_party_size_distribution.csv` | `processor.summarize.summaries.joint_travel.joint_party_size` | `party_size: Int32`
    `joint_tour_count: Float64` | tours: `tour_category`, `NUMBER_HH`, `finalweight` | -| `joint_tours_by_household_size` | `joint_tours_by_household_size.csv` | `processor.summarize.summaries.joint_travel.joint_tours_hhsize` | `household_size: Int32`
    `household_count: Float64`
    `joint_tour_hh_count: Float64` | hh: `household_id`, `HHSIZE`, `finalweight`
    tours: `tour_category`, `household_id` | -| `jtf_distribution` | `jtf_distribution.csv` | `processor.summarize.summaries.joint_travel.joint_tour_freq` | `jtf_code: Int32`
    `jtf_label: String`
    `household_count: Float64` | hh: `household_id`, `finalweight` | -| `license_holding_status_distribution` | `license_holding_status_distribution.csv` | `processor.summarize.summaries.long_term_person.license_holding_status` | `person_type: String`
    `license_holding_status: String`
    `person_type_label: String`
    `person_count: Float64` | per: `person_type`, `has_license`, `finalweight`, `age` | -| `link_validation_summary` | `link_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.link_validation_summary` | `id: Int64`
    `From_Node: Int64`
    `To_Node: Int64`
    `FACTYPE: Int64`
    `am_vol: Float64`
    `md_vol: Float64`
    `pm_vol: Float64`
    `day_vol: Float64` | - | -| `mandatory_tour_frequency_by_person_type` | `mandatory_tour_frequency_by_person_type.csv` | `processor.summarize.summaries.daily_travel_activity.mandatory_tour_freq` | `person_type: String`
    `mandatory_tour_frequency: Int32`
    `person_count: Float64` | per: `person_type`, `imf_choice`, `finalweight` | -| `non_motorized_vmt_by_home_geography_income_hhsize_time_period` | `non_motorized_vmt_by_home_geography_income_hhsize_time_period.csv` | `processor.summarize.summaries.validation.non_motorized_vmt_by_home_geography_income_hhsize_time_period` | `geography_type: String`
    `geography_id: String`
    `income_segment: String`
    `household_size: String`
    `time_period: String`
    `mode: String`
    `non_motorized_vmt: Float64`
    `trip_count: Float64`
    `distance_source: String`
    `time_period_source: String` | trips: `finalweight`, `trip_mode` | -| `nonmandatory_tour_frequency_by_person_type` | `nonmandatory_tour_frequency_by_person_type.csv` | `processor.summarize.summaries.daily_travel_activity.indiv_nm_summary` | `person_type: String`
    `nonmandatory_tour_frequency: String`
    `person_count: Float64` | joint_participants: `person_id`
    per: `person_id`, `person_type`, `finalweight`
    tours: `person_id`, `tour_category` | -| `park_and_ride_location_residual_histogram` | `park_and_ride_location_residual_histogram.csv` | `processor.summarize.summaries.long_term_geography.park_and_ride_location_residual_histogram` | `geography_type: String`
    `bin_start: Float64`
    `bin_end: Float64`
    `geography_count: Float64` | land_use: -
    tours: `tour_mode`, `finalweight` | -| `park_and_ride_location_residuals` | `park_and_ride_location_residuals.csv` | `processor.summarize.summaries.long_term_geography.park_and_ride_location_residuals` | `geography_type: String`
    `geography_id: String`
    `pnr_tour_count: Float64`
    `pnr_lot_capacity: Float64`
    `residual_count: Float64`
    `absolute_residual_count: Float64`
    `percent_error: Float64` | land_use: -
    tours: `tour_mode`, `finalweight` | -| `parking_locations` | `parking_locations.csv` | `processor.summarize.summaries.trip.parking_locations` | `geography_type: String`
    `geography_id: String`
    `trip_count: Float64` | trips: `parking_zone`, `finalweight` | -| `person_jtp_by_household_size` | `person_jtp_by_household_size.csv` | `processor.summarize.summaries.joint_travel.joint_participation_person_by_hhsize` | `household_size: Int64`
    `joint_tour_person_count: Float64`
    `total_person_count: Float64` | hh: `household_id`, `hhsize`
    per: `household_id`, `num_joint_tours`, `finalweight` | -| `person_type_distribution` | `person_type_distribution.csv` | `processor.summarize.summaries.demographics.person_type` | `person_type: String`
    `person_type_label: String`
    `person_count: Float64` | per: `person_type`, `finalweight` | -| `population_totals` | `population_totals.csv` | `processor.summarize.summaries.demographics.population_totals` | `person_count: Float64`
    `household_count: Float64`
    `tour_count: Float64`
    `trip_count: Float64`
    `stop_count: Float64` | hh: `finalweight`
    per: `finalweight`
    tours: `finalweight`
    trips: `finalweight`, `stops` | -| `school_escorted_tours_by_escort_type_and_direction` | `school_escorted_tours_by_escort_type_and_direction.csv` | `processor.summarize.summaries.daily_travel_escort_counts.escorted_tours_to_from_school` | `escort_type: String`
    `direction: String`
    `tour_count: Float64` | tours: `tour_purpose`, `school_esc_outbound`, `school_esc_inbound`, `finalweight` | -| `school_location_distance_distribution_by_geography` | `school_location_distance_distribution_by_geography.csv` | `processor.summarize.summaries.long_term_distance.schl_tlfd` | `distance_bin: Int32`
    `geography_type: String`
    `geography_id: String`
    `person_count: Float64` | per: `distance_to_school`, `finalweight` | -| `school_location_enrollment_comparison` | `school_location_enrollment_comparison.csv` | `processor.summarize.summaries.long_term_geography.school_loc_vs_land_use_enrollment` | `geography_type: String`
    `geography_id: String`
    `student_type: String`
    `enrollment_count: Float64`
    `student_count: Float64` | land_use: `MAZ`, `enrollment_count`, `student_type`
    per: `school_zone_id`, `is_student`, `finalweight` | -| `school_shadow_pricing_residual_histogram` | `school_shadow_pricing_residual_histogram.csv` | `processor.summarize.summaries.long_term_geography.school_shadow_pricing_residual_histogram` | `geography_type: String`
    `student_type: String`
    `bin_start: Float64`
    `bin_end: Float64`
    `geography_count: Float64` | land_use: `MAZ`, `enrollment_count`, `student_type`
    per: `school_zone_id`, `is_student`, `finalweight`, `student_type` | -| `school_shadow_pricing_residuals` | `school_shadow_pricing_residuals.csv` | `processor.summarize.summaries.long_term_geography.school_shadow_pricing_residuals` | `geography_type: String`
    `geography_id: String`
    `student_type: String`
    `target_count: Float64`
    `modeled_count: Float64`
    `residual_count: Float64`
    `absolute_residual_count: Float64`
    `percent_error: Float64` | land_use: `MAZ`, `enrollment_count`, `student_type`
    per: `school_zone_id`, `is_student`, `finalweight`, `student_type` | -| `schoolkids_per_escorted_tour_by_student_count_and_direction` | `schoolkids_per_escorted_tour_by_student_count_and_direction.csv` | `processor.summarize.summaries.daily_travel_escort_counts.schoolkids_per_escorted_tour_by_student_count_and_direction` | `student_count: Int64`
    `direction: String`
    `avg_schoolkids_per_tour: Float64`
    `tour_count: Float64` | hh: `household_id`, `finalweight`
    per: `household_id`, `person_type`
    tours: `school_esc_outbound`, `school_esc_inbound`, `num_escortees`, `finalweight` | -| `screenline_flow_comparisons` | `screenline_flow_comparisons.csv` | `processor.summarize.summaries.validation.screenline_flow_comparisons` | `screenline_id: String`
    `direction: String`
    `count_period: String`
    `observed_volume: Float64`
    `modeled_volume: Float64` | - | -| `skimjoin_tour_component_ecdf` | `skimjoin_tour_component_ecdf.csv` | `processor.summarize.summaries.skimjoin.tour_skim_component_ecdf` | `skim_scenario: String`
    `tour_mode: String`
    `component: String`
    `percentile: Float64`
    `value: Float64`
    `n_valid: Float64` | tours: `tour_mode`, `finalweight` | -| `skimjoin_tour_component_stats` | `skimjoin_tour_component_stats.csv` | `processor.summarize.summaries.skimjoin.tour_skim_component_stats` | `skim_scenario: String`
    `tour_mode: String`
    `component: String`
    `n_total: Float64`
    `n_valid: Float64`
    `mean: Float64`
    `std: Float64`
    `min: Float64`
    `max: Float64`
    `median: Float64`
    `mode: Float64`
    `zero_share: Float64`
    `missing_share: Float64` | tours: `tour_mode`, `finalweight` | -| `skimjoin_trip_component_ecdf` | `skimjoin_trip_component_ecdf.csv` | `processor.summarize.summaries.skimjoin.trip_skim_component_ecdf` | `skim_scenario: String`
    `trip_mode: String`
    `component: String`
    `percentile: Float64`
    `value: Float64`
    `n_valid: Float64` | trips: `trip_mode`, `finalweight` | -| `skimjoin_trip_component_stats` | `skimjoin_trip_component_stats.csv` | `processor.summarize.summaries.skimjoin.trip_skim_component_stats` | `skim_scenario: String`
    `trip_mode: String`
    `component: String`
    `n_total: Float64`
    `n_valid: Float64`
    `mean: Float64`
    `std: Float64`
    `min: Float64`
    `max: Float64`
    `median: Float64`
    `mode: Float64`
    `zero_share: Float64`
    `missing_share: Float64` | trips: `trip_mode`, `finalweight` | -| `stop_destination_purpose_by_tour_purpose` | `stop_destination_purpose_by_tour_purpose.csv` | `processor.summarize.summaries.trip.stop_purpose_by_tour_purpose` | `stop_destination_purpose: String`
    `tour_purpose: String`
    `stop_count: Float64` | trips: `stops`, `tour_purpose`, `trip_purpose`, `finalweight` | -| `stop_out_of_direction_distance_by_tour_purpose` | `stop_out_of_direction_distance_by_tour_purpose.csv` | `processor.summarize.summaries.trip_distributions.stop_ood_distance` | `distance_bin: Int32`
    `tour_purpose: String`
    `stop_count: Float64` | trips: `stops`, `out_dir_dist`, `tour_purpose`, `finalweight` | -| `student_households_by_student_count` | `student_households_by_student_count.csv` | `processor.summarize.summaries.daily_travel_escort_counts.student_households_by_student_count` | `student_count: Int64`
    `household_count: Float64` | hh: `household_id`, `finalweight`
    per: `household_id`, `person_type` | -| `student_school_escort_status_by_direction` | `student_school_escort_status_by_direction.csv` | `processor.summarize.summaries.daily_travel_escort_counts.student_school_escort_status_by_direction` | `direction: String`
    `escort_type: String`
    `tour_count: Float64` | tours: `tour_purpose`, `school_esc_outbound`, `school_esc_inbound`, `finalweight` | -| `telecommute_frequency_distribution` | `telecommute_frequency_distribution.csv` | `processor.summarize.summaries.long_term_person.telecommute` | `geography_type: String`
    `geography_id: String`
    `telecommute_frequency: String`
    `person_count: Float64` | per: `telecommute_frequency`, `finalweight`, `is_worker`, `work_from_home`, `home_zone_id` | -| `tour_category_distribution` | `tour_category_distribution.csv` | `processor.summarize.summaries.tour.tour_category` | `tour_category: String`
    `tour_count: Float64` | tours: `tour_category`, `finalweight` | -| `tour_distance_by_tour_purpose` | `tour_distance_by_tour_purpose.csv` | `processor.summarize.summaries.tour_profiles.tour_distance` | `distance_bin: String`
    `tour_purpose: String`
    `tour_count: Float64` | tours: `tour_purpose`, `tour_category`, `number_of_participants`, `SKIMDIST`, `finalweight` | -| `tour_mode_by_tour_purpose_and_auto_sufficiency` | `tour_mode_by_tour_purpose_and_auto_sufficiency.csv` | `processor.summarize.summaries.tour_profiles.tour_mode` | `tour_mode: String`
    `tour_purpose: String`
    `tour_count_zero_auto: Float64`
    `tour_count_auto_deficient: Float64`
    `tour_count_auto_sufficient: Float64`
    `tour_count_all_households: Float64` | tours: `tour_mode`, `tour_purpose`, `finalweight`, `AUTOSUFF` | -| `tour_purpose_distribution` | `tour_purpose_distribution.csv` | `processor.summarize.summaries.tour.tour_purpose` | `tour_purpose: String`
    `tour_count: Float64` | tours: `tour_purpose`, `finalweight` | -| `tour_rates_by_person_type_and_tour_purpose` | `tour_rates_by_person_type_and_tour_purpose.csv` | `processor.summarize.summaries.daily_travel_activity.tour_rate_per_person` | `person_type: String`
    `tour_purpose: String`
    `tour_rate: Float64` | per: `person_id`, `person_type`, `finalweight`
    tours: `person_id`, `tour_purpose` | -| `tour_stop_frequency_by_tour_purpose` | `tour_stop_frequency_by_tour_purpose.csv` | `processor.summarize.summaries.tour_profiles.stop_freq` | `tour_purpose: String`
    `outbound_stop_count: Int32`
    `inbound_stop_count: Int32`
    `total_stop_count: Int32`
    `tour_count: Float64` | tours: `tour_purpose`, `tour_category`, `num_ob_stops`, `num_ib_stops`, `num_tot_stops`, `finalweight` | -| `tour_time_of_day_by_tour_purpose` | `tour_time_of_day_by_tour_purpose.csv` | `processor.summarize.summaries.tour_profiles.tour_tod` | `time_bin: Int32`
    `tour_purpose: String`
    `departure_tour_count: Float64`
    `arrival_tour_count: Float64`
    `duration_tour_count: Float64` | tours: `tour_category`, `tour_purpose`, `finalweight` | -| `traffic_count_comparisons` | `traffic_count_comparisons.csv` | `processor.summarize.summaries.validation.traffic_count_comparisons` | `count_location_id: String`
    `direction: String`
    `count_period: String`
    `observed_volume: Float64`
    `modeled_volume: Float64` | - | -| `transit_boardings_by_operator_and_technology` | `transit_boardings_by_operator_and_technology.csv` | `processor.summarize.summaries.validation.total_transit_boardings` | `operator: String`
    `technology: String`
    `boardings: Float64` | - | -| `transit_pass_ownership_by_person_type` | `transit_pass_ownership_by_person_type.csv` | `processor.summarize.summaries.long_term_person.transit_pass` | `person_type: String`
    `transit_pass_ownership_status: String`
    `person_type_label: String`
    `person_count: Float64` | per: `person_type`, `transit_pass_ownership`, `finalweight` | -| `transit_subsidy_by_person_type` | `transit_subsidy_by_person_type.csv` | `processor.summarize.summaries.long_term_person.transit_subsidy` | `person_type: String`
    `transit_subsidy_status: String`
    `transit_subsidy_label: String`
    `person_type_label: String`
    `person_count: Float64` | per: `person_type`, `transit_pass_subsidy`, `is_worker`, `is_student`, `finalweight` | -| `transit_transfer_rate` | `transit_transfer_rate.csv` | `processor.summarize.summaries.validation.transit_transfer_rate` | `operator: String`
    `technology: String`
    `access_mode: String`
    `transfer_rate: Float64` | - | -| `trip_departure_time_by_purpose` | `trip_departure_time_by_purpose.csv` | `processor.summarize.summaries.trip_distributions.trip_stop_tod` | `tour_purpose: String`
    `time_bin: Int32`
    `departure_trip_count: Float64`
    `departure_stop_count: Float64` | trips: `tour_purpose`, `stops`, `finalweight` | -| `trip_distance_by_purpose` | `trip_distance_by_purpose.csv` | `processor.summarize.summaries.trip_distributions.trip_distance` | `distance_bin: String`
    `tour_purpose: String`
    `trip_count: Float64` | trips: `tour_purpose`, `od_dist`, `num_participants`, `finalweight` | -| `trip_mode_by_tour_purpose_and_tour_mode` | `trip_mode_by_tour_purpose_and_tour_mode.csv` | `processor.summarize.summaries.trip.trip_mode` | `tour_purpose: String`
    `tour_mode: String`
    `trip_mode: String`
    `trip_count: Float64` | trips: `tour_purpose`, `tour_mode`, `trip_mode`, `finalweight` | -| `trip_purpose_distribution` | `trip_purpose_distribution.csv` | `processor.summarize.summaries.trip.trip_purpose` | `tour_purpose: String`
    `trip_purpose: String`
    `trip_count: Float64` | trips: `tour_purpose`, `trip_purpose`, `finalweight` | -| `trip_rates_by_person_type_and_trip_purpose` | `trip_rates_by_person_type_and_trip_purpose.csv` | `processor.summarize.summaries.daily_travel_activity.trip_rate_per_person` | `person_type: String`
    `trip_purpose: String`
    `trip_rate: Float64` | per: `person_id`, `person_type`, `finalweight`
    trips: `person_id`, `trip_purpose`, `finalweight` | -| `university_location_distance_distribution_by_geography` | `university_location_distance_distribution_by_geography.csv` | `processor.summarize.summaries.long_term_distance.univ_tlfd` | `distance_bin: Int32`
    `geography_type: String`
    `geography_id: String`
    `person_count: Float64` | per: `distance_to_school`, `finalweight` | -| `vehicle_age_distribution` | `vehicle_age_distribution.csv` | `processor.summarize.summaries.long_term_vehicle.vehicle_char_age` | `age: String`
    `vehicle_count: Float64` | vehicles: `vehicle_age`, `finalweight` | -| `vehicle_body_type_distribution` | `vehicle_body_type_distribution.csv` | `processor.summarize.summaries.long_term_vehicle.vehicle_char_body` | `body_type: String`
    `vehicle_count: Float64` | vehicles: `body_type`, `finalweight` | -| `vehicle_fuel_type_distribution` | `vehicle_fuel_type_distribution.csv` | `processor.summarize.summaries.long_term_vehicle.vehicle_char_fuel` | `fuel_type: String`
    `vehicle_count: Float64` | vehicles: `fuel_type`, `finalweight` | -| `work_from_home_rate_by_geography` | `work_from_home_rate_by_geography.csv` | `processor.summarize.summaries.long_term_geography.wfh` | `geography_type: String`
    `geography_id: String`
    `worker_count: Float64`
    `work_from_home_worker_count: Float64` | per: `is_worker`, `home_zone_id`, `finalweight` | -| `work_from_home_validation_summary` | `work_from_home_validation_summary.csv` | `processor.summarize.summaries.validation_scaffolds.work_from_home_validation_summary` | `District: String`
    `Workers: Float64`
    `WFH: Float64` | - | -| `work_location_distance_distribution_by_geography` | `work_location_distance_distribution_by_geography.csv` | `processor.summarize.summaries.long_term_distance.work_tlfd` | `distance_bin: Int32`
    `geography_type: String`
    `geography_id: String`
    `person_count: Float64` | per: `distance_to_work`, `finalweight` | -| `workplace_location_employment_comparison` | `workplace_location_employment_comparison.csv` | `processor.summarize.summaries.long_term_geography.workplace_vs_land_use_employment` | `geography_type: String`
    `geography_id: String`
    `employment_count: Float64`
    `worker_count: Float64` | land_use: `MAZ`, `employment_count`
    per: `workplace_zone_id`, `is_worker`, `finalweight` | -| `workplace_shadow_pricing_residual_histogram` | `workplace_shadow_pricing_residual_histogram.csv` | `processor.summarize.summaries.long_term_geography.workplace_shadow_pricing_residual_histogram` | `geography_type: String`
    `bin_start: Float64`
    `bin_end: Float64`
    `geography_count: Float64` | land_use: `MAZ`, `employment_count`
    per: `workplace_zone_id`, `is_worker`, `finalweight` | -| `workplace_shadow_pricing_residuals` | `workplace_shadow_pricing_residuals.csv` | `processor.summarize.summaries.long_term_geography.workplace_shadow_pricing_residuals` | `geography_type: String`
    `geography_id: String`
    `target_count: Float64`
    `modeled_count: Float64`
    `residual_count: Float64`
    `absolute_residual_count: Float64`
    `percent_error: Float64` | land_use: `MAZ`, `employment_count`
    per: `workplace_zone_id`, `is_worker`, `finalweight` | - diff --git a/wiki/25-skimjoin-config-reference.md b/wiki/25-skimjoin-config-reference.md deleted file mode 100644 index 7543637..0000000 --- a/wiki/25-skimjoin-config-reference.md +++ /dev/null @@ -1,543 +0,0 @@ -# 25 - Skimjoin Config Reference - -This page is the field-by-field reference for the standalone skimjoin config -file used by the main visualizer `skimjoin` step. For the workflow overview, -start with [22 - Skimjoin](22-skimjoin.md). The canonical example is -[`example_skimjoin_config.yaml`](../example_skimjoin_config.yaml). - -Skimjoin config answers four questions: - -1. Which skim files and optional `network_los.yaml` should be used? -2. Which prepared trip and tour columns provide modes, ids, dimensions, and OD - lookup columns? -3. Which matrix or sidecar table should be read for each mode/component? -4. What should happen when matrices, OD pairs, or dimension values are missing? - -## Common Recipes - -### Basic OD Lookup - -```yaml -project: - skim_files: - - C:\skims\auto.omx - -activitysim: - trip_mode_column: trip_mode - tour_mode_column: tour_mode - trip_id_column: trip_id - tour_id_column: tour_id - outbound_column: outbound - -defaults: - origin: OTAZ - destination: DTAZ - output_prefix: skim_ - -zone_mapping: - lookup_name: taz - -modes: - SOV: - time: SOV_TIME - distance: SOV_DIST -``` - -`time: SOV_TIME` is shorthand for: - -```yaml -time: - matrix: SOV_TIME -``` - -### Period Dimension Lookup - -```yaml -project: - skim_files: - - C:\skims\auto.omx - network_los_file: C:\skims\network_los.yaml - -activitysim: - trip_mode_column: trip_mode - tour_mode_column: tour_mode - trip_id_column: trip_id - tour_id_column: tour_id - outbound_column: outbound - -defaults: - origin: OTAZ - destination: DTAZ - -dimensions: - PERIOD: - source_columns: - trip_source_column: depart - outbound_tour_source_column: start - inbound_tour_source_column: first_inbound_trip_depart - values_from_network_los: true - -modes: - SOV: - time: SOV_TIME__{PERIOD} -``` - -### Segmented PNR Lookup - -```yaml -modes: - PNR_TRANSIT: - output_prefix: skim_ - segment_on: outbound - segments: - true: - auto_time: - matrix: SOV_TIME__{PERIOD} - origin: OTAZ - destination: pnr_taz - transit_time: - matrix: WTW_TIV__{PERIOD} - origin: pnr_taz - destination: DTAZ - false: - auto_time: - matrix: SOV_TIME__{PERIOD} - origin: pnr_taz - destination: DTAZ - transit_time: - matrix: WTW_TIV__{PERIOD} - origin: OTAZ - destination: pnr_taz -``` - -### Fallback Lookup - -```yaml -modes: - SOV: - time: - output: skim_auto_time - matrix: SOV_TIME__{PERIOD} - fallbacks: - - matrix: SOV_TIME__MD -``` - -Fallbacks run after the primary lookup for rows where the earlier step did not -produce a valid value. Fallback steps share the same final output column. - -### Tour Aggregation - -```yaml -tour_aggregation: - method: aggregate_trips - aggregations: - skim_auto_time: sum - skim_auto_distance: sum - skim_transit_fare: sum - directional_outputs: - skim_auto_time: true -``` - -Tour lookups are also generated directly from mode rules. For tour lookup rules, -outputs receive `_outbound` and `_inbound` suffixes. - -## Top-Level Sections - -| Section | Type | Default | Purpose | -|---|---|---|---| -| `project` | mapping | optional | Skim paths and standalone CLI paths. | -| `skim_files` | list | promoted from `project.skim_files` | Direct skim file list. Usually set under `project`. | -| `activitysim` | mapping | required | Prepared trip/tour source column names. | -| `defaults` | mapping | built-in lookup defaults | Origin, destination, output prefix, missing-data policy, and sentinels. | -| `zone_mapping` | mapping | no mapping name | OMX zone lookup name behavior. | -| `dimensions` | mapping | `{}` | Placeholder definitions for matrix names. | -| `ignore_modes` | list | `[]` | Trip modes allowed to have no lookup rules. | -| `modes` | mapping | required | Mode-specific lookup rules. | -| `tour_aggregation` | mapping | `aggregate_trips` with no configured aggregations | Trip-to-tour aggregation settings. | - -Unknown keys are rejected by the Pydantic schema for typed sections. - -## `project` - -| Field | Type | Default | Notes | -|---|---|---|---| -| `skim_files` | list of path strings | `[]` | OMX, CSV, HDF5, or H5 skim inputs. In integrated visualizer use, main config overrides may replace this list. | -| `network_los_file` | path string | none | ActivitySim `network_los.yaml`, used when `dimensions.PERIOD.values_from_network_los` is true. | -| `trips_table` | path string | none | Standalone skimjoin CLI input. Not required for integrated visualizer use. | -| `tours_table` | path string | none | Standalone skimjoin CLI input. Optional. | -| `output_dir` | path string | none | Standalone skimjoin CLI output directory. | - -```yaml -project: - skim_files: - - C:\skims\*.omx - - C:\skims\maz_stop_walk.csv - network_los_file: C:\skims\network_los.yaml -``` - -## `activitysim` - -`activitysim` names columns in prepared trip and tour tables. - -| Field | Type | Default | Notes | -|---|---|---|---| -| `trips_table` | path string | none | Standalone CLI trips table. May also be promoted from `project.trips_table`. | -| `tours_table` | path string | none | Standalone CLI tours table. May also be promoted from `project.tours_table`. | -| `trip_mode_column` | string | `trip_mode` | Mode column in prepared trips. | -| `tour_mode_column` | string | `tour_mode` | Mode column in prepared tours. | -| `trip_id_column` | string | `trip_id` | Trip id column. | -| `tour_id_column` | string | `tour_id` | Tour id column. | -| `outbound_column` | string | `outbound` | Trip outbound/inbound flag. | - -Column names cannot be blank. - -## `defaults` - -Defaults are inherited by every mode, segment, and component unless overridden -closer to the rule. - -| Field | Type | Default | Allowed values | Notes | -|---|---|---|---|---| -| `origin` | string | `origin` | source column | Origin column for OD lookups. | -| `destination` | string | `destination` | source column | Destination column for OD lookups. | -| `output_prefix` | string | `skim_` | any string | Prefix used when a component does not set `output`. | -| `missing_matrix_policy` | string | `error` | `error`, `warn`, `set_null` | Policy for absent matrices or matrix names that cannot be resolved. | -| `missing_od_policy` | string | `error` | `error`, `warn`, `set_null` | Policy for missing/out-of-bounds OD values. | -| `sentinel_values` | list of numbers | `[]` | numeric list | Lookup results equal to these values are treated as missing. | - -```yaml -defaults: - origin: OTAZ - destination: DTAZ - output_prefix: skim_auto_ - missing_matrix_policy: warn - missing_od_policy: set_null - sentinel_values: [9999, 999999] -``` - -## Context Inheritance - -These keys may be set at the top `defaults` level, on a mode, inside a mode -`defaults` block, inside a segment, or inside a component: - -`origin`, `destination`, `output_prefix`, `missing_matrix_policy`, -`missing_od_policy`, `sentinel_values`, `when`, and `dimensions`. - -Closer settings override or merge with parent settings: - -| Key | Merge behavior | -|---|---| -| `origin`, `destination`, `output_prefix`, policies | Override parent value. | -| `sentinel_values` | Override parent list. | -| `when` | Merge by source column; child values replace same-column parent values. | -| `dimensions` | Merge by dimension name; child dimension replaces same-name parent dimension. | - -```yaml -modes: - SOV: - output_prefix: skim_auto_ - defaults: - missing_od_policy: warn - time: - matrix: SOV_TIME - missing_od_policy: set_null -``` - -## `zone_mapping` - -`zone_mapping` controls OMX lookup-name selection. - -| Field | Type | Default | Allowed values | Notes | -|---|---|---|---|---| -| `lookup_name` | string or null | `null` | OMX mapping name | Default mapping name used for OMX OD matrices. | -| `file_lookup_names` | mapping | `{}` | file pattern to lookup name | Overrides `lookup_name` for matching file paths or file names. Patterns use shell-style matching. | -| `missing_zone_policy` | string | `error` | `error`, `warn`, `set_null` | Policy for missing zone mappings. | - -```yaml -zone_mapping: - lookup_name: taz - file_lookup_names: - fares.omx: zone_number - "*maz*.omx": maz - missing_zone_policy: error -``` - -## `dimensions` - -Dimensions provide placeholder values for matrix names such as -`SOV_TIME__{PERIOD}`. - -Each dimension entry has this shape: - -| Field | Type | Default | Notes | -|---|---|---|---| -| `source_columns.trip_source_column` | string | required | Source column used for trip lookup rules. | -| `source_columns.outbound_tour_source_column` | string | required | Source column used for outbound tour lookup rules. | -| `source_columns.inbound_tour_source_column` | string | required | Source column used for inbound tour lookup rules. | -| `values_from_network_los` | boolean | `false` | Only supported for `PERIOD`. Requires `project.network_los_file`. | -| `values` | mapping | `{}` | Raw source value to matrix-name token. Keys and values are normalized to strings. | - -If `values` is empty, the raw source value is converted to a string and inserted -into the matrix name. If `values` is present, observed values must have a -mapping. - -```yaml -dimensions: - PERIOD: - source_columns: - trip_source_column: depart - outbound_tour_source_column: start - inbound_tour_source_column: first_inbound_trip_depart - values_from_network_los: true - - VOT: - source_columns: - trip_source_column: vot_bin - outbound_tour_source_column: vot_bin - inbound_tour_source_column: vot_bin - values: - L: L - M: M - H: H -``` - -## `ignore_modes` - -`ignore_modes` lists trip modes that are allowed to appear in prepared trips -without a matching `modes` rule. This is useful for modes where skim enrichment -is intentionally skipped. - -```yaml -ignore_modes: - - BIKE - - WALK - - OTHER -``` - -## `modes` - -`modes` is the heart of skimjoin. Each key is a prepared trip or tour mode. Each -mode block may contain context keys plus component lookup rules. - -Reserved keys inside a mode block: - -| Key | Purpose | -|---|---| -| `output_prefix` | Overrides inherited output prefix. | -| `origin` | Overrides inherited origin column. | -| `destination` | Overrides inherited destination column. | -| `dimensions` | Overrides or adds dimension definitions for this mode. | -| `when` | Adds row filters for this mode. | -| `segment_on` | Splits the mode into segment-specific rule blocks. | -| `segments` | Segment value to component block mapping. | -| `defaults` | Nested context defaults for this mode. | -| `missing_matrix_policy` | Mode-level missing matrix policy. | -| `missing_od_policy` | Mode-level missing OD policy. | -| `sentinel_values` | Mode-level sentinel list. | -| `skip` | If `true`, skip the mode block. | -| `apply_to` | Reserved for component rules. | -| `combine` | Reserved for component rules. | -| `fallbacks` | Reserved for component rules. | -| `tour_origin` | Reserved for future/compatibility context. | -| `tour_destination` | Reserved for future/compatibility context. | - -Every non-reserved key in a mode or segment block is treated as a component -name. - -```yaml -modes: - HOV2: - output_prefix: skim_auto_ - time: SR2_TIME - distance: SR2_DIST -``` - -The output names above are `skim_auto_time` and `skim_auto_distance`. - -## Component Rules - -A component rule may be a string matrix name or a mapping. - -| Field | Type | Default | Allowed values | Notes | -|---|---|---|---|---| -| `matrix` | string | required | matrix/table value name | Matrix name or matrix-name template using `{DIMENSION}` placeholders. | -| `output` | string | `output_prefix` + component name | output column | Final output column. Tour lookup outputs also receive `_outbound` or `_inbound`. | -| `lookup` | string | `od` | `od`, `key` | Lookup type. | -| `key_column` | string | none | source column | Required when `lookup: key`. | -| `origin` | string | inherited | source column | Origin column for OD lookup, or key value when `lookup: key` and `key_column` is absent. | -| `destination` | string | inherited | source column | Destination column for OD lookup. Ignored for `lookup: key`. | -| `when` | mapping | inherited/merged | equality or `in` filters | Additional row filters. | -| `dimensions` | mapping | inherited/merged | dimension definitions | Component-specific placeholder definitions. | -| `missing_matrix_policy` | string | inherited | `error`, `warn`, `set_null` | Component missing-matrix policy. | -| `missing_od_policy` | string | inherited | `error`, `warn`, `set_null` | Component missing-OD policy. | -| `sentinel_values` | list of numbers | inherited | numeric list | Component sentinel values. | -| `combine` | string | `replace` | `replace`, `sum` | How to combine overlapping outputs for a row. | -| `apply_to` | string | `both` | `trips`, `tours`, `both` | Whether the component creates trip rules, tour rules, or both. | -| `fallbacks` | list | `[]` | list of component mappings or strings | Lookup chain attempted after the primary step. | - -```yaml -modes: - WALK_TRANSIT: - output_prefix: skim_ - access_walk_time: - output: skim_walk_time - combine: sum - matrix: WTW_ACC__{PERIOD} - egress_walk_time: - output: skim_walk_time - combine: sum - matrix: WTW_EGR__{PERIOD} -``` - -When multiple rules write the same output on overlapping rows, use -`combine: sum` on all overlapping rules. Otherwise validation treats the overlap -as an output collision. - -## `when` Filters - -`when` narrows a rule to rows that match source column conditions. Conditions -may be scalar equality or an `in` list. - -```yaml -modes: - DRIVE: - time: - matrix: SOV_TIME - when: - income_segment: - in: [1, 2] - outbound: true -``` - -`when` filters merge through context inheritance. A mode-level filter applies to -all of its components unless a child filter replaces the same column key. - -## `segment_on` And `segments` - -Use `segment_on` when one mode needs different lookup rules for different -source values. Each key under `segments` is a value from the `segment_on` -column. Skimjoin automatically adds a matching `when` filter for each segment. - -```yaml -modes: - PNR_TRANSIT: - segment_on: outbound - segments: - true: - auto_time: - matrix: SOV_TIME - origin: OTAZ - destination: pnr_taz - false: - auto_time: - matrix: SOV_TIME - origin: pnr_taz - destination: DTAZ -``` - -Validation checks that observed segment values for a covered mode have -configured segment blocks. - -## `fallbacks` - -Fallback entries use the same string or mapping shape as primary component -rules. They are attempted in list order after failed prior steps. A fallback -inherits the parent component output unless it explicitly sets an output, and -validation requires all steps in a fallback chain to share the same final -output. - -```yaml -modes: - SOV: - time: - matrix: SOV_TIME__{PERIOD} - fallbacks: - - matrix: SOV_TIME__MD - - matrix: SOV_TIME - missing_matrix_policy: set_null -``` - -Fallback reports are written to `fallback_lookup_report`. - -## Lookup Types - -| `lookup` | Required fields | Behavior | -|---|---|---| -| `od` | `matrix`, `origin`, `destination` | Reads an OMX OD matrix or CSV OD table by origin and destination. | -| `key` | `matrix`, `key_column` | Reads a keyed sidecar table by one source column. | - -For CSV skim files, inventory code identifies key/value or origin/destination -columns from the file structure. For OMX, OD lookups use the configured -`zone_mapping` lookup name. - -```yaml -modes: - WALK: - terminal_walk: - lookup: key - key_column: MAZ - matrix: walk_dist_local_bus - output: skim_walk_dist -``` - -## Trip And Tour Rules - -Every component creates trip and tour lookup rules by default: - -| Target | Source mode column | Dimension source | Output name | -|---|---|---|---| -| Trips | `activitysim.trip_mode_column` | `trip_source_column` | `output` | -| Outbound tours | `activitysim.tour_mode_column` | `outbound_tour_source_column` | `output_outbound` | -| Inbound tours | `activitysim.tour_mode_column` | `inbound_tour_source_column` | `output_inbound` | - -Set `apply_to: trips` or `apply_to: tours` when a component should only run on -one target table. - -## `tour_aggregation` - -`tour_aggregation` controls trip-to-tour rollups for skim columns. - -| Field | Type | Default | Allowed values | Notes | -|---|---|---|---|---| -| `method` | string | `aggregate_trips` | `aggregate_trips` | Only supported aggregation method. | -| `aggregations` | mapping | `{}` | `sum`, `mean`, `min`, `max`, `first`, `last` | Output column to aggregation method. | -| `directional_outputs` | mapping | `{}` | output column to boolean | Requests directional outbound/inbound tour outputs for selected components. | - -```yaml -tour_aggregation: - method: aggregate_trips - aggregations: - skim_auto_time: sum - skim_auto_distance: sum - skim_transit_fare: sum - directional_outputs: - skim_auto_time: true -``` - -## Missing Data And Reports - -Skimjoin writes report artifacts during integrated prepare: - -| Report | Purpose | -|---|---| -| `skim_lookup_summary` | Successful lookup counts and output summaries. | -| `missing_lookup_report` | Missing matrix, missing OD, missing dimension, and skipped lookup details. | -| `fallback_lookup_report` | Fallback attempts and outcomes. | -| `skipped_rule_report` | Rules skipped by missing source columns or other selection conditions. | -| `tour_aggregation_summary` | Tour lookup and aggregation details. | -| `failure_report` | Runtime failure detail when skimjoin cannot complete. | - -Policies: - -| Policy | Behavior | -|---|---| -| `error` | Treat the missing condition as a validation/runtime failure where enforced. | -| `warn` | Record warning/missing report rows and continue. | -| `set_null` | Write null for the missing value and continue. | - -## Related Chapters - -- [13 - Configuration Reference](13-configuration-reference.md#skimjoin) -- [13 - Configuration Reference](13-configuration-reference.md#skimjoin) -- [22 - Skimjoin](22-skimjoin.md) -- [90 - Troubleshooting](90-troubleshooting.md) diff --git a/wiki/25-summary-functions.md b/wiki/25-summary-functions.md new file mode 100644 index 0000000..0b15d70 --- /dev/null +++ b/wiki/25-summary-functions.md @@ -0,0 +1,324 @@ +# 25 - Summary Functions + +Summary functions convert prepared `RunData` into Polars `DataFrame` objects +for the dashboard. Each function keeps its identity, requirements, output +schema, cache name, and builder in one declaration. + +## Data flow + +```text +RunData + Config + -> @summary declaration and builder + -> validated Polars DataFrame + -> weighted/unweighted summary cache + -> dashboard page +``` + +Builders live under [`processor/summarize/summaries`](../processor/summarize/summaries). +`processor.summarize.catalog` imports those modules and discovers their +declarations, so there is no separate summary specification registry to edit. + +## Summary Declaration + +Use `@summary(...)` from `processor.summarize`. The declaration provides: + +- the stable summary ID and optional cache file name +- an ordered Polars output schema +- required prepared tables and columns +- a typed empty result +- strict result validation +- default build status + +```python +import polars as pl + +from processor.models import RunData +from processor.summarize import summary +from runtime.config import Config + + +@summary( + id="trip_distance_by_mode", + schema={ + "trip_mode": pl.Utf8, + "trip_count": pl.Float64, + "average_distance": pl.Float64, + }, + required_columns={ + "trips": ("trip_mode", "od_dist", "finalweight"), + }, +) +def trip_distance_by_mode(run: RunData, config: Config) -> pl.DataFrame: + return ( + run.trips.group_by("trip_mode") + .agg( + trip_count=pl.col("finalweight").sum(), + average_distance=( + (pl.col("od_dist") * pl.col("finalweight")).sum() + / pl.col("finalweight").sum() + ), + ) + .with_columns( + pl.col("trip_mode").cast(pl.Utf8), + pl.col("trip_count").cast(pl.Float64), + pl.col("average_distance").cast(pl.Float64), + ) + .select("trip_mode", "trip_count", "average_distance") + ) +``` + +A successful builder must return the declared columns, in order, with the +declared data types. Before running the builder, the workflow checks for missing +declared input and returns the typed empty result if any input is unavailable. + +Use `required_tables` only when a complete table or `skim` is enough to state +the requirement. For standard table dependencies, use `required_columns`, +which also requires the named runtime table. Specify `RunData` names such as +`hh`, `per`, `tours`, `trips`, `joint_participants`, and `land_use`, not +configuration IDs such as `households` or `persons`. + +## Weighting + +Builders aggregate `finalweight`. They do not select a weighting mode. The +summary workflow supplies the required prepared data for weighted and +unweighted builds. + +### Weight Resolution And Edge Cases + +The primary weighted mode is prepared as follows: + +1. An explicit run-level household/person/trip weight column is cast to + `Float64` on its table. +2. If no explicit household, person, or trip run weight is supplied, a + household `sample_rate` produces `1 / sample_rate`. `day_weight_col` does + not disable this household expansion. +3. Otherwise, household weight defaults to `1.0` when no household source was + selected. Supplying only a person or trip weight therefore disables + household sample-rate expansion. +4. Missing lower-level sources inherit through household/person/tour + relationships. An unmatched inherited row normally falls back to `1.0`. +5. When an explicit trip weight is used, tour weight is the mean trip weight + for that `tour_id`. +6. `day_weight_col` defaults to `day_weight`. A non-null day source is + authoritative; missing values inherit person weight, then household weight, + then `1.0`. Set the field to null to ignore the source for every day row. + +The unweighted mode changes existing `finalweight` columns to `1.0`; it does +not add that column to a custom prepared table that omitted it. Named column +modes follow the propagation rules in chapter 43. + +The runtime casts weights but does not apply a universal quality rule for +zero, negative, null, infinite, or extreme values. Consequences are +calculation-specific: + +- a null weight is ignored by a Polars sum and can remove that row's + contribution; +- zero weights contribute no count and can create a zero denominator; +- negative weights subtract from totals; +- `sample_rate: 0` can produce an infinite expansion weight; and +- a weighted average with a zero or invalid denominator can return null, NaN, + or infinity unless that builder handles the case. + +Validate source weights before production use. A practical contract is finite, +non-null, nonnegative weights and strictly positive sample rates. If zero +weights are intentional, test every rate and average that consumes them. + +### Units + +The visualizer does not maintain a separate unit registry or automatically +convert source values. Units are part of the source/prepared/summary contract: + +| Output kind | Unit rule | +|---|---| +| Counts and totals | `finalweight` expansion units; unweighted mode is row counts unless a builder applies occupancy/party logic. | +| Rates and shares | Ratio of the builder's declared numerator and denominator; dimensionless unless the label states a per-person or per-day basis. | +| Distance and VMT | Uses prepared distance values as supplied. Existing dashboard labels assume miles. Convert upstream or in prepare if the model uses another unit. | +| Time | Uses prepared time/hour/period fields and configured time-period mapping. Skim time components keep the skim's unit. | +| Cost and other skim components | Keeps the matrix or sidecar unit; skimjoin does not convert cents, dollars, minutes, seconds, or generalized cost. | +| Geography IDs and categories | Labels/identifiers, not measured units. | + +When you add a summary, state the unit in its column name, page axis/tooltip, or +calculation note. Do not combine runs whose underlying distance, time, or cost +units differ without normalizing them first. + +### Joint travel and tour-distance conventions + +Legacy-compatible summaries distinguish person travel from unique travel +records: + +- every eligible source row contributes under its active weight; builders do + not discard repeated rows merely because identifiers match; +- person-level tour/trip rates, mode counts, and population travel totals + expand only records categorized as joint; +- when joint-participant rows are available, person-tour and person-trip + summaries use the sum of every matching participant's person weight; valid + scalar party-size expansion is only the fallback; +- household-tour summaries (including tour purpose/category, joint party and + composition, stop frequency, time-of-day, duration, and distance) do not + expand joint tours by party size. When a verified survey group stores one row + per participant, every row is retained with `1 / party size` representation + weight so the complete group contributes one household tour; and +- a usable participant count is greater than zero and less than `995`. + Invalid, missing, and sentinel counts fall back to one when a person-level + record must still be counted, while party-size descriptive summaries exclude + them. + +This fractional household-tour representation is not record deduplication: the +source rows and their attributes remain in the calculation. If participant rows +report different time or distance values for the same joint-tour identifier, +the histogram shows their fractional mixture rather than silently selecting one +participant as the canonical record. + +Daily tour rates exclude at-work subtours. The tour-distance histogram and +nonmandatory average both use prepared `SKIMDIST` for every run. They do not +fall back to `tour_distance` or generic `distance_miles`, because either can +represent full traveled tour length instead of origin-to-primary-destination +distance. + +Both builders currently check for `SKIMDIST` inside the function but omit it +from their `@summary(required_columns=...)` declarations. Chapter 26 therefore +under-reports that prerequisite. Treat this as a summary-contract +implementation gap until the declarations include the field. + +The generated Required Inputs column contains declared mechanical +prerequisites, not every builder-level condition. For example, the joint-tour +composition builders need `composition` or `tour_composition`; the +composition-by-party-size builder also needs `number_of_participants` or +`NUMBER_HH`; and person participation by household size needs `HHSIZE` or +`hhsize`. These alternative requirements are resolved inside their builders. + +Daily tour and trip rate denominators use every attributable day row when a +nonempty day table with `person_id` is available, and their numerators retain +only travelers whose `person_id` occurs in that exposure set. A false +`surveyable` value excludes a person even when a day table is present. Without +day rows, every remaining prepared person represents one modeled day. + +The current numerator gate uses `person_id`, not `(person_id, day_id)`. It can +therefore retain activity from a different day for a person who has at least +one eligible day. This is an implementation gap for multi-day inputs that reuse +person identifiers; inputs with a unique person identifier per day do not +expose it. + +Daily activity summaries use a non-null person-table DAP or mandatory-frequency +choice for that person, then fall back only for remaining persons to all +eligible day rows and their day weights. Nonmandatory frequency is classified +per person-day for genuinely multi-day inputs only when all individual tours +and joint participations can be assigned to a day without guessing. If those +keys are incomplete or ambiguous, the summary retains the one-day/person +fallback rather than discarding unmatched rows. + +Joint-tour frequency and household participation use household observations. +For a genuinely multi-day diary, the observation key is +`(household_id, day_num)` when tours carry the matching day column. The current +implementation does not validate those tour days against the day table: null +or unmatched values simply do not contribute. Repeated records collapse first +by a valid `joint_tour_id`, then a valid `tour_id`; a row with neither becomes +its own identity. If the tour day column is absent, the result is empty rather +than pooling a week into one choice. One-day inputs continue to use one +observation per household. + +The legacy JTF labels use minimum thresholds for only shopping, maintenance, +eating out, visiting/social, and other discretionary. Joint purposes outside +those five counters are ignored, and a count above a threshold can still +receive its two-tour label. A household-day with only an outside purpose can +therefore appear as `No Joint Tours`. These are current implementation gaps, +not documented recoding rules for new inputs. + +The JTF summary declaration requires only the household table. On a one-day +input, an unavailable or zero-column tour table can therefore produce a +plausible all-`No Joint Tours` result instead of an unavailable summary. Check +tour availability before accepting that distribution. + +Person joint-tour participation similarly uses person-days when the day table +identifies household, person, and day and every unique joint tour maps to one +day. It uses the day weight when available and otherwise the person weight. An +absent or ambiguous tour-day mapping produces an empty result. If the day table +lacks household, person, or day fields, the existing person-level participation +fallback remains. Once the multi-day branch is selected, it also assumes a +schemaful joint-participant table: a zero-column table can fail the builder and +a schemaful empty table yields zero participating person-days instead of using +the person-level fallback. + +The Joint Travel page currently labels these charts as households or people +even when a multi-day run contributes household-day or person-day +observations. The Daily Activity Pattern page likewise labels person-day +frequency distributions as persons. Treat those static axes as a dashboard UI +gap and use the catalog's unit description when interpreting multi-day runs. + +## Adding A Summary Function + +For an example with a calculation, contract test, catalog, and page connection, +use the [Summary Function Cookbook](44-summary-function-cookbook.md). + +1. Put the builder in the domain module that owns the calculation. +2. Decorate it with `@summary(...)` and declare identity, ordered schema, and + mechanical prerequisites. +3. Read prepared `RunData` tables, not raw files. +4. Aggregate `finalweight` and return one long-form `pl.DataFrame`. +5. Cast and select explicitly at the end of the builder. +6. Use `builder.empty()` only for domain-specific empty conditions that the + declared prerequisites cannot express. +7. Add focused calculation and contract tests. +8. Add the summary ID to a page's required or optional summaries when needed. +9. Add the analytical and output-field descriptions to + `scripts/summary_catalog_metadata.yaml`. Add shared required-input metadata + there when the declaration introduces a new prepared table or field. +10. Use `uv run python scripts/generate_wiki_catalogs.py`. + +The catalog import rejects duplicate IDs. Standard summarize workflows build +every declaration with `build_by_default=True`, regardless of enabled page +requirements. Setting `build_by_default=False` registers the contract without +adding it to standard builds. Use this setting for an external table in the +public workflow and supply the table through `summary_table_map`. Referencing a +non-default ID in a page declaration does not start its builder. + +## Summary CSV Boundary + +Summary caches are the dashboard input. The visualizer stores registered tables +as CSV files for each run and weighting mode, and standard summarize workflows +write any that are missing or stale. Use `--skip-summary-cache-write` to prevent +these writes. + +For a developer diagnostic, the following command ignores reusable summary +caches, rebuilds the configured summaries, and writes their CSV files and +manifests: + +```bash +uv run activitysim-viz --config local_config.yaml --summarize --write-csvs +``` + +The command does not create a second export format or a separate calibration +directory. Cache storage uses the shared +`processor.summarize.csv_export.write_summary_csvs()` writer. Dashboard pages +load registered summaries through `self.data`. They do not open the CSV files +directly. + +To register a new dashboard-ready table produced outside the visualizer, use +the [outside summary table recipe](41-data-extension-cookbook.md#worked-example-add-an-outside-summary-table). + +## Segmentation + +Segmentation runs in the summarize workflow and builds the same declarations +for related subsets of prepared data. A source can be a prepared column or a +CSV lookup. The runtime slices `RunData`, applies the normal weighting modes, +and writes each result below the segment's summary-cache path. See +[24 - Segmentation](24-segmentation.md) for source-table relationships, settings, +outputs, cache behavior, and dashboard selection. + +## Summary Catalog + +The generated [26 - Summary Catalog](26-summary-catalog.md) combines each current +declaration with the analytical descriptions in +`scripts/summary_catalog_metadata.yaml`. The same generator combines those +summaries with prepared-output metadata and sidecar contracts in the portable +`reference/processor-output-table-reference.md`. Regenerate both references +after you change any of those sources. + +## Related Chapters + +- [20 - Output Processor](20-output-processor.md) +- [21 - Prepared Tables](21-prepared-tables.md) +- [24 - Segmentation](24-segmentation.md) +- [27 - Geography](27-geography.md) +- [31 - Dashboard Page Contract](31-dashboard-pages.md) +- [44 - Summary Function Cookbook](44-summary-function-cookbook.md) diff --git a/wiki/26-summary-catalog.md b/wiki/26-summary-catalog.md new file mode 100644 index 0000000..732717d --- /dev/null +++ b/wiki/26-summary-catalog.md @@ -0,0 +1,331 @@ +# 26 - Summary Catalog + +This page is the data dictionary for summary CSV tables from the Output +Processor. It explains what one row represents, how each table is used, and +what each output field means. The category references combine that analytical +context with authoritative file names, schemas, summary functions, and declared +input requirements. + +## How to Interpret the Tables + +- Weighting follows the analysis unit named in each field. Unless a table says + otherwise, weighted people sum `per.finalweight`, weighted households sum + `hh.finalweight`, weighted person-days sum `day.finalweight`, weighted tours + sum `tours.finalweight`, weighted trips sum `trips.finalweight`, and weighted + vehicles sum `vehicles.finalweight`. +- A lower-level table can inherit its weight from a person or household, but + its unit does not change. One person with weight `3.0` and four trip rows + contributes `3.0` weighted persons and `12.0` weighted trips. +- In an unweighted cache, existing `finalweight` columns are set to `1.0`. + Builder-specific participant and occupancy adjustments still apply. Measures + read from assignment tables or externally supplied contracts are not + automatically reweighted. +- Joint-tour participant expansion uses prepared `NUMBER_HH`, derived from the + participant table when a tour has participant records, then from + `number_of_participants`. An existing `NUMBER_HH` is retained as a final + source before defaulting to one. +- Rates, percentages, averages, standard deviations, medians, percentiles, and + fields with names such as `geography_count` use the calculation stated in + their table entry; they do not share one universal weighting rule. +- Values such as `all_geographies`, `all_person_types`, `all_tour_purposes`, + `all_tour_modes`, `All Modes`, `All Auto`, and `Daily` are rollups. Use either + the rollup or its component rows, but do not add them together. +- `geography_type` names the configured spatial system, such as MAZ, TAZ, + county, MPO, or a custom geography. `geography_id` is the identifier in that + system. Each table description identifies the home, work, school, + destination, or parking geography. +- Distance values use the units of the prepared distance or skim fields, + usually miles. Distance histograms use `0` only for exact zero, `>0-<1` for + positive sub-mile values, and whole-mile lower-bound labels thereafter. + Missing, nonfinite, and negative distances are excluded. End bins such as + `40+`, `20+`, or `51+` include all larger values. +- `time_bin` is the prepared ActivitySim period index: 1--24 for hourly inputs + or 1--48 for half-hour-period inputs. Named `time_period` and `count_period` + values come from configured or supplied period labels. +- Category codes and labels come from prepared ActivitySim values and the + configured category mappings. Keep the code field for joins. Use the label + field for presentation. +- A valid calculation can produce an empty CSV. That is distinct from a summary + marked unavailable because an input table or field was absent. Use the + summary manifest to identify the status. + +## Build Status + +The standard summarize workflow builds **85** tables. Of the **15** tables with +`Default build = no`, two are optional skim ECDF products and 13 are validation +contracts supplied by an external process through `summary_table_map`. The +visualizer does not calculate those contracts from `RunData`. The references +below describe all 100 contracts. + +## Summary Table Reference + +The tables are grouped by analytical category and identify the related dashboard +pages. Each row combines the user-facing data dictionary with developer details. +Category, analytical-use, logical input-type, and field-description text is +maintained in +[`scripts/summary_catalog_metadata.yaml`](../scripts/summary_catalog_metadata.yaml). +File names, exact output data types, build defaults, summary functions, and +declared input requirements come from the registered summary declarations. + +Required-input types give the canonical stored type where prepare guarantees +one and a logical or source-compatible type otherwise; they are not raw +ActivitySim file-type requirements. Output types are the exact stored CSV +contract. + +A few categories have additional conventions: + +- The Required Tables/Fields column reports each `@summary` declaration's + mechanical prerequisites. Builder-level aliases and conditional inputs can add + requirements that the declaration cannot express. For example, joint-tour + composition needs `composition` or `tour_composition`, composition by party + size also needs `number_of_participants` or `NUMBER_HH`, and person + participation by household size needs `HHSIZE` or `hhsize`. +- `jtf_distribution` declares only household inputs. On a one-day run, an + unavailable tour table can still produce an all-`No Joint Tours` result + instead of being marked unavailable. Verify tour availability before using + that distribution. +- Skimjoin diagnostics use `skim_scenario` for the selected mode or a hypothetical + sidecar mode. `all_records` identifies hypothetical values for all applicable + records, and each table also includes an all-modes group. +- Processor-built validation summaries that use optional assignment tables are valid + but empty when those inputs are absent. +- `tour_distance_by_tour_purpose` and + `average_nonmandatory_tour_distance_by_purpose_and_geography` need + `tours.SKIMDIST` to produce results, although their current declarations omit + it from the generated Required Tables/Fields column. This is a code-contract + gap. +- The 13 externally supplied validation contracts are registered for external CSV + input. Their builders do not calculate values, so the supplying workflow is + responsible for consistent units, period definitions, and values. + +Regenerate these tables and the standalone processor output reference after +changing summary metadata or declarations: + +```bash +uv run python scripts/generate_wiki_catalogs.py +``` + + +_Generated from `processor.summarize.catalog.SUMMARY_DEFINITIONS` and `scripts/summary_catalog_metadata.yaml`._ + +Registered summaries: **100**; built by default: **85**. + +### Population and Demographics + +_Related dashboard pages: Overview. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `household_size_distribution`
    `household_size_distribution.csv` | Household totals by modeled household size. Use it to profile household composition, calculate size shares, and compare population-synthesis results across runs. | `hh`: Prepared household table.
    • `HHSIZE` (`Int32`): Household person count clipped to 1 through 5, where 5 represents households of five or more people.
    • `finalweight` (`Float64`): Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback. | `household_size` (`Int64`): number of people in the household.
    `household_count` (`Float64`): weighted households in that size category. | `processor.summarize.summaries.demographics.hh_size`
    Default build: **yes** | +| `person_type_distribution`
    `person_type_distribution.csv` | Person totals by ActivitySim person type, with a display label. Use it to compare demographic market segments and as a denominator for person-type travel rates. | `per`: Prepared person table.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `person_type` (`String`): stable person-type code.
    `person_type_label` (`String`): configured readable label for the code.
    `person_count` (`Float64`): weighted people of that type. | `processor.summarize.summaries.demographics.person_type`
    Default build: **yes** | +| `population_totals`
    `population_totals.csv` | One run-level control-total row for people, households, person-tours, person-trips, and person-stops. Joint tour and trip records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback; sentinel, missing, or nonpositive fallback counts resolve to one. Use it for reasonableness checks and top-level comparisons; the measures use their own table weights and are not additive to one another. | `per`: Prepared person table.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback.

    `hh`: Prepared household table.
    • `finalweight` (`Float64`): Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback.

    `tours`: Prepared tour table.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights.

    `trips`: Prepared trip table.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback.
    • `stops` (`Int32`): Intermediate-stop indicator, 1 before the last trip on a half-tour and 0 for the final leg. | `person_count` (`Float64`): weighted persons.
    `household_count` (`Float64`): weighted households.
    `tour_count` (`Float64`): weighted person-tours; joint records are participant-expanded.
    `trip_count` (`Float64`): weighted person-trips; joint records are participant-expanded.
    `stop_count` (`Float64`): weighted person-trip records flagged as intermediate stops; joint records are participant-expanded. | `processor.summarize.summaries.demographics.population_totals`
    Default build: **yes** | + +### Person Attributes and Long-Term Choices + +_Related dashboard pages: Individual Choices, Mandatory Location Choice. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `license_holding_status_distribution`
    `license_holding_status_distribution.csv` | Licensed and unlicensed people age 16 or older by person type, including an all-person-types rollup. Use it to assess access to driving and explain auto-mode availability. | `per`: Prepared person table.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.
    • `has_license` (`Boolean-like`): Whether the person holds a driver's license.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback.
    • `age` (`integer`): Person age in years. | `person_type` (`String`): person-type code or `all_person_types` rollup.
    `license_holding_status` (`String`): `has_license` or `no_license`.
    `person_type_label` (`String`): configured readable person-type label.
    `person_count` (`Float64`): weighted people in the group. | `processor.summarize.summaries.long_term_person.license_holding_status`
    Default build: **yes** | +| `bicycle_comfort_level_distribution`
    `bicycle_comfort_level_distribution.csv` | Bicycle comfort categories by person type, including an all-person-types rollup. Use it to understand the population assumed willing to use different bicycle facilities. | `per`: Prepared person table.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.
    • `bike_comfort` (`category`): Modeled bicycle-comfort alternative.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `person_type` (`String`): person-type code or rollup.
    `bicycle_comfort_level` (`String`): prepared bicycle-comfort category.
    `person_type_label` (`String`): configured readable person-type label.
    `person_count` (`Float64`): weighted people in the group. | `processor.summarize.summaries.long_term_person.bicycle_comfort_level`
    Default build: **yes** | +| `transit_pass_ownership_by_person_type`
    `transit_pass_ownership_by_person_type.csv` | Transit-pass ownership status by person type, including an all-person-types rollup. Use it to evaluate transit market eligibility and pass-ownership model results. | `per`: Prepared person table.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.
    • `transit_pass_ownership` (`Boolean-like`): Whether the person owns a transit pass.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `person_type` (`String`): person-type code or rollup.
    `transit_pass_ownership_status` (`String`): `has_transit_pass` or `no_transit_pass`.
    `person_type_label` (`String`): configured readable person-type label.
    `person_count` (`Float64`): weighted people in the group. | `processor.summarize.summaries.long_term_person.transit_pass`
    Default build: **yes** | +| `transit_subsidy_by_person_type`
    `transit_subsidy_by_person_type.csv` | Transit-pass subsidy alternatives for workers, by person type and with an all-person-types rollup. Use it to examine employer or institutional transit-benefit assumptions. | `per`: Prepared person table.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.
    • `transit_pass_subsidy` (`category`): Modeled transit-pass-subsidy alternative.
    • `is_worker` (`String`): Whether the person is classified as a worker.
    • `is_student` (`Boolean-like`): Whether the person is classified as a student.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `person_type` (`String`): person-type code or rollup.
    `transit_subsidy_status` (`String`): prepared subsidy alternative code.
    `transit_subsidy_label` (`String`): configured readable subsidy label.
    `person_type_label` (`String`): configured readable person-type label.
    `person_count` (`Float64`): weighted eligible workers in the group. | `processor.summarize.summaries.long_term_person.transit_subsidy`
    Default build: **yes** | +| `telecommute_frequency_distribution`
    `telecommute_frequency_distribution.csv` | Non-work-from-home workers by telecommute-frequency category and home geography, plus an all-geographies rollup. Use it to analyze recurring telecommuting among workers who still have an external workplace. | `per`: Prepared person table.
    • `telecommute_frequency` (`category`): Modeled telecommute-frequency alternative.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback.
    • `is_worker` (`String`): Whether the person is classified as a worker.
    • `work_from_home` (`String`): Whether the worker has a home workplace.
    • `home_zone_id` (`identifier`): Prepared home-zone identifier. | `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `telecommute_frequency` (`String`): prepared telecommute-frequency alternative.
    `person_count` (`Float64`): weighted workers in the group. | `processor.summarize.summaries.long_term_person.telecommute`
    Default build: **yes** | + +### Household Vehicles and Vehicle Characteristics + +_Related dashboard pages: Vehicle Ownership and Type. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `autonomous_vehicle_ownership_totals`
    `autonomous_vehicle_ownership_totals.csv` | One run-level total of households modeled as owning an autonomous vehicle. Use it to report scenario penetration and compare AV-ownership assumptions. | `hh`: Prepared household table.
    • `av_ownership` (`Boolean-like`): Whether the household owns an autonomous vehicle.
    • `finalweight` (`Float64`): Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback. | `household_with_autonomous_vehicle_count` (`Float64`): weighted households with `av_ownership` true. | `processor.summarize.summaries.long_term_vehicle.av_ownership`
    Default build: **yes** | +| `auto_ownership_distribution`
    `auto_ownership_distribution.csv` | Household totals jointly classified by household size and vehicle count; household sizes of five or more are grouped as `5+`. Use it to assess motorization and auto sufficiency. | `hh`: Prepared household table.
    • `HHSIZE` (`Int32`): Household person count clipped to 1 through 5, where 5 represents households of five or more people.
    • `HHVEH` (`Int32`): Household vehicle count clipped to 0 through 4 for standard summary categories.
    • `finalweight` (`Float64`): Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback. | `household_size` (`String`): household-size category, with `5+` as the terminal group.
    `household_vehicle_count` (`Int64`): vehicles available to the household.
    `household_count` (`Float64`): weighted households in the joint category. | `processor.summarize.summaries.long_term_vehicle.auto_ownership`
    Default build: **yes** | +| `vehicle_age_distribution`
    `vehicle_age_distribution.csv` | Household vehicles by age, with ages 20 and older grouped as `20+`. Use it for fleet turnover, emissions, and technology analyses. | `vehicles`: Prepared household-vehicle table.
    • `vehicle_age` (`Int64`): Vehicle age in years parsed from vehicle_type.
    • `finalweight` (`Float64`): Vehicle expansion weight inherited from the owning household or set to 1.0 when unavailable. | `age` (`String`): vehicle age in years or `20+`.
    `vehicle_count` (`Float64`): weighted vehicles in the age category. | `processor.summarize.summaries.long_term_vehicle.vehicle_char_age`
    Default build: **yes** | +| `vehicle_fuel_type_distribution`
    `vehicle_fuel_type_distribution.csv` | Household vehicles by prepared fuel or powertrain type. Use it for fleet composition, energy, and emissions analysis. | `vehicles`: Prepared household-vehicle table.
    • `fuel_type` (`String`): Vehicle fuel or powertrain category parsed from vehicle_type.
    • `finalweight` (`Float64`): Vehicle expansion weight inherited from the owning household or set to 1.0 when unavailable. | `fuel_type` (`String`): prepared vehicle fuel/powertrain category.
    `vehicle_count` (`Float64`): weighted vehicles of that type. | `processor.summarize.summaries.long_term_vehicle.vehicle_char_fuel`
    Default build: **yes** | +| `vehicle_body_type_distribution`
    `vehicle_body_type_distribution.csv` | Household vehicles by prepared body type. Use it to characterize the light-duty fleet and support occupancy or emissions comparisons. | `vehicles`: Prepared household-vehicle table.
    • `body_type` (`String`): Vehicle body-style category parsed from vehicle_type.
    • `finalweight` (`Float64`): Vehicle expansion weight inherited from the owning household or set to 1.0 when unavailable. | `body_type` (`String`): prepared vehicle body-style category.
    `vehicle_count` (`Float64`): weighted vehicles of that type. | `processor.summarize.summaries.long_term_vehicle.vehicle_char_body`
    Default build: **yes** | + +### Long-Term Geography, Location, and Shadow Pricing + +_Related dashboard pages: Mandatory Location Choice, Employment\Enrollment Match By Geography, Park-and-Ride Location, Regional Validation. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `work_from_home_rate_by_geography`
    `work_from_home_rate_by_geography.csv` | All workers and work-from-home workers by home geography, plus a regional rollup. Divide the WFH count by the worker count to calculate the WFH rate and map its spatial pattern. | `per`: Prepared person table.
    • `is_worker` (`String`): Whether the person is classified as a worker.
    • `home_zone_id` (`identifier`): Prepared home-zone identifier.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `worker_count` (`Float64`): weighted workers living in the geography.
    `work_from_home_worker_count` (`Float64`): weighted workers flagged as working from home. | `processor.summarize.summaries.long_term_geography.wfh`
    Default build: **yes** | +| `internal_external_worker_by_geography`
    `internal_external_worker_by_geography.csv` | Internal and external workers by home geography, plus a regional rollup. Use it to understand external-worker incidence and its residential distribution. | `per`: Prepared person table.
    • `is_worker` (`String`): Whether the person is classified as a worker.
    • `is_external_worker` (`Boolean-like`): Whether the person's assigned workplace is external to the modeled area.
    • `home_zone_id` (`identifier`): Prepared home-zone identifier.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `internal_worker_count` (`Float64`): weighted workers with an internal workplace.
    `external_worker_count` (`Float64`): weighted workers classified as external. | `processor.summarize.summaries.long_term_geography.internal_vs_external`
    Default build: **yes** | +| `external_worker_workplace_locations`
    `external_worker_workplace_locations.csv` | External workers by their external workplace geography, plus a regional rollup. Use it to analyze external commute orientation; the all-worker denominator is repeated to support share calculations. | `per`: Prepared person table.
    • `is_worker` (`String`): Whether the person is classified as a worker.
    • `is_external_worker` (`Boolean-like`): Whether the person's assigned workplace is external to the modeled area.
    • `external_workplace_zone_id` (`identifier`): Assigned external workplace zone identifier.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `geography_type` (`String`): external-workplace geography system or rollup.
    `geography_id` (`String`): external-workplace geography identifier or rollup value.
    `external_worker_count` (`Float64`): weighted external workers assigned to that destination.
    `all_worker_count` (`Float64`): weighted workers in the full run, repeated on every row. | `processor.summarize.summaries.long_term_geography.external_workplace_loc`
    Default build: **yes** | +| `workplace_location_employment_comparison`
    `workplace_location_employment_comparison.csv` | Land-use employment and modeled worker workplace choices aligned by workplace geography. Use it to compare attraction targets with assigned workers and diagnose location-choice balance. | `land_use`: Prepared land-use and zone-attribute table.
    • `MAZ` (`Int64`): Canonical micro-analysis-zone identifier.
    • `employment_count` (`Float64`): Employment opportunities associated with the zone.

    `per`: Prepared person table.
    • `workplace_zone_id` (`identifier`): Assigned internal workplace-zone identifier.
    • `is_worker` (`String`): Whether the person is classified as a worker.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `geography_type` (`String`): workplace-geography system.
    `geography_id` (`String`): workplace-geography identifier.
    `employment_count` (`Float64`): employment opportunities from land use.
    `worker_count` (`Float64`): weighted workers assigned to the geography. | `processor.summarize.summaries.long_term_geography.workplace_vs_land_use_employment`
    Default build: **yes** | +| `commuting_flows`
    `commuting_flows.csv` | Worker flows from home geography to workplace geography at matching configured geography levels, plus a regional total. Use it as an origin-destination matrix for commute sheds, self-containment, and interjurisdictional flows. | `per`: Prepared person table.
    • `home_zone_id` (`identifier`): Prepared home-zone identifier.
    • `workplace_zone_id` (`identifier`): Assigned internal workplace-zone identifier.
    • `is_worker` (`String`): Whether the person is classified as a worker.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `origin_geography_type` (`String`): home-geography system.
    `origin_geography_id` (`String`): home-geography identifier.
    `destination_geography_type` (`String`): workplace-geography system.
    `destination_geography_id` (`String`): workplace-geography identifier.
    `commuter_count` (`Float64`): weighted workers in the OD pair. | `processor.summarize.summaries.long_term_geography.commuting_flows`
    Default build: **yes** | +| `school_location_enrollment_comparison`
    `school_location_enrollment_comparison.csv` | Land-use enrollment and modeled student school locations aligned by geography and student type. Use it to compare school-location targets with assigned students. | `land_use`: Prepared land-use and zone-attribute table.
    • `MAZ` (`Int64`): Canonical micro-analysis-zone identifier.
    • `enrollment_count` (`Float64`): Enrollment opportunities associated with the zone and student type.
    • `student_type` (`String`): Student or enrollment market segment.

    `per`: Prepared person table.
    • `school_zone_id` (`identifier`): Assigned school-zone identifier.
    • `is_student` (`Boolean-like`): Whether the person is classified as a student.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `geography_type` (`String`): school-geography system.
    `geography_id` (`String`): school-geography identifier.
    `student_type` (`String`): prepared school/enrollment market segment.
    `enrollment_count` (`Float64`): target enrollment from land use.
    `student_count` (`Float64`): weighted students assigned to the geography and type. | `processor.summarize.summaries.long_term_geography.school_loc_vs_land_use_enrollment`
    Default build: **yes** | +| `workplace_shadow_pricing_residuals`
    `workplace_shadow_pricing_residuals.csv` | Zone-level workplace target-versus-modeled residuals. Use positive residuals to find over-assigned workplace geographies and negative residuals to find under-assigned ones. | `land_use`: Prepared land-use and zone-attribute table.
    • `MAZ` (`Int64`): Canonical micro-analysis-zone identifier.
    • `employment_count` (`Float64`): Employment opportunities associated with the zone.

    `per`: Prepared person table.
    • `workplace_zone_id` (`identifier`): Assigned internal workplace-zone identifier.
    • `is_worker` (`String`): Whether the person is classified as a worker.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `geography_type` (`String`): workplace-geography system.
    `geography_id` (`String`): workplace-geography identifier.
    `target_count` (`Float64`): land-use employment target.
    `modeled_count` (`Float64`): weighted assigned workers.
    `residual_count` (`Float64`): `modeled_count - target_count`.
    `absolute_residual_count` (`Float64`): absolute residual magnitude.
    `percent_error` (`Float64`): residual divided by target, times 100; null when target is zero. | `processor.summarize.summaries.long_term_geography.workplace_shadow_pricing_residuals`
    Default build: **yes** | +| `school_shadow_pricing_residuals`
    `school_shadow_pricing_residuals.csv` | Zone-level school target-versus-modeled residuals by student type. Use it to diagnose school-location shadow-pricing convergence and segment-specific imbalance. | `land_use`: Prepared land-use and zone-attribute table.
    • `MAZ` (`Int64`): Canonical micro-analysis-zone identifier.
    • `enrollment_count` (`Float64`): Enrollment opportunities associated with the zone and student type.
    • `student_type` (`String`): Student or enrollment market segment.

    `per`: Prepared person table.
    • `school_zone_id` (`identifier`): Assigned school-zone identifier.
    • `is_student` (`Boolean-like`): Whether the person is classified as a student.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback.
    • `student_type` (`String`): Prepared school or enrollment market segment. | `geography_type` (`String`): school-geography system.
    `geography_id` (`String`): school-geography identifier.
    `student_type` (`String`): school/enrollment market segment.
    `target_count` (`Float64`): land-use enrollment target.
    `modeled_count` (`Float64`): weighted assigned students.
    `residual_count` (`Float64`): modeled minus target.
    `absolute_residual_count` (`Float64`): absolute residual magnitude.
    `percent_error` (`Float64`): residual divided by target, times 100; null for zero targets. | `processor.summarize.summaries.long_term_geography.school_shadow_pricing_residuals`
    Default build: **yes** | +| `workplace_shadow_pricing_residual_histogram`
    `workplace_shadow_pricing_residual_histogram.csv` | Distribution of workplace residuals by geography system. Use it to assess convergence across all zones without inspecting each zone separately; zero residuals receive their own zero-width bin. | `land_use`: Prepared land-use and zone-attribute table.
    • `MAZ` (`Int64`): Canonical micro-analysis-zone identifier.
    • `employment_count` (`Float64`): Employment opportunities associated with the zone.

    `per`: Prepared person table.
    • `workplace_zone_id` (`identifier`): Assigned internal workplace-zone identifier.
    • `is_worker` (`String`): Whether the person is classified as a worker.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `geography_type` (`String`): workplace-geography system.
    `bin_start` (`Float64`): inclusive lower residual bound.
    `bin_end` (`Float64`): upper residual bound; both bounds are zero for the exact-zero bin.
    `geography_count` (`Float64`): number of geography records in the bin. | `processor.summarize.summaries.long_term_geography.workplace_shadow_pricing_residual_histogram`
    Default build: **yes** | +| `school_shadow_pricing_residual_histogram`
    `school_shadow_pricing_residual_histogram.csv` | Distribution of school residuals by geography system and student type. Use it to compare convergence across student markets. | `land_use`: Prepared land-use and zone-attribute table.
    • `MAZ` (`Int64`): Canonical micro-analysis-zone identifier.
    • `enrollment_count` (`Float64`): Enrollment opportunities associated with the zone and student type.
    • `student_type` (`String`): Student or enrollment market segment.

    `per`: Prepared person table.
    • `school_zone_id` (`identifier`): Assigned school-zone identifier.
    • `is_student` (`Boolean-like`): Whether the person is classified as a student.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback.
    • `student_type` (`String`): Prepared school or enrollment market segment. | `geography_type` (`String`): school-geography system.
    `student_type` (`String`): school/enrollment market segment.
    `bin_start` (`Float64`): lower residual bound.
    `bin_end` (`Float64`): upper residual bound, or zero for the exact-zero bin.
    `geography_count` (`Float64`): number of geography/student-type records in the bin. | `processor.summarize.summaries.long_term_geography.school_shadow_pricing_residual_histogram`
    Default build: **yes** | +| `park_and_ride_location_residuals`
    `park_and_ride_location_residuals.csv` | Modeled park-and-ride tour use compared with lot capacity by lot geography. Use it to identify over-capacity or underused PNR locations. | `tours`: Prepared tour table.
    • `tour_mode` (`String`): Main mode selected for the tour.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights.

    `land_use`: Prepared land-use and zone-attribute table. | `geography_type` (`String`): PNR-lot geography system.
    `geography_id` (`String`): PNR-lot geography identifier.
    `pnr_tour_count` (`Float64`): weighted PNR tours assigned to the location.
    `pnr_lot_capacity` (`Float64`): supplied lot capacity target.
    `residual_count` (`Float64`): tours minus capacity.
    `absolute_residual_count` (`Float64`): absolute residual magnitude.
    `percent_error` (`Float64`): residual divided by capacity, times 100; null for zero capacity. | `processor.summarize.summaries.long_term_geography.park_and_ride_location_residuals`
    Default build: **yes** | +| `park_and_ride_location_residual_histogram`
    `park_and_ride_location_residual_histogram.csv` | Distribution of PNR use-minus-capacity residuals by geography system. Use it for systemwide capacity-fit assessment. | `tours`: Prepared tour table.
    • `tour_mode` (`String`): Main mode selected for the tour.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights.

    `land_use`: Prepared land-use and zone-attribute table. | `geography_type` (`String`): PNR-lot geography system.
    `bin_start` (`Float64`): lower residual bound.
    `bin_end` (`Float64`): upper residual bound, or zero for the exact-zero bin.
    `geography_count` (`Float64`): number of PNR geography records in the bin. | `processor.summarize.summaries.long_term_geography.park_and_ride_location_residual_histogram`
    Default build: **yes** | +| `free_parking_eligibility_by_workplace_geography`
    `free_parking_eligibility_by_workplace_geography.csv` | Workers with and without free workplace parking by workplace geography. Use it to analyze parking-cost exposure and its effect on commute mode choice. | `per`: Prepared person table.
    • `is_worker` (`String`): Whether the person is classified as a worker.
    • `free_parking_at_work` (`Boolean-like or category`): Free-workplace-parking eligibility or modeled alternative.
    • `workplace_zone_id` (`identifier`): Assigned internal workplace-zone identifier.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `geography_type` (`String`): workplace-geography system.
    `geography_id` (`String`): workplace-geography identifier.
    `workers_without_free_parking_count` (`Float64`): weighted workers not eligible for free parking.
    `workers_with_free_parking_count` (`Float64`): weighted workers eligible for free parking. | `processor.summarize.summaries.long_term_geography.free_parking`
    Default build: **yes** | + +### Long-Term Location Distance + +_Related dashboard pages: Mandatory Location Choice. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `work_location_distance_distribution_by_geography`
    `work_location_distance_distribution_by_geography.csv` | Internal, non-work-from-home workers with valid workplace locations by home geography and home-to-work distance. Use it for commute-length distributions and spatial comparisons. | `per`: Prepared person table.
    • `distance_to_work` (`Float64`): Prepared home-to-work distance in the configured distance units.
    • `external_workplace_zone_id` (`identifier`): Assigned external workplace zone identifier.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback.
    • `is_external_worker` (`Boolean-like`): Whether the person's assigned workplace is external to the modeled area.
    • `is_worker` (`String`): Whether the person is classified as a worker.
    • `work_from_home` (`String`): Whether the worker has a home workplace.
    • `workplace_zone_id` (`identifier`): Assigned internal workplace-zone identifier. | `distance_bin` (`String`): exact-zero, positive-sub-mile, whole-mile lower-bound, or `51+` category.
    `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `person_count` (`Float64`): weighted workers in the bin. | `processor.summarize.summaries.long_term_distance.work_tlfd`
    Default build: **yes** | +| `university_location_distance_distribution_by_geography`
    `university_location_distance_distribution_by_geography.csv` | University students, identified by person type 3, by home geography and home-to-school distance. Use it to examine university travel markets and campus catchments. | `per`: Prepared person table.
    • `distance_to_school` (`Float64`): Prepared home-to-school distance in the configured distance units.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `distance_bin` (`String`): exact-zero, positive-sub-mile, whole-mile lower-bound, or `51+` category.
    `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `person_count` (`Float64`): weighted university students in the bin. | `processor.summarize.summaries.long_term_distance.univ_tlfd`
    Default build: **yes** | +| `school_location_distance_distribution_by_geography`
    `school_location_distance_distribution_by_geography.csv` | School students, identified by person types 6 and higher, by home geography and home-to-school distance. Use it for K--12 travel-distance and school-catchment analysis. | `per`: Prepared person table.
    • `distance_to_school` (`Float64`): Prepared home-to-school distance in the configured distance units.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `distance_bin` (`String`): exact-zero, positive-sub-mile, whole-mile lower-bound, or `51+` category.
    `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup value.
    `person_count` (`Float64`): weighted school students in the bin. | `processor.summarize.summaries.long_term_distance.schl_tlfd`
    Default build: **yes** | + +### Daily Activity Patterns and Travel Rates + +_Related dashboard pages: Daily Activity Pattern. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `daily_activity_pattern_by_person_type`
    `daily_activity_pattern_by_person_type.csv` | Daily activity pattern alternatives by person type, using each person's non-null person-table CDAP value and falling back only for remaining persons to all observed day rows with day weights when available, including an all-person-types rollup. No day rows are deduplicated. Use it to compare mandatory, nonmandatory, and home-stay behavior. | `per`: Prepared person table.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `person_type` (`String`): person-type code or rollup.
    `daily_activity_pattern` (`String`): prepared CDAP/activity-pattern category.
    `person_count` (`Float64`): weighted people or observed person-days in the pattern. | `processor.summarize.summaries.daily_travel_activity.dap_summary`
    Default build: **yes** | +| `mandatory_tour_frequency_by_person_type`
    `mandatory_tour_frequency_by_person_type.csv` | Positive mandatory-tour-frequency choice by person type, using each person's non-null prepared choice and falling back only for remaining persons to choices derived from work and school tours on every person-day, with day weights when available, plus an all-person-types rollup. Use it to analyze how many mandatory tours travelers make. The table excludes records with a choice of zero. | `per`: Prepared person table.
    • `person_id` (`Int64`): Canonical person identifier.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback. | `person_type` (`String`): person-type code or rollup.
    `mandatory_tour_frequency` (`Int32`): prepared or tour-derived positive mandatory-tour frequency alternative.
    `person_count` (`Float64`): weighted people or person-days choosing that frequency. | `processor.summarize.summaries.daily_travel_activity.mandatory_tour_freq`
    Default build: **yes** | +| `nonmandatory_tour_frequency_by_person_type`
    `nonmandatory_tour_frequency_by_person_type.csv` | Count of individual nonmandatory tours plus joint-tour participation, grouped as 0, 1, 2, or 3+, by person type and for all types. Genuinely multi-day inputs are classified per person-day only when every tour and participant has a safe day key; otherwise the one-day/person fallback is retained rather than dropping records or guessing keys. Use it to compare discretionary travel propensity. | `per`: Prepared person table.
    • `person_id` (`Int64`): Canonical person identifier.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback.

    `tours`: Prepared tour table.
    • `person_id` (`Int64`): Canonical person identifier associated with an individual tour.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.

    `joint_participants`: Prepared joint-tour participant table.
    • `person_id` (`Int64`): Canonical identifier of a person participating in a joint tour. | `person_type` (`String`): person-type code or rollup.
    `nonmandatory_tour_frequency` (`String`): combined nonmandatory-tour category `0`, `1`, `2`, or `3+`.
    `person_count` (`Float64`): weighted people or person-days in the category. | `processor.summarize.summaries.daily_travel_activity.indiv_nm_summary`
    Default build: **yes** | +| `tour_rates_by_person_type_and_tour_purpose`
    `tour_rates_by_person_type_and_tour_purpose.csv` | Person-tours per person-day by person type and tour purpose under the selected weighting mode, plus all-person-types rates. Every eligible source row contributes and at-work subtours are excluded. Joint tours are attributed using each known participant's person weight or expanded by a valid party size when participant records are unavailable. Use it to compare tour-generation rates while controlling for population composition. | `per`: Prepared person table.
    • `person_id` (`Int64`): Canonical person identifier.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback.

    `tours`: Prepared tour table.
    • `person_id` (`Int64`): Canonical person identifier associated with an individual tour.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `person_type` (`String`): person-type code or rollup.
    `tour_purpose` (`String`): prepared tour-purpose category.
    `tour_rate` (`Float64`): sum of eligible person-tour weights, divided by summed person or person-day weights for that person type. | `processor.summarize.summaries.daily_travel_activity.tour_rate_per_person`
    Default build: **yes** | +| `trip_rates_by_person_type_and_trip_purpose`
    `trip_rates_by_person_type_and_trip_purpose.csv` | Person-trips per person-day by person type and trip purpose under the selected weighting mode, plus all-person-types rates. Every eligible source row contributes. Joint trips are attributed using each known participant's person weight or expanded by a valid party size when participant records are unavailable. Use it to compare trip-generation rates across demographic markets. | `per`: Prepared person table.
    • `person_id` (`Int64`): Canonical person identifier.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback.

    `trips`: Prepared trip table.
    • `person_id` (`Int64`): Canonical person identifier associated with the trip.
    • `trip_purpose` (`String`): Destination purpose of the trip leg.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `person_type` (`String`): person-type code or rollup.
    `trip_purpose` (`String`): destination purpose of the trip.
    `trip_rate` (`Float64`): sum of eligible person-trip weights, divided by summed person or person-day weights for that person type. | `processor.summarize.summaries.daily_travel_activity.trip_rate_per_person`
    Default build: **yes** | + +### School Escorting + +_Related dashboard pages: Escorted Tours. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `escorted_tour_totals`
    `escorted_tour_totals.csv` | One run-level total of adult-side tours with an outbound or inbound school-escort condition. Use it as the top-level escorted-tour control total. | `tours`: Prepared tour table.
    • `school_esc_outbound` (`String`): Outbound school-escort alternative or status.
    • `school_esc_inbound` (`String`): Inbound school-escort alternative or status.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `tour_count` (`Float64`): weighted distinct eligible tours with at least one escorted direction. | `processor.summarize.summaries.daily_travel_escort_counts.total_escorted_tours`
    Default build: **yes** | +| `school_escorted_tours_by_escort_type_and_direction`
    `school_escorted_tours_by_escort_type_and_direction.csv` | Adult-side escorted tours by escort arrangement and direction, with an `all_directions` incidence rollup. Use it to compare ride-share and pure-escort patterns. | `tours`: Prepared tour table.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `school_esc_outbound` (`String`): Outbound school-escort alternative or status.
    • `school_esc_inbound` (`String`): Inbound school-escort alternative or status.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `escort_type` (`String`): prepared escort arrangement label.
    `direction` (`String`): `outbound`, `inbound`, or `all_directions`.
    `tour_count` (`Float64`): weighted escorted-tour incidences. | `processor.summarize.summaries.daily_travel_escort_counts.escorted_tours_to_from_school`
    Default build: **yes** | +| `adult_escorted_tour_purposes_by_direction`
    `adult_escorted_tour_purposes_by_direction.csv` | Purposes of the adult tours that do school escorting, by direction and with an all-directions incidence rollup. Use it to see how escorting connects with work or other adult activities. | `tours`: Prepared tour table.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `school_esc_outbound` (`String`): Outbound school-escort alternative or status.
    • `school_esc_inbound` (`String`): Inbound school-escort alternative or status.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `tour_purpose` (`String`): adult tour's primary purpose.
    `direction` (`String`): escorted half or `all_directions`.
    `tour_count` (`Float64`): weighted escorted-tour incidences. | `processor.summarize.summaries.daily_travel_escort_counts.adult_escorted_tour_purposes_by_direction`
    Default build: **yes** | +| `adult_escorted_tours_by_person_type_and_direction`
    `adult_escorted_tours_by_person_type_and_direction.csv` | Adult-side escorted tours by the adult traveler's person type and escorted direction. Use it to identify who performs school escorting. | `tours`: Prepared tour table.
    • `school_esc_outbound` (`String`): Outbound school-escort alternative or status.
    • `school_esc_inbound` (`String`): Inbound school-escort alternative or status.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `person_type` (`String`): adult traveler person-type code.
    `direction` (`String`): `outbound`, `inbound`, or `both`.
    `tour_count` (`Float64`): weighted tours meeting that directional condition. | `processor.summarize.summaries.daily_travel_escort_counts.adult_escorted_tours_by_person_type_and_direction`
    Default build: **yes** | +| `student_school_escort_status_by_direction`
    `student_school_escort_status_by_direction.csv` | Student school tours classified by normalized escort type for each direction and for tours escorted both ways. Use it to measure the student-side escort experience. | `tours`: Prepared tour table.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `school_esc_outbound` (`String`): Outbound school-escort alternative or status.
    • `school_esc_inbound` (`String`): Inbound school-escort alternative or status.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `direction` (`String`): `outbound`, `inbound`, or `both`.
    `escort_type` (`String`): normalized escort arrangement, including unescorted alternatives where present.
    `tour_count` (`Float64`): weighted student school tours in the group. | `processor.summarize.summaries.daily_travel_escort_counts.student_school_escort_status_by_direction`
    Default build: **yes** | +| `student_households_by_student_count`
    `student_households_by_student_count.csv` | Households by the number of school-age/student household members recognized by the escort logic. Use it as a denominator for household escort participation. | `hh`: Prepared household table.
    • `household_id` (`Int64`): Canonical household identifier used to join people, tours, trips, days, and vehicles.
    • `finalweight` (`Float64`): Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback.

    `per`: Prepared person table.
    • `household_id` (`Int64`): Canonical household identifier for the person.
    • `person_type` (`source-compatible category`): ActivitySim person-type code. | `student_count` (`Int64`): students in the household.
    `household_count` (`Float64`): weighted households with that count. | `processor.summarize.summaries.daily_travel_escort_counts.student_households_by_student_count`
    Default build: **yes** | +| `households_with_school_escorting_by_student_count_and_direction`
    `households_with_school_escorting_by_student_count_and_direction.csv` | Unique households with at least one escorted student school tour, by number of students and directional condition. Use it to calculate escort-participation rates by household composition. | `hh`: Prepared household table.
    • `household_id` (`Int64`): Canonical household identifier used to join people, tours, trips, days, and vehicles.
    • `finalweight` (`Float64`): Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback.

    `per`: Prepared person table.
    • `household_id` (`Int64`): Canonical household identifier for the person.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.

    `tours`: Prepared tour table.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `school_esc_outbound` (`String`): Outbound school-escort alternative or status.
    • `school_esc_inbound` (`String`): Inbound school-escort alternative or status.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `student_count` (`Int64`): students in the household.
    `direction` (`String`): `outbound`, `inbound`, or `both`.
    `household_count` (`Float64`): weighted unique households meeting the condition. | `processor.summarize.summaries.daily_travel_escort_counts.households_with_school_escorting_by_student_count_and_direction`
    Default build: **yes** | +| `schoolkids_per_escorted_tour_by_student_count_and_direction`
    `schoolkids_per_escorted_tour_by_student_count_and_direction.csv` | Average number of escorted children on adult-side escorted tours by household student count and direction. Use it to analyze escorting efficiency and child grouping. | `hh`: Prepared household table.
    • `household_id` (`Int64`): Canonical household identifier used to join people, tours, trips, days, and vehicles.
    • `finalweight` (`Float64`): Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback.

    `per`: Prepared person table.
    • `household_id` (`Int64`): Canonical household identifier for the person.
    • `person_type` (`source-compatible category`): ActivitySim person-type code.

    `tours`: Prepared tour table.
    • `school_esc_outbound` (`String`): Outbound school-escort alternative or status.
    • `school_esc_inbound` (`String`): Inbound school-escort alternative or status.
    • `num_escortees` (`Int64`): Number of students escorted on the tour.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `student_count` (`Int64`): students in the adult traveler's household.
    `direction` (`String`): `outbound`, `inbound`, or `both`.
    `avg_schoolkids_per_tour` (`Float64`): weighted mean number of escortees per eligible tour.
    `tour_count` (`Float64`): weighted eligible tours used as the mean denominator. | `processor.summarize.summaries.daily_travel_escort_counts.schoolkids_per_escorted_tour_by_student_count_and_direction`
    Default build: **yes** | +| `adult_escorted_tour_distance_distribution_by_direction`
    `adult_escorted_tour_distance_distribution_by_direction.csv` | Adult-side escorted tours by lower-bound tour-distance bin and directional escort condition. Use it to compare the length of outbound-only, inbound-only, and both-way escort tours. | `tours`: Prepared tour table.
    • `SKIMDIST` (`Float64`): Prepared tour distance in the configured skim distance units.
    • `school_esc_outbound` (`String`): Outbound school-escort alternative or status.
    • `school_esc_inbound` (`String`): Inbound school-escort alternative or status.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `distance_bin` (`String`): exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`.
    `direction` (`String`): `outbound`, `inbound`, or `both`.
    `tour_count` (`Float64`): weighted eligible tours in the bin. | `processor.summarize.summaries.daily_travel_escort_distributions.adult_escorted_tour_distance_distribution_by_direction`
    Default build: **yes** | +| `adult_escorted_trip_distance_distribution_by_direction`
    `adult_escorted_trip_distance_distribution_by_direction.csv` | Trips on adult tours marked as escorted, by outbound or inbound half and lower-bound trip-distance bin. Use it to examine the trip-leg distance for escorting. | `tours`: Prepared tour table.
    • `tour_id` (`Int64`): Canonical tour identifier.
    • `school_esc_outbound` (`String`): Outbound school-escort alternative or status.
    • `school_esc_inbound` (`String`): Inbound school-escort alternative or status.

    `trips`: Prepared trip table.
    • `tour_id` (`Int64`): Canonical identifier of the parent tour.
    • `od_dist` (`Float64`): Origin-to-destination trip distance from the configured distance skim or supported survey distance, in source units.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `distance_bin` (`String`): exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`.
    `direction` (`String`): `outbound`, `inbound`, or `both` condition.
    `trip_count` (`Float64`): weighted eligible trips in the bin. | `processor.summarize.summaries.daily_travel_escort_distributions.adult_escorted_trip_distance_distribution_by_direction`
    Default build: **yes** | +| `adult_escort_event_stop_distribution`
    `adult_escort_event_stop_distribution.csv` | Intermediate stops before and after school drop-off or pickup on adult tours marked as escorted. Use it to analyze trip chains around escort events. | `tours`: Prepared tour table.
    • `tour_id` (`Int64`): Canonical tour identifier.
    • `school_esc_outbound` (`String`): Outbound school-escort alternative or status.
    • `school_esc_inbound` (`String`): Inbound school-escort alternative or status.

    `trips`: Prepared trip table.
    • `tour_id` (`Int64`): Canonical identifier of the parent tour.
    • `escort_event_role` (`String`): Adult escort-event role assigned to the trip.
    • `escort_stops_before_event` (`Int32`): Stops occurring before the associated escort event.
    • `escort_stops_after_event` (`Int32`): Stops occurring after the associated escort event.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `segment` (`String`): one of `outbound_before_dropoff`, `outbound_after_dropoff`, `inbound_before_pickup`, or `inbound_after_pickup`.
    `stop_count` (`Int32`): prepared count of stops in that segment.
    `tour_count` (`Float64`): weighted escort-event records with that stop count. | `processor.summarize.summaries.daily_travel_escort_distributions.adult_escort_event_stop_distribution`
    Default build: **yes** | +| `adult_escort_trip_stop_frequency`
    `adult_escort_trip_stop_frequency.csv` | Adult-side escorted tours jointly classified by purpose and outbound, inbound, and total stop counts. Use it to compare stop-making complexity on escort tours. | `tours`: Prepared tour table.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `school_esc_outbound` (`String`): Outbound school-escort alternative or status.
    • `school_esc_inbound` (`String`): Inbound school-escort alternative or status.
    • `num_ob_stops` (`Int32`): Number of outbound intermediate stops parsed from stop_frequency.
    • `num_ib_stops` (`Int32`): Number of inbound intermediate stops parsed from stop_frequency.
    • `num_tot_stops` (`Int32`): Total intermediate stops on the tour.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `tour_purpose` (`String`): adult tour purpose.
    `outbound_stop_count` (`Int32`): outbound stops capped at 3.
    `inbound_stop_count` (`Int32`): inbound stops capped at 3.
    `total_stop_count` (`Int32`): total stops capped at 6.
    `tour_count` (`Float64`): weighted escorted tours in the combination. | `processor.summarize.summaries.daily_travel_escort_distributions.adult_escort_trip_stop_frequency`
    Default build: **yes** | + +### Joint Travel + +_Related dashboard pages: Joint Travel. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `jtf_distribution`
    `jtf_distribution.csv` | Household observations across the legacy 21 joint-tour-frequency labels, selected by minimum counts in five fixed purpose slots covering shopping, maintenance, eating out, visiting/social, and other discretionary. Purposes outside those slots do not affect the label, and counts above a label's threshold are not rejected, so the labels are not strict total-tour counts. When the day table shows a genuinely multi-day diary and tours contain `day_num`, observations use `(household_id, day_num)`; null or unmatched tour days do not contribute. Without the tour day column, the result is empty rather than pooled into a weekly choice. On a one-day input, the declaration does not require tours, so an unavailable tour table can instead classify every household as `No Joint Tours`; verify tour availability before using that result. Use it to validate compatible joint-tour generation inputs. | `hh`: Prepared household table.
    • `household_id` (`Int64`): Canonical household identifier used to join people, tours, trips, days, and vehicles.
    • `finalweight` (`Float64`): Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback. | `jtf_code` (`Int32`): integer alternative 1--21.
    `jtf_label` (`String`): readable minimum-frequency/purpose-combination label.
    `household_count` (`Float64`): sum of `hh.finalweight` across households or household-day observations assigned to the alternative. | `processor.summarize.summaries.joint_travel.joint_tour_freq`
    Default build: **yes** | +| `joint_tours_by_household_size`
    `joint_tours_by_household_size.csv` | All household observations and observations making at least one joint tour by household size. A genuinely multi-day diary uses household-days when tours contain `day_num`; null or unmatched tour days do not contribute, and a missing tour day column produces an empty result rather than pooling days. Valid shared joint-tour or tour IDs collapse repeated records; records without either valid ID remain distinct. Use the two counts to calculate joint-tour participation rates. | `tours`: Prepared tour table.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `household_id` (`Int64`): Canonical household identifier associated with the tour.

    `hh`: Prepared household table.
    • `household_id` (`Int64`): Canonical household identifier used to join people, tours, trips, days, and vehicles.
    • `HHSIZE` (`Int32`): Household person count clipped to 1 through 5, where 5 represents households of five or more people.
    • `finalweight` (`Float64`): Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback. | `household_size` (`Int32`): number of household members.
    `household_count` (`Float64`): sum of `hh.finalweight` across households or household-day observations of that size.
    `joint_tour_hh_count` (`Float64`): sum of `hh.finalweight` for those observations with a joint tour. | `processor.summarize.summaries.joint_travel.joint_tours_hhsize`
    Default build: **yes** | +| `joint_tour_party_size_distribution`
    `joint_tour_party_size_distribution.csv` | Household joint tours by valid participant count, with parties of five or more stored in bin 5. Verified survey participant-row groups retain every row at a fractional representation weight that sums to one household tour; ordinary model joint-tour rows retain full weight. Sentinel values of 995 or greater, missing values, and nonpositive values are excluded. Use it to assess joint-tour occupancy. | `tours`: Prepared tour table.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `NUMBER_HH` (`Int32`): Joint-party size from participant rows, number_of_participants, an existing NUMBER_HH value, or a fallback of one.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `party_size` (`Int32`): household participants; value 5 represents `5+`.
    `joint_tour_count` (`Float64`): weighted joint tours in the party-size bin. | `processor.summarize.summaries.joint_travel.joint_party_size`
    Default build: **yes** | +| `joint_tour_composition_distribution`
    `joint_tour_composition_distribution.csv` | Household joint tours by prepared party-composition category. Verified participant-row representations retain every row fractionally so each complete group contributes one household tour. Use it to compare adult-only, child-inclusive, and other modeled compositions. | `tours`: Prepared tour table.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `tour_composition` (`String`): prepared joint-party composition.
    `joint_tour_count` (`Float64`): weighted joint tours in the category. | `processor.summarize.summaries.joint_travel.joint_composition`
    Default build: **yes** | +| `joint_tour_composition_by_party_size`
    `joint_tour_composition_by_party_size.csv` | Household joint tours jointly classified by party composition and exact valid participant count. Verified participant-row representations retain every row fractionally so each complete group contributes one household tour. Sentinel values of 995 or greater, missing values, and nonpositive values are excluded. Use it to study how household makeup and group size interact. | `tours`: Prepared tour table.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `tour_composition` (`String`): prepared party-composition category.
    `party_size` (`Int64`): number of tour participants.
    `joint_tour_count` (`Float64`): weighted joint tours in the combination. | `processor.summarize.summaries.joint_travel.joint_composition_by_party_size`
    Default build: **yes** | +| `person_jtp_by_household_size`
    `person_jtp_by_household_size.csv` | All person observations and observations participating in one or more joint tours by household size. When a multi-day input has person-day records and each joint-tour identity maps to one day, the observations are person-days and participation comes from linked joint participants. A missing or ambiguous tour-day mapping produces an empty result; if the day table lacks household, person, or day fields, the person-level fallback remains. The multi-day path's participant-table limitations are documented in chapter 25. Use the two counts to calculate participation rates. | `per`: Prepared person table.
    • `household_id` (`Int64`): Canonical household identifier for the person.
    • `person_id` (`Int64`): Canonical person identifier.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback.

    `hh`: Prepared household table.
    • `household_id` (`Int64`): Canonical household identifier used to join people, tours, trips, days, and vehicles. | `household_size` (`Int64`): size of the person's household.
    `joint_tour_person_count` (`Float64`): sum of the applicable person or day weight for people or person-days participating in a joint tour.
    `total_person_count` (`Float64`): sum of the applicable person or day weight for people or person-days in households of that size. | `processor.summarize.summaries.joint_travel.joint_participation_person_by_hhsize`
    Default build: **yes** | +| `household_jtp_by_household_size_and_jtf`
    `household_jtp_by_household_size_and_jtf.csv` | For household observations of size two or more, weighted percentage distribution across 0, 1, and 2+ joint-tour identities within each household size. A genuinely multi-day diary uses household-days when tours contain `day_num`; null or unmatched tour days do not contribute, and a missing tour day column produces an empty result rather than pooling days. Valid shared joint-tour or tour IDs collapse repeated records; records without either valid ID remain distinct. Use it to compare joint-tour propensity independent of household-size totals across compatible runs. | `hh`: Prepared household table.
    • `household_id` (`Int64`): Canonical household identifier used to join people, tours, trips, days, and vehicles.
    • `HHSIZE` (`Int32`): Household person count clipped to 1 through 5, where 5 represents households of five or more people.
    • `finalweight` (`Float64`): Household expansion weight from the configured weight, inverse sample rate, or 1.0 fallback.

    `tours`: Prepared tour table.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `household_id` (`Int64`): Canonical household identifier associated with the tour. | `jtf` (`String`): joint-tour count category `0`, `1`, or `2+`.
    `household_size` (`String`): household size as a category.
    `household_percent` (`Float64`): percent of summed `hh.finalweight` for household observations of that size in the JTF category. | `processor.summarize.summaries.joint_travel.jtf_by_hhsize`
    Default build: **yes** | + +### Basic Tour Distributions + +_Related dashboard pages: Tour Purpose. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `tour_category_distribution`
    `tour_category_distribution.csv` | Household-tour-unit counts by ActivitySim category, such as mandatory, nonmandatory, at-work, or joint. Verified participant-row representations of one joint tour are retained fractionally rather than collapsed. Use it for high-level tour-system composition. | `tours`: Prepared tour table.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `tour_category` (`String`): prepared tour category.
    `tour_count` (`Float64`): weighted tours in the category. | `processor.summarize.summaries.tour.tour_category`
    Default build: **yes** | +| `tour_purpose_distribution`
    `tour_purpose_distribution.csv` | Household-tour-unit counts by configured summary purpose. Verified participant-row representations of one joint tour are retained fractionally rather than collapsed. Use it to compare the volume and share of work, school, escort, shopping, and other travel. | `tours`: Prepared tour table.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `tour_purpose` (`String`): canonical summary tour purpose.
    `tour_count` (`Float64`): weighted tours for that purpose. | `processor.summarize.summaries.tour.tour_purpose`
    Default build: **yes** | + +### Vehicles Allocated to Tours + +_Related dashboard pages: Tour Mode. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `allocated_vehicle_age_by_occupancy`
    `allocated_vehicle_age_by_occupancy.csv` | Allocated vehicle ages by occupancy condition. Use it to analyze how fleet age is associated with single- and shared-occupant travel. | `tours`: Prepared tour table.
    • `vehicle_occup_1` (`vehicle-type code`): Vehicle type allocated to single-occupant auto tours.
    • `vehicle_occup_2` (`vehicle-type code`): Vehicle type allocated to two-occupant auto tours.
    • `vehicle_occup_3.5` (`vehicle-type code`): Vehicle type allocated to auto tours with three or more occupants.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `age` (`String`): decoded vehicle age, with `20+` terminal.
    `occupancy` (`String`): allocation condition `1`, `2`, or `3+`.
    `vehicle_count` (`Float64`): weighted tour allocation incidences. | `processor.summarize.summaries.tour_vehicles.allocated_vehicle_age`
    Default build: **yes** | +| `allocated_vehicle_fuel_type_by_occupancy`
    `allocated_vehicle_fuel_type_by_occupancy.csv` | Allocated vehicle fuel/powertrain type by occupancy condition. Use it for energy or emissions segmentation of auto travel. | `tours`: Prepared tour table.
    • `vehicle_occup_1` (`vehicle-type code`): Vehicle type allocated to single-occupant auto tours.
    • `vehicle_occup_2` (`vehicle-type code`): Vehicle type allocated to two-occupant auto tours.
    • `vehicle_occup_3.5` (`vehicle-type code`): Vehicle type allocated to auto tours with three or more occupants.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `fuel_type` (`String`): decoded fuel/powertrain category.
    `occupancy` (`String`): allocation condition `1`, `2`, or `3+`.
    `vehicle_count` (`Float64`): weighted tour allocation incidences. | `processor.summarize.summaries.tour_vehicles.allocated_vehicle_fuel`
    Default build: **yes** | +| `allocated_vehicle_body_type_by_occupancy`
    `allocated_vehicle_body_type_by_occupancy.csv` | Allocated vehicle body type by occupancy condition. Use it to relate party size to the modeled vehicle used. | `tours`: Prepared tour table.
    • `vehicle_occup_1` (`vehicle-type code`): Vehicle type allocated to single-occupant auto tours.
    • `vehicle_occup_2` (`vehicle-type code`): Vehicle type allocated to two-occupant auto tours.
    • `vehicle_occup_3.5` (`vehicle-type code`): Vehicle type allocated to auto tours with three or more occupants.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `body_type` (`String`): decoded vehicle body-style category.
    `occupancy` (`String`): allocation condition `1`, `2`, or `3+`.
    `vehicle_count` (`Float64`): weighted tour allocation incidences. | `processor.summarize.summaries.tour_vehicles.allocated_vehicle_body`
    Default build: **yes** | + +### Tour Mode, Stops, Time, and Distance + +_Related dashboard pages: Tour Mode, Tour Time, Tour Distance, Tour Stop Frequency. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `tour_mode_by_tour_purpose_and_auto_sufficiency`
    `tour_mode_by_tour_purpose_and_auto_sufficiency.csv` | Person-tour mode counts by purpose and household auto sufficiency, with all-purpose rows. Joint records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback; sentinel, missing, or nonpositive fallback sizes resolve to one. Use it to compare mode choice across vehicle-availability markets. | `tours`: Prepared tour table.
    • `tour_mode` (`String`): Main mode selected for the tour.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `NUMBER_HH` (`Int32`): Joint-party size from participant rows, number_of_participants, an existing NUMBER_HH value, or a fallback of one.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights.
    • `AUTOSUFF` (`Int32`): Prepared household auto-sufficiency category for the tour. | `tour_mode` (`String`): prepared tour mode.
    `tour_purpose` (`String`): tour purpose or `all_tour_purposes`.
    `tour_count_zero_auto` (`Float64`): weighted person-tours from zero-auto households.
    `tour_count_auto_deficient` (`Float64`): weighted person-tours from households with fewer autos than workers.
    `tour_count_auto_sufficient` (`Float64`): weighted person-tours from auto-sufficient households.
    `tour_count_all_households` (`Float64`): sum of the three auto-sufficiency counts. | `processor.summarize.summaries.tour_profiles.tour_mode`
    Default build: **yes** | +| `tour_stop_frequency_by_tour_purpose`
    `tour_stop_frequency_by_tour_purpose.csv` | Household-tour-unit counts jointly classified by purpose and outbound, inbound, and total intermediate-stop counts. Verified participant-row joint-tour groups retain every row at fractional weight summing to one tour. Use it to measure tour complexity and stop-generation patterns. | `tours`: Prepared tour table.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `num_ob_stops` (`Int32`): Number of outbound intermediate stops parsed from stop_frequency.
    • `num_ib_stops` (`Int32`): Number of inbound intermediate stops parsed from stop_frequency.
    • `num_tot_stops` (`Int32`): Total intermediate stops on the tour.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `tour_purpose` (`String`): canonical tour purpose.
    `outbound_stop_count` (`Int32`): outbound stops capped at 3.
    `inbound_stop_count` (`Int32`): inbound stops capped at 3.
    `total_stop_count` (`Int32`): total stops capped at 6.
    `tour_count` (`Float64`): weighted tours in the combination. | `processor.summarize.summaries.tour_profiles.stop_freq`
    Default build: **yes** | +| `atwork_subtour_frequency_distribution`
    `atwork_subtour_frequency_distribution.csv` | Mandatory work tours by their at-work-subtour-frequency alternative. Use it to validate subtour generation from the workplace. | `tours`: Prepared tour table.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `atwork_subtour_frequency` (`category`): Modeled at-work subtour-frequency alternative.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `atwork_subtour_frequency_category` (`String`): prepared at-work subtour-frequency category.
    `atwork_subtour_count` (`Float64`): weighted parent work tours choosing the category. | `processor.summarize.summaries.tour_profiles.at_work_sub_tour_freq`
    Default build: **yes** | +| `tour_time_of_day_by_tour_purpose`
    `tour_time_of_day_by_tour_purpose.csv` | Dense household-tour-unit departure, arrival, and duration profiles by tour purpose plus all-purpose totals. Every eligible row is retained; verified participant-row representations of one joint tour contribute fractionally so their combined weight equals one tour. Use it to compare scheduling and duration distributions. | `tours`: Prepared tour table.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `time_bin` (`Int32`): ActivitySim period index.
    `tour_purpose` (`String`): tour purpose or `all_tour_purposes`.
    `departure_tour_count` (`Float64`): weighted tours starting in the bin.
    `arrival_tour_count` (`Float64`): weighted tours ending in the bin.
    `duration_tour_count` (`Float64`): weighted tours whose prepared duration falls in the bin. | `processor.summarize.summaries.tour_profiles.tour_tod`
    Default build: **yes** | +| `tour_distance_by_tour_purpose`
    `tour_distance_by_tour_purpose.csv` | Nonmandatory, joint, and at-work household-tour-unit records by lower-bound `SKIMDIST` bin and purpose, plus all-purpose totals; mandatory tours are excluded to match the legacy distribution. The metric is fixed to prepared origin-to-primary-destination `SKIMDIST` and does not switch to `tour_distance` or generic `distance_miles`. Every row is retained, with verified participant-row joint-tour groups contributing fractional weight that sums to one tour. | `tours`: Prepared tour table.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `distance_bin` (`String`): exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`.
    `tour_purpose` (`String`): purpose or `all_tour_purposes`.
    `tour_count` (`Float64`): sum of eligible household-tour representation weight in the bin. | `processor.summarize.summaries.tour_profiles.tour_distance`
    Default build: **yes** | + +### Tour Geography + +_Related dashboard pages: Mandatory Location Choice, Tour Distance, Internal vs. External Tours. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `average_mandatory_tour_distance_by_purpose_and_geography`
    `average_mandatory_tour_distance_by_purpose_and_geography.csv` | Weighted average home-to-work or home-to-school distance for internal non-WFH workers, university students, and school students, by home geography and regionwide. Use it to compare mandatory destination accessibility. | `per`: Prepared person table.
    • `finalweight` (`Float64`): Person expansion weight from the configured person weight, inherited household weight, or 1.0 fallback.
    • `home_zone_id` (`identifier`): Prepared home-zone identifier. | `mandatory_tour_purpose` (`String`): `work`, `university`, or `school`.
    `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup.
    `average_tour_distance` (`Float64`): mean person-level mandatory distance weighted by `per.finalweight`.
    `person_count` (`Float64`): sum of `per.finalweight` for people contributing to the mean. | `processor.summarize.summaries.tour_geography.avg_mand_tour_distance`
    Default build: **yes** | +| `average_nonmandatory_tour_distance_by_purpose_and_geography`
    `average_nonmandatory_tour_distance_by_purpose_and_geography.csv` | Weighted average prepared origin-to-primary-destination `SKIMDIST` for individual nonmandatory tours by purpose and traveler home geography, plus regional rows. The metric does not switch to `tour_distance` or generic `distance_miles`. Use it to compare discretionary travel reach. | `per`: Prepared person table.
    • `person_id` (`Int64`): Canonical person identifier.
    • `home_zone_id` (`identifier`): Prepared home-zone identifier.

    `tours`: Prepared tour table.
    • `person_id` (`Int64`): Canonical person identifier associated with an individual tour.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `tour_purpose` (`String`): Primary purpose of the tour.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `nonmandatory_tour_purpose` (`String`): nonmandatory purpose.
    `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup.
    `average_tour_distance` (`Float64`): mean prepared `SKIMDIST` weighted by `tours.finalweight`.
    `tour_count` (`Float64`): sum of `tours.finalweight` for tours contributing to the mean. | `processor.summarize.summaries.tour_geography.avg_non_mand_tour_distance`
    Default build: **yes** | +| `internal_external_nonmandatory_tour_frequency_by_home_geography`
    `internal_external_nonmandatory_tour_frequency_by_home_geography.csv` | Internal and external nonmandatory tours by traveler home geography, plus regional totals. Use it to calculate external-tour shares and locate households producing external discretionary travel. | `per`: Prepared person table.
    • `person_id` (`Int64`): Canonical person identifier.
    • `home_zone_id` (`identifier`): Prepared home-zone identifier.

    `tours`: Prepared tour table.
    • `person_id` (`Int64`): Canonical person identifier associated with an individual tour.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `is_external_tour` (`Boolean-like`): Whether the tour destination is external to the modeled area.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup.
    `internal_nonmandatory_tour_count` (`Float64`): weighted internal nonmandatory tours.
    `external_nonmandatory_tour_count` (`Float64`): weighted external nonmandatory tours. | `processor.summarize.summaries.tour_geography.int_vs_ext_non_mand_tour_freq`
    Default build: **yes** | +| `external_nonmandatory_tour_locations`
    `external_nonmandatory_tour_locations.csv` | External nonmandatory tours by destination geography, plus a regional total. Use it to analyze external destination orientation and gateway demand. | `tours`: Prepared tour table.
    • `tour_category` (`String`): Tour category such as mandatory, nonmandatory, joint, or at-work.
    • `is_external_tour` (`Boolean-like`): Whether the tour destination is external to the modeled area.
    • `destination` (`identifier`): Tour destination-zone identifier.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `geography_type` (`String`): destination-geography system or rollup.
    `geography_id` (`String`): destination-geography identifier or rollup.
    `external_nonmandatory_tour_count` (`Float64`): weighted external nonmandatory tours ending there. | `processor.summarize.summaries.tour_geography.ext_non_mand_tour_loc`
    Default build: **yes** | + +### Trip Purpose, Mode, and Parking + +_Related dashboard pages: Trip and Stop Purpose, Trip Mode, Parking Location. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `trip_purpose_distribution`
    `trip_purpose_distribution.csv` | Person-trip destination purpose cross-classified by parent tour purpose, with all-tour-purpose rows. Joint records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to analyze activity chains within different kinds of tours. | `trips`: Prepared trip table.
    • `tour_purpose` (`String`): Primary purpose of the parent tour.
    • `trip_purpose` (`String`): Destination purpose of the trip leg.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `tour_purpose` (`String`): parent tour purpose or `all_tour_purposes`.
    `trip_purpose` (`String`): destination purpose of the trip leg.
    `trip_count` (`Float64`): weighted person-trips in the combination. | `processor.summarize.summaries.trip.trip_purpose`
    Default build: **yes** | +| `stop_destination_purpose_by_tour_purpose`
    `stop_destination_purpose_by_tour_purpose.csv` | Person-stop destination purposes by parent tour purpose. Joint records use the sum of all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to understand what activities are chained into tours. | `trips`: Prepared trip table.
    • `stops` (`Int32`): Intermediate-stop indicator, 1 before the last trip on a half-tour and 0 for the final leg.
    • `tour_purpose` (`String`): Primary purpose of the parent tour.
    • `trip_purpose` (`String`): Destination purpose of the trip leg.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `stop_destination_purpose` (`String`): destination purpose of trip records flagged as stops.
    `tour_purpose` (`String`): parent tour purpose.
    `stop_count` (`Float64`): weighted person-stops in the combination. | `processor.summarize.summaries.trip.stop_purpose_by_tour_purpose`
    Default build: **yes** | +| `trip_mode_by_tour_purpose_and_tour_mode`
    `trip_mode_by_tour_purpose_and_tour_mode.csv` | Person-trip mode counts by parent tour purpose and main tour mode, including all-purpose, all-tour-mode, and grand rollups. Joint records use the sum of all matching participant person weights when available; valid party-size expansion is the fallback, and sentinel, missing, or nonpositive party sizes fall back to one. Use it to examine access/egress and mode combinations within tours. | `trips`: Prepared trip table.
    • `tour_purpose` (`String`): Primary purpose of the parent tour.
    • `tour_mode` (`String`): Main mode of the parent tour.
    • `trip_mode` (`String`): Mode used for the trip leg.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `tour_purpose` (`String`): parent purpose or `all_tour_purposes`.
    `tour_mode` (`String`): main tour mode or `all_tour_modes`.
    `trip_mode` (`String`): mode of the individual trip leg.
    `trip_count` (`Float64`): weighted person-trips in the combination. | `processor.summarize.summaries.trip.trip_mode`
    Default build: **yes** | +| `parking_locations`
    `parking_locations.csv` | Auto-trip parking events by configured parking geography. Use it to map modeled parking demand and compare locations across runs. | `trips`: Prepared trip table.
    • `parking_zone` (`Int64`): Zone where the auto trip parks.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `geography_type` (`String`): parking-geography system.
    `geography_id` (`String`): valid positive parking-zone identifier at that geography.
    `trip_count` (`Float64`): weighted trips parking there. | `processor.summarize.summaries.trip.parking_locations`
    Default build: **yes** | + +### Trip Time and Distance + +_Related dashboard pages: Trip and Stop Time, Trip and Stop Distance. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `trip_departure_time_by_purpose`
    `trip_departure_time_by_purpose.csv` | Dense person-trip and person-stop departure-period profiles by parent tour purpose and for all purposes. Joint records use all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to compare trip timing with stop timing. | `trips`: Prepared trip table.
    • `tour_purpose` (`String`): Primary purpose of the parent tour.
    • `stops` (`Int32`): Intermediate-stop indicator, 1 before the last trip on a half-tour and 0 for the final leg.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `tour_purpose` (`String`): parent purpose or `all_tour_purposes`.
    `time_bin` (`Int32`): prepared trip departure period index.
    `departure_trip_count` (`Float64`): weighted person-trips departing in the bin.
    `departure_stop_count` (`Float64`): weighted departing person-stops. | `processor.summarize.summaries.trip_distributions.trip_stop_tod`
    Default build: **yes** | +| `trip_distance_by_purpose`
    `trip_distance_by_purpose.csv` | Person-trips by lower-bound OD-distance bin and parent tour purpose, plus all-purpose totals. Joint records use the sum of all matching participant person weights when available; valid party-size expansion is the fallback, non-joint trips contribute once, and sentinel, missing, or nonpositive fallback sizes resolve to one. Use it for purpose-specific trip length distributions. | `trips`: Prepared trip table.
    • `tour_purpose` (`String`): Primary purpose of the parent tour.
    • `od_dist` (`Float64`): Origin-to-destination trip distance from the configured distance skim or supported survey distance, in source units.
    • `num_participants` (`Int32`): Number of travelers represented by a joint trip record.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `distance_bin` (`String`): exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`.
    `tour_purpose` (`String`): parent purpose or `all_tour_purposes`.
    `trip_count` (`Float64`): sum of participant-adjusted trip weight in the bin. | `processor.summarize.summaries.trip_distributions.trip_distance`
    Default build: **yes** | +| `stop_out_of_direction_distance_by_tour_purpose`
    `stop_out_of_direction_distance_by_tour_purpose.csv` | Person-stops by lower-bound out-of-direction-distance bin, with a dense distribution for each purpose and all purposes. Joint records use all matching participant person weights when available and valid party-size expansion only as a fallback. Use it to quantify detour burden from stop-making. | `trips`: Prepared trip table.
    • `stops` (`Int32`): Intermediate-stop indicator, 1 before the last trip on a half-tour and 0 for the final leg.
    • `out_dir_dist` (`Float64`): Additional distance caused by visiting the stop rather than traveling directly to the half-tour endpoint.
    • `tour_purpose` (`String`): Primary purpose of the parent tour.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `distance_bin` (`String`): exact `0`, positive sub-mile `>0-<1`, whole-mile lower bound, or `40+`.
    `tour_purpose` (`String`): parent purpose or `all_tour_purposes`.
    `stop_count` (`Float64`): weighted person-stops in the bin. | `processor.summarize.summaries.trip_distributions.stop_ood_distance`
    Default build: **yes** | + +### Skimjoin Diagnostics + +_Related dashboard pages: Tour Skims, Trip Skims. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `skimjoin_trip_component_stats`
    `skimjoin_trip_component_stats.csv` | Weighted descriptive statistics for every numeric `skim_` component on trips, by trip mode and skim scenario. Use it to QA joined time, distance, and cost values and identify missing or zero-heavy components. | `trips`: Prepared trip table.
    • `trip_mode` (`String`): Mode used for the trip leg.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `skim_scenario` (`String`): chosen or hypothetical evaluation scenario.
    `trip_mode` (`String`): trip mode or `All Modes`.
    `component` (`String`): numeric skim column name.
    `n_total` (`Float64`): total trip weight eligible for the component/mode group.
    `n_valid` (`Float64`): trip weight with non-null component values, including zeros.
    `mean` (`Float64`): weighted mean including zeros.
    `mean_nonzero` (`Float64`): weighted mean excluding zeros.
    `std` (`Float64`): weighted population standard deviation.
    `min` (`Float64`): minimum observed value.
    `max` (`Float64`): maximum observed value.
    `median` (`Float64`): weighted 50th percentile.
    `mode` (`Float64`): value with greatest total weight, using the smaller value on ties.
    `zero_share` (`Float64`): valid weight at exactly zero divided by `n_valid`.
    `missing_share` (`Float64`): missing weight divided by `n_total`. | `processor.summarize.summaries.skimjoin.trip_skim_component_stats`
    Default build: **yes** | +| `skimjoin_trip_component_ecdf`
    `skimjoin_trip_component_ecdf.csv` | Optional 0th-through-100th weighted percentile curves for trip skim components by mode and scenario. Use it for distribution comparison when the compact stats table is insufficient. | `trips`: Prepared trip table.
    • `trip_mode` (`String`): Mode used for the trip leg.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `skim_scenario` (`String`): chosen or hypothetical scenario.
    `trip_mode` (`String`): trip mode or `All Modes`.
    `component` (`String`): numeric skim column name.
    `percentile` (`Float64`): cumulative probability from 0.00 through 1.00 in 0.01 steps.
    `value` (`Float64`): weighted quantile at that probability.
    `n_valid` (`Float64`): total valid trip weight behind the curve. | `processor.summarize.summaries.skimjoin.trip_skim_component_ecdf`
    Default build: **no** | +| `skimjoin_tour_component_stats`
    `skimjoin_tour_component_stats.csv` | Weighted descriptive statistics for numeric tour `skim_` components by tour mode and scenario. Use it to QA tour-level round-trip or composite skims. | `tours`: Prepared tour table.
    • `tour_mode` (`String`): Main mode selected for the tour.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `skim_scenario` (`String`): chosen or hypothetical evaluation scenario.
    `tour_mode` (`String`): tour mode or `All Modes`.
    `component` (`String`): numeric skim column name.
    `n_total` (`Float64`): total tour weight in scope.
    `n_valid` (`Float64`): tour weight with a non-null value, including zeros.
    `mean` (`Float64`): weighted mean including zeros.
    `mean_nonzero` (`Float64`): weighted mean excluding zeros.
    `std` (`Float64`): weighted population standard deviation.
    `min` (`Float64`): minimum value.
    `max` (`Float64`): maximum value.
    `median` (`Float64`): weighted median.
    `mode` (`Float64`): highest-weight value, smaller on ties.
    `zero_share` (`Float64`): valid weight at zero divided by `n_valid`.
    `missing_share` (`Float64`): missing weight divided by `n_total`. | `processor.summarize.summaries.skimjoin.tour_skim_component_stats`
    Default build: **yes** | +| `skimjoin_tour_component_ecdf`
    `skimjoin_tour_component_ecdf.csv` | Optional weighted percentile curves for tour skim components by mode and scenario. Use it to compare complete tour-level distributions across runs. | `tours`: Prepared tour table.
    • `tour_mode` (`String`): Main mode selected for the tour.
    • `finalweight` (`Float64`): Tour expansion weight inherited from trips, persons, households, or a 1.0 fallback according to available configured weights. | `skim_scenario` (`String`): chosen or hypothetical scenario.
    `tour_mode` (`String`): tour mode or `All Modes`.
    `component` (`String`): numeric skim column name.
    `percentile` (`Float64`): probability from 0.00 through 1.00.
    `value` (`Float64`): weighted quantile at that probability.
    `n_valid` (`Float64`): valid tour weight behind the curve. | `processor.summarize.summaries.skimjoin.tour_skim_component_ecdf`
    Default build: **no** | + +### Processor-Built Validation Summaries + +_Related dashboard pages: Overview, Traffic Validation, Transit Validation, VMT Validation. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `traffic_count_comparisons`
    `traffic_count_comparisons.csv` | Observed and modeled traffic counts that agree at count-location, direction, and period level. Use it for count scatterplots, percent differences, RMSE, and facility calibration. The table keeps only keys in both sources. | None declared. | `count_location_id` (`String`): traffic-count station/location identifier.
    `direction` (`String`): observed/modeled direction label.
    `count_period` (`String`): count time-period label.
    `observed_volume` (`Float64`): summed observed count for the key.
    `modeled_volume` (`Float64`): summed assigned volume for the matching key. | `processor.summarize.summaries.validation.traffic_count_comparisons`
    Default build: **yes** | +| `screenline_flow_comparisons`
    `screenline_flow_comparisons.csv` | Observed and modeled screenline flows matched by screenline, direction, and period, with a representative facility type. Use it for corridor-level flow validation and regression analysis. | None declared. | `screenline_id` (`String`): screenline/cutline identifier.
    `direction` (`String`): flow direction.
    `count_period` (`String`): comparison period.
    `facility_type` (`String`): supplied facility class, or `All` if absent.
    `observed_volume` (`Float64`): summed observed flow.
    `modeled_volume` (`Float64`): summed modeled flow for the matching key. | `processor.summarize.summaries.validation.screenline_flow_comparisons`
    Default build: **yes** | +| `transit_boardings_by_operator_and_technology`
    `transit_boardings_by_operator_and_technology.csv` | Assigned transit boardings summed by operator and transit technology. Use it to compare ridership scale across agencies and modes. | None declared. | `operator` (`String`): supplied transit operator identifier or name.
    `technology` (`String`): supplied transit mode/technology category.
    `boardings` (`Float64`): total assigned unlinked passenger boardings. | `processor.summarize.summaries.validation.total_transit_boardings`
    Default build: **yes** | +| `transit_transfer_rate`
    `transit_transfer_rate.csv` | Assigned boardings divided by linked transit trips by operator, technology, and access mode. The value is boardings per linked trip, so values above one indicate transfers; subtract one if a transfers-per-trip measure is needed. | None declared. | `operator` (`String`): transit operator.
    `technology` (`String`): transit technology/mode.
    `access_mode` (`String`): mode used to access transit.
    `transfer_rate` (`Float64`): assigned boardings divided by linked trips; null for a zero linked-trip denominator. | `processor.summarize.summaries.validation.transit_transfer_rate`
    Default build: **yes** | +| `auto_vmt_totals`
    `auto_vmt_totals.csv` | One run-level personal-auto VMT total using the same eligible records, distance source, and legacy-compatible occupancy calculation as the detailed personal-auto VMT summary. Every eligible row is retained. Household-level joint rows use one vehicle, while repeated person-level joint rows use their fixed auto-mode occupancy. A prepared auto skim is preferred; otherwise eligible non-drive-transit trips use `od_dist`. Use it for overview controls and scenario comparison. | `trips`: Prepared trip table.
    • `trip_mode` (`String`): Mode used for the trip leg.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `auto_vmt` (`Float64`): total weighted personal-auto vehicle miles traveled. | `processor.summarize.summaries.validation.auto_vmt_totals`
    Default build: **yes** | +| `auto_vmt_by_home_geography_income_hhsize_time_period`
    `auto_vmt_by_home_geography_income_hhsize_time_period.csv` | Personal-auto VMT and trip counts by home geography, income, household size, time period, and auto mode. When component periods exist, Daily rows are derived only from those non-Daily rows and replace any stale source Daily row; Daily-only groups are retained. Use it for equity, temporal, modal, and spatial VMT analysis. | `trips`: Prepared trip table.
    • `trip_mode` (`String`): Mode used for the trip leg.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback. | `geography_type` (`String`): home-geography system or `all_geographies`.
    `geography_id` (`String`): home-geography identifier or rollup.
    `income_segment` (`String`): prepared household income segment or fallback rollup.
    `household_size` (`String`): prepared household size or fallback rollup.
    `time_period` (`String`): configured period or `Daily`.
    `mode` (`String`): trip mode or `All Auto` fallback.
    `auto_vmt` (`Float64`): sum of distance times `trips.finalweight`, divided by legacy-compatible vehicle occupancy.
    `trip_count` (`Float64`): sum of `trips.finalweight` for eligible auto trips.
    `distance_source` (`String`): provenance of the distance used, such as a skim or OD-distance field.
    `time_period_source` (`String`): provenance of the time-period assignment. | `processor.summarize.summaries.validation.auto_vmt_by_home_geography_income_hhsize_time_period`
    Default build: **yes** | +| `non_motorized_vmt_by_home_geography_income_hhsize_time_period`
    `non_motorized_vmt_by_home_geography_income_hhsize_time_period.csv` | Walk, bicycle, and e-bike weighted miles and trip counts by home geography, income, household size, period, and mode, including derived daily rows. Use it for active-travel exposure and equity analysis. | `trips`: Prepared trip table.
    • `finalweight` (`Float64`): Trip expansion weight from the configured trip weight, inherited person or household weight, or 1.0 fallback.
    • `trip_mode` (`String`): Mode used for the trip leg. | `geography_type` (`String`): home-geography system or rollup.
    `geography_id` (`String`): home-geography identifier or rollup.
    `income_segment` (`String`): household income segment or fallback rollup.
    `household_size` (`String`): household size or fallback rollup.
    `time_period` (`String`): configured period or `Daily`.
    `mode` (`String`): `WALK`, `BIKE`, or `EBIKE` as available.
    `non_motorized_vmt` (`Float64`): distance times `trips.finalweight`; despite the VMT name, this is weighted traveler mileage.
    `trip_count` (`Float64`): sum of `trips.finalweight` for eligible trips.
    `distance_source` (`String`): prepared or skim distance source used for the mode.
    `time_period_source` (`String`): provenance of the period assignment. | `processor.summarize.summaries.validation.non_motorized_vmt_by_home_geography_income_hhsize_time_period`
    Default build: **yes** | +| `commercial_vmt_totals`
    `commercial_vmt_totals.csv` | Commercial-vehicle VMT by vehicle type, split between internal and external travel. Use it for freight VMT totals and internal/external shares. | None declared. | `commercial_vehicle_type` (`String`): supplied commercial vehicle/truck class.
    `external_vmt` (`Float64`): VMT from records classified as external.
    `internal_vmt` (`Float64`): VMT from records classified as internal. | `processor.summarize.summaries.validation.commercial_vehicle_vmt`
    Default build: **yes** | +| `bicycle_vmt_by_facility_type`
    `bicycle_vmt_by_facility_type.csv` | Bicycle VMT by facility type, read directly when supplied or calculated as assigned bicycle trips times link distance. Use it to evaluate bicycle use by facility class. | None declared. | `facility_type` (`String`): supplied bicycle/network facility category.
    `bicycle_vmt` (`Float64`): summed bicycle vehicle/traveler miles on that facility type. | `processor.summarize.summaries.validation.bicycle_vmt_by_facility`
    Default build: **yes** | + +### Externally Supplied Validation Contracts + +_Related dashboard pages: Traffic Validation, VMT Validation, Regional Validation. See the [Dashboard User Guide](16-dashboard-user-guide.md#page-by-page-guide)._ + +| Summary Table | Information and Analytical Use | Required Tables/Fields | Output Fields | Summary Function | +|---|---|---|---|---| +| `link_validation_summary`
    `link_validation_summary.csv` | Modeled network link volumes with link endpoints and facility class. Use it to aggregate modeled flow by facility or inspect high-volume links. | None declared. | `id` (`Int64`): link identifier.
    `From_Node` (`Int64`): upstream node identifier.
    `To_Node` (`Int64`): downstream node identifier.
    `FACTYPE` (`Int64`): facility-type code.
    `am_vol` (`Float64`): AM-period modeled link volume.
    `md_vol` (`Float64`): midday modeled link volume.
    `pm_vol` (`Float64`): PM-period modeled link volume.
    `day_vol` (`Float64`): daily modeled link volume. | `processor.summarize.summaries.validation_scaffolds.link_validation_summary`
    Default build: **no** | +| `count_location_counts_validation_summary`
    `count_location_counts_validation_summary.csv` | Observed traffic-count volumes by count location and facility class. Use it as the observed side of location-level modeled-versus-observed comparisons. | None declared. | `id` (`Int64`): count-location identifier.
    `FACTYPE` (`Int64`): facility-type code.
    `am_vol` (`Float64`): observed AM volume.
    `md_vol` (`Float64`): observed midday volume.
    `pm_vol` (`Float64`): observed PM volume.
    `day_vol` (`Float64`): observed daily volume. | `processor.summarize.summaries.validation_scaffolds.count_location_counts_validation_summary`
    Default build: **no** | +| `count_location_volumes_validation_summary`
    `count_location_volumes_validation_summary.csv` | Modeled volumes at the traffic-count locations. Use it as the modeled side of location-level count comparisons. | None declared. | `id` (`Int64`): count-location identifier matching the observed table.
    `FACTYPE` (`Int64`): facility-type code.
    `am_vol` (`Float64`): modeled AM volume.
    `md_vol` (`Float64`): modeled midday volume.
    `pm_vol` (`Float64`): modeled PM volume.
    `day_vol` (`Float64`): modeled daily volume. | `processor.summarize.summaries.validation_scaffolds.count_location_volumes_validation_summary`
    Default build: **no** | +| `count_location_scatter_validation_summary`
    `count_location_scatter_validation_summary.csv` | Long-form observed/modeled point pairs already prepared for count scatterplots. Use it when the source workflow supplies paired values directly. | None declared. | `id` (`Int64`): count-location identifier.
    `facility_type` (`String`): facility class code or label.
    `period` (`String`): comparison period.
    `observed_volume` (`Float64`): observed traffic volume.
    `modeled_volume` (`Float64`): modeled traffic volume. | `processor.summarize.summaries.validation_scaffolds.count_location_scatter_validation_summary`
    Default build: **no** | +| `count_location_fit_validation_summary`
    `count_location_fit_validation_summary.csv` | Precomputed linear-fit diagnostics for observed-versus-modeled counts by facility type and period. Use it to draw regression lines and report calibration fit. | None declared. | `facility_type` (`String`): facility class used for the fit.
    `period` (`String`): time period used for the fit.
    `slope` (`Float64`): fitted slope for modeled volume as a function of observed volume.
    `intercept` (`Float64`): fitted modeled-volume intercept.
    `r_squared` (`Float64`): coefficient of determination.
    `n_locations` (`Int64`): paired count locations in the fit.
    `observed_min` (`Float64`): minimum observed volume in the fitting data.
    `observed_max` (`Float64`): maximum observed volume.
    `equation_label` (`String`): preformatted regression-equation text.
    `r_squared_label` (`String`): preformatted R-squared text. | `processor.summarize.summaries.validation_scaffolds.count_location_fit_validation_summary`
    Default build: **no** | +| `district_commuting_flows_validation_summary`
    `district_commuting_flows_validation_summary.csv` | Supplied district-to-district commute-flow matrix for Albany, Corvallis, Lebanon, and Philomath. Use it as a local validation/control matrix. | None declared. | `""` (`String`): origin district or row label.
    `Albany` (`Float64`): commuters to Albany.
    `Corvallis` (`Float64`): commuters to Corvallis.
    `Lebanon` (`Float64`): commuters to Lebanon.
    `Philomath` (`Float64`): commuters to Philomath.
    `Total` (`Float64`): row total across destinations. | `processor.summarize.summaries.validation_scaffolds.district_commuting_flows_validation_summary`
    Default build: **no** | +| `county_commuting_flows_validation_summary`
    `county_commuting_flows_validation_summary.csv` | Supplied county-to-county commute-flow matrix for Benton, Linn, and Marion counties. Use it as a regional commute-flow validation/control matrix. | None declared. | `""` (`String`): origin county or row label.
    `Benton` (`Float64`): commuters to Benton County.
    `Linn` (`Float64`): commuters to Linn County.
    `Marion` (`Float64`): commuters to Marion County.
    `Total` (`Float64`): row total across destinations. | `processor.summarize.summaries.validation_scaffolds.county_commuting_flows_validation_summary`
    Default build: **no** | +| `commercial_vehicle_validation_summary`
    `commercial_vehicle_validation_summary.csv` | Supplied commercial-vehicle trip totals by time of day and vehicle class. Use it to compare commercial demand composition and daily profiles. | None declared. | `tod` (`String`): time-of-day row label.
    `car` (`Float64`): commercial-car/light-vehicle trips.
    `mu` (`Float64`): multi-unit truck trips.
    `su` (`Float64`): single-unit truck trips.
    `Total` (`Float64`): total commercial trips across classes. | `processor.summarize.summaries.validation_scaffolds.commercial_vehicle_validation_summary`
    Default build: **no** | +| `commercial_vehicle_vmt_validation_summary`
    `commercial_vehicle_vmt_validation_summary.csv` | Supplied commercial-vehicle VMT by time of day and vehicle class. Use it to compare freight mileage composition and temporal patterns. | None declared. | `tod` (`String`): time-of-day row label.
    `car` (`Float64`): commercial-car/light-vehicle VMT.
    `mu` (`Float64`): multi-unit truck VMT.
    `su` (`Float64`): single-unit truck VMT.
    `Total` (`Float64`): total commercial VMT across classes. | `processor.summarize.summaries.validation_scaffolds.commercial_vehicle_vmt_validation_summary`
    Default build: **no** | +| `external_trip_validation_summary`
    `external_trip_validation_summary.csv` | Supplied external trip totals by time of day and purpose/class. Use it to analyze gateway demand by travel market. | None declared. | `tod` (`String`): time-of-day row label.
    `hbcoll` (`Float64`): home-based college trips.
    `hbo` (`Float64`): home-based other trips.
    `hbr` (`Float64`): home-based recreation trips.
    `hbs` (`Float64`): home-based shopping trips.
    `hbsch` (`Float64`): home-based school trips.
    `hbw` (`Float64`): home-based work trips.
    `nhbnw` (`Float64`): non-home-based non-work trips.
    `nhbw` (`Float64`): non-home-based work trips.
    `truck` (`Float64`): truck trips.
    `Total` (`Float64`): total external trips across purposes/classes. | `processor.summarize.summaries.validation_scaffolds.external_trip_validation_summary`
    Default build: **no** | +| `external_vmt_validation_summary`
    `external_vmt_validation_summary.csv` | Supplied external VMT by time of day and purpose/class. Use it to identify which external markets contribute mileage. | None declared. | `tod` (`String`): time-of-day row label.
    `hbcoll` (`Float64`): home-based college VMT.
    `hbo` (`Float64`): home-based other VMT.
    `hbr` (`Float64`): home-based recreation VMT.
    `hbs` (`Float64`): home-based shopping VMT.
    `hbsch` (`Float64`): home-based school VMT.
    `hbw` (`Float64`): home-based work VMT.
    `nhbnw` (`Float64`): non-home-based non-work VMT.
    `nhbw` (`Float64`): non-home-based work VMT.
    `truck` (`Float64`): truck VMT.
    `Total` (`Float64`): total external VMT across purposes/classes. | `processor.summarize.summaries.validation_scaffolds.external_vmt_validation_summary`
    Default build: **no** | +| `auto_vmt_validation_summary`
    `auto_vmt_validation_summary.csv` | Supplied auto and truck VMT by time of day and occupancy class. Use it as an independent control for modeled VMT. | None declared. | `TOD` (`String`): time-of-day row label.
    `SOV` (`Float64`): single-occupant-vehicle VMT.
    `HOV2` (`Float64`): two-occupant shared-ride VMT.
    `HOV3` (`Float64`): three-or-more-occupant shared-ride VMT.
    `Truck` (`Float64`): truck VMT.
    `Total` (`Float64`): total VMT across listed classes. | `processor.summarize.summaries.validation_scaffolds.auto_vmt_validation_summary`
    Default build: **no** | +| `work_from_home_validation_summary`
    `work_from_home_validation_summary.csv` | Supplied worker and work-from-home controls by district. Use it to compare modeled WFH counts or rates with external targets. | None declared. | `District` (`String`): district name or identifier.
    `Workers` (`Float64`): total workers in the district.
    `WFH` (`Float64`): workers who work from home. | `processor.summarize.summaries.validation_scaffolds.work_from_home_validation_summary`
    Default build: **no** | + diff --git a/wiki/27-geography.md b/wiki/27-geography.md new file mode 100644 index 0000000..9868a5f --- /dev/null +++ b/wiki/27-geography.md @@ -0,0 +1,255 @@ +# 27 - Geography + +The geography feature adds consistent spatial groupings to prepared tables and +summary output. Use it to report the same measures by districts, counties, +subregions, or other zone-based systems without adding regional logic to each +summary builder. + +## Geography Layers + +The visualizer can use several kinds of geography at the same time: + +1. **Canonical zones** come from `zones`. Prepare creates MAZ and TAZ fields + such as `home_zone_id`, `home_taz`, `OTAZ`, and `DTAZ`. +2. **Native home geographies** such as `home_county` and `home_mpo` are retained + when they already exist in model output. +3. **A legacy land-use geography** can use `summarize.geography.landuse_col` to + create the compatibility fields `HGEO` and `WGEO`. +4. **Named aggregations** under `summarize.geography.aggregations` map MAZ or TAZ + IDs to any number of custom geography systems. + +Named aggregations are the preferred method for new custom geography work. +They keep geography IDs and source zone systems explicit and can support +several systems in one run. + +## Data Flow + +```text +inline mapping or CSV zone lookup + -> validate one geography label per zone + -> prepare role-specific geography columns + -> summary builders emit geography_type and geography_id + -> dashboard pages expose available geography options +``` + +Although the configuration lives under `summarize`, named geography mappings +also change prepared tables. Their normalized lookup rows are part of the +prepare and summary cache identities. + +## File-Based Example + +Given this CSV: + +```csv +MAZ,district +101,North +102,North +201,South +``` + +configure a named aggregation as follows: + +```yaml +zones: + use_maz: true + maz_col: [MAZ, zone_id] + taz_col: [TAZ, taz] + +summarize: + geography: + enabled: true + aggregations: + district: + source_zone_system: maz + file: lookups\maz_district.csv + zone_id_col: MAZ + geography_col: district +``` + +Relative file paths start from the main configuration directory. The file must +be CSV and contain both named columns. Zone IDs must be integers, geography +labels cannot be blank, and one zone cannot map to different labels. + +## Inline Example + +For a small, stable mapping, list zone IDs directly: + +```yaml +summarize: + geography: + enabled: true + aggregations: + market_area: + source_zone_system: taz + mapping: + Core: [1, 2, 3] + Suburban: [4, 5, 6] + External: [99] +``` + +The mapping direction is `geography label -> zone ID or list of zone IDs`. +Use either `mapping` or `file` for one aggregation, never both. + +## Settings + +| Field | Default | Behavior | +|---|---|---| +| `summarize.geography.enabled` | `false` | Enables the legacy geography and all named aggregations. When `false`, aggregation definitions are ignored. | +| `summarize.geography.landuse_col` | none | Names one existing land-use column used to create compatibility `HGEO` and `WGEO` fields. | +| `summarize.geography.mapping` | none | Maps raw values from `landuse_col` to normalized labels for the legacy geography. | +| `summarize.geography.aggregations` | `{}` | Defines one or more named MAZ- or TAZ-based lookups. | +| `dashboard.enable_maz_geographies` | `false` | Allows MAZ options on dashboard pages that support them. This is a presentation setting and does not create geography columns. | +| `display.labels.geography` | none | Changes display labels for geography type IDs, such as displaying `district` as `School District`. It does not remap zone membership. | + +Each named aggregation requires `source_zone_system: maz` or `taz` and exactly +one lookup form: + +| Lookup form | Required fields | +|---|---| +| Inline | `mapping` | +| CSV | `file`, `zone_id_col`, `geography_col` | + +Use a short, stable aggregation name such as `district` or `county`. That name +becomes the `geography_type` value in summaries and part of each prepared column +name. + +## Prepared Outputs + +For an aggregation named `district`, prepare can create: + +| Prepared table | Output columns | +|---|---| +| households | `home_geo__district` | +| persons | `home_geo__district`, `work_geo__district`, `school_geo__district` | +| tours | `origin_geo__district`, `destination_geo__district` | +| trips | `origin_geo__district`, `destination_geo__district` | +| land use | `land_use_geo__district` | + +The source columns depend on `source_zone_system`: + +| Role | MAZ source | TAZ source | +|---|---|---| +| household/person home | `home_zone_id` | `home_taz` | +| person work | `workplace_zone_id` | `work_taz` | +| person school | `school_zone_id` | `school_taz` | +| tour/trip origin | `origin` | `OTAZ` | +| tour/trip destination | `destination` | `DTAZ` | +| land use | `MAZ` | `TAZ` | + +Prepare first resolves the canonical MAZ and TAZ fields from `zones`. A named +aggregation can therefore fail to populate if the corresponding zone system is +missing or misconfigured. Zones absent from the lookup receive null geography +values; geography-specific summaries generally exclude those null rows. + +## Summary And Dashboard Outputs + +Summary tables that support geography use a long-form pair: + +- `geography_type` identifies the system, such as `maz`, `home_taz`, + `home_county`, or `district`. +- `geography_id` contains the zone or mapped label within that system. + +The exact supported roles vary by summary. For example, population summaries +use home geography, mandatory-location summaries can use work or school +geography, and destination summaries use destination geography. Check the +[Summary Catalog](26-summary-catalog.md) for each table's meaning and required +prepared columns. + +Native `home_county` and `home_mpo` columns can appear in supported summaries +without a named aggregation. `dashboard.enable_maz_geographies` controls only +whether supporting pages expose MAZ-level choices; it does not affect TAZ, +native, or named aggregation columns. + +To relabel geography type IDs in the dashboard: + +```yaml +display: + labels: + geography: + mapping: + district: School District + home_county: County + home_taz: TAZ +``` + +Keep membership changes under `summarize.geography`. Display labels do not +change joins, cache data, or geography IDs. + +## Compatibility Matrix + +Configuring a named aggregation creates every role column in the prepared +output, but a summary uses that aggregation only when its implementation +requests the corresponding role. The main supported paths are: + +| Geography role | Prepared source | Summary families that use it | Dashboard pages | +|---|---|---|---| +| Home | household/person home zone and `home_geo__` | worker internal/external status, work/school/university distance, work from home, telecommuting, average tour distance, internal/external non-mandatory tours, personal-auto and non-motorized VMT | Mandatory Location Choice, Tour Distance, Internal vs. External Tours, VMT Validation | +| Work | person workplace zone and `work_geo__` | workplace/employment comparison, workplace shadow-price residuals, commuting flows, external workplace locations | Mandatory Location Choice, Employment/Enrollment Match, Regional Validation | +| School | person school zone and `school_geo__` | school/enrollment comparison and school shadow-price residuals | Mandatory Location Choice, Employment/Enrollment Match | +| Tour/trip origin and destination | `origin_geo__`, `destination_geo__` | commuting flow matrices and external destination summaries | Mandatory Location Choice, Internal vs. External Tours, Regional Validation | +| Land use | `land_use_geo__` | employment/enrollment targets, shadow-price comparisons, and PNR capacity comparisons when the owning summary joins land use | Employment/Enrollment Match, Park-and-Ride Location | +| Parking location | `parking_zone` base zone only | `parking_locations` | Parking Location | + +Important limits: + +- Overview, generic household/person distributions, Tour Purpose/Mode/Time, + and most Trip Purpose/Mode/Time summaries do not gain a geography dimension + merely because an aggregation is configured. +- `parking_locations` currently reports its base MAZ or TAZ parking zone. The + prepare step does not create `parking_geo__`, so named aggregations do + not automatically appear on that page. +- Regional Validation uses only modeled geography types that agree with the + configured outside flow contract, such as `district`/`home_district` or + `county`/`home_county`. +- A page option is present only when at least one usable run contains non-null + rows for that geography type. Configuration alone does not force an empty + option into the selector. + +For an exact output, find its ID in chapter 26 and confirm that the schema has +`geography_type`/`geography_id` (or origin/destination geography pairs). Then +inspect the summary's prepared requirements and the relevant page declaration +in chapter 31. + +## Cache And Refresh Behavior + +The normalized mapping rows contribute to prepare and summary identity. A +change to a lookup file, inline mapping, aggregation name, or source zone system +invalidates incompatible prepared and summary caches automatically. For a +repeatable manual rebuild, include `prepare` in `pipeline.steps` and use +`refresh: [prepare]`; this also rebuilds later skimjoin and summary output. + +Setting `summarize.geography.enabled: false` disables both the legacy mapping +and named aggregations. Definitions left below the disabled setting do not +affect cache identity. + +## Implementation And Extension Points + +| Task | Start here | +|---|---| +| Config and lookup validation | `runtime/config/normalize_geography.py` | +| Cache identity | `runtime/config/signatures.py` | +| Zone context and lookup joins | `processor/prepare/enrichment/zones.py` | +| Household/person role columns | `processor/prepare/enrichment/households_persons.py` | +| Tour and trip role columns | `processor/prepare/enrichment/tours.py` and `trips.py` | +| Summary geography helpers | `processor/summarize/summaries/summary_helpers.py` | +| Dashboard geography options | `dashboard/helpers/geography_helpers.py` | + +## Troubleshooting + +| Symptom | Check | +|---|---| +| No custom geography columns | `summarize.geography.enabled`, prepared cache identity, and the aggregation name. | +| All mapped values are null | `source_zone_system`, `zones`, the prepared source columns, and lookup zone IDs. | +| CSV fails during config load | File path, required column names, integer zone IDs, blank labels, and conflicting duplicate zones. | +| Geography missing from a page | Whether that summary supports the role and whether any usable run has non-null data. | +| MAZ option missing | `dashboard.enable_maz_geographies` and the page's supported geography levels. | +| Labels are wrong but membership is correct | `display.labels.geography`, not the aggregation lookup. | + +## Related Chapters + +- [11 - Configuring Your Data](11-configuring-your-data.md) +- [13 - Configuration Reference](13-configuration-reference.md#summarize) +- [21 - Prepared Tables](21-prepared-tables.md) +- [26 - Summary Catalog](26-summary-catalog.md) +- [42 - Config, Columns, And Labels](42-config-column-label-cookbook.md) +- [90 - Troubleshooting](90-troubleshooting.md) diff --git a/wiki/30-output-visualizer.md b/wiki/30-output-visualizer.md index 964f751..4497d81 100644 --- a/wiki/30-output-visualizer.md +++ b/wiki/30-output-visualizer.md @@ -1,7 +1,7 @@ # 30 - Output Visualizer -The Output Visualizer reads processor outputs and presents them as either a -live Panel dashboard or a standalone HTML export. +The Output Visualizer turns processor output into either a live Panel dashboard +or a standalone HTML file. ```text summary caches + optional prepared tables @@ -23,13 +23,76 @@ The visualizer is responsible for: - rendering figures, tables, cards, and widgets - exporting supported page states to standalone HTML -It should not rebuild summaries. If a summary is missing, run the processor +The visualizer does not rebuild summaries. If one is missing, run the processor workflow first. +## Use The Live Dashboard + +After the server starts, open the URL printed in the terminal. The default is +`http://localhost:5006`. The left rail identifies the loaded runs and provides +the dashboard-wide controls; the main area contains standalone page tabs and +group tabs such as Tour Summaries and Validation Summaries. + +| Control | Effect | +|---|---| +| Runs Loaded | Shows the color and label used for each run. It is a legend, not a run filter. | +| Weighting | Selects one stored weighting mode. The control is disabled when only one mode is available. | +| Values: Percent | Shared distribution charts divide each run's values by the relevant plotted total. This supports shape comparison across runs of different sizes. | +| Values: Count | Shared distribution charts use stored weighted or unweighted values. | +| Page selectors | Filter or change only the registered sections that depend on them. Options come from usable data and can differ by configuration. | +| Calculation notes | Expand below supported output to show source summaries, filters, formulas, and aggregation details. | + +Some outputs deliberately ignore the Percent/Count switch. Examples include +rates, averages, validation statistics, tables, and charts whose builder sets a +fixed value mode. Read the axis label and calculation note; do not assume every +number on a Percent dashboard is a share. + +Configured segmented summaries appear as separate series such as +`Base (North)`. `segment.dashboard.visibility` determines whether the full run, +segments, or both are included, and `segmentation_type` selects the displayed +definition. These are configuration choices, not live sidebar controls. + +## Read Comparisons Correctly + +Use the following order when interpreting a chart: + +1. Confirm the weighting and Values controls. +2. Read the chart title, axis units, and active page selectors. +3. Identify each run or segment by its rail color and full hover label. +4. Check whether the output is a count, share, rate, average, residual, or + modeled-versus-observed comparison. +5. Expand the calculation note when available. + +Percent mode normally normalizes each run independently, so it compares +distributions rather than regional totals. Count mode can compare totals only +when runs use compatible sample expansion, model coverage, and source units. +Distance labels in existing pages assume miles; skim component units remain +the units in their source matrices or sidecars. + +The first configured run is the base for outputs that calculate a difference +or percent difference. Reordering `runs` can therefore change the comparison +reference as well as duplicate-label run-key suffixes. + +## Missing And Partial Data + +A page can use some runs while excluding others. A standard unavailable card +identifies missing files, unavailable summaries, failed calculations, or +schema mismatches. A partial result means at least one run was usable and at +least one was excluded; the chart still renders the usable runs. Hover labels +and the Runs Loaded legend do not prove that every run contributed to every +visualization. + +Set `display.missing_data_display: blank` only when you intentionally want to +hide diagnostic cards. During setup and extension work, keep the default +`card` behavior. + +For a page-by-page description, use +[16 - Dashboard User Guide](16-dashboard-user-guide.md). + ## Live Dashboard -The live dashboard is assembled in -[`dashboard/app.py`](../dashboard/app.py). It creates: +[`dashboard/app.py`](../dashboard/app.py) assembles the live dashboard. It +creates: - run colors and run legend - `DashboardState` @@ -45,7 +108,7 @@ pipeline: dashboard_mode: live ``` -Then run the normal config command: +Then use the standard configuration command: ```bash uv run activitysim-viz --config local_config.yaml @@ -53,9 +116,9 @@ uv run activitysim-viz --config local_config.yaml ## HTML Export -HTML export uses the same page registry, but serializes supported page content -into one self-contained HTML document. Export only includes states and selector -variants generated at export time. +HTML export uses the same page registry and converts supported content into one +self-contained document. It includes only the states and selector variants +available at export time. Configure `pipeline.dashboard_mode: export` and an output path: @@ -69,12 +132,12 @@ dashboard: output_path: exports/dashboard.html ``` -The same normal config command then writes the export. For details, read +The standard configuration command then writes the export. For details, see [34 - HTML Export](34-html-export.md). ## Dashboard State -`DashboardState` centralizes the global state pages react to: +`DashboardState` contains the global state that pages use: - loaded run labels - selected weighting mode @@ -82,13 +145,14 @@ The same normal config command then writes the export. For details, read - optional segmentation type and visibility - prepared-data provider state -Pages should read state through the `DashboardPage` helpers instead of -duplicating cache or run-selection logic. +Pages read state through the `DashboardPage` helpers, which avoids duplicating +cache and run-selection logic. ## Extension Path -The [Dashboard Extension Cookbook](45-dashboard-extension-cookbook.md) shows -complete page, page-group, selector, widget, table, and figure examples. +The [Dashboard Extension Cookbook](45-dashboard-extension-cookbook.md) gives +complete examples for pages, page groups, selectors, widgets, tables, and +figures. When adding visual output: @@ -103,7 +167,9 @@ When adding visual output: ## Related Chapters -- [31 - Dashboard Pages](31-dashboard-pages.md) +- [16 - Dashboard User Guide](16-dashboard-user-guide.md) +- [17 - Publish An Export With Posit Connect Cloud](17-posit-connect-cloud.md) +- [31 - Dashboard Page Contract](31-dashboard-pages.md) - [32 - Figures and Widgets](32-figures-and-widgets.md) - [33 - Dashboard Page Recipes](33-dashboard-page-recipes.md) - [34 - HTML Export](34-html-export.md) diff --git a/wiki/31-dashboard-pages.md b/wiki/31-dashboard-pages.md index 8796af5..981f6b3 100644 --- a/wiki/31-dashboard-pages.md +++ b/wiki/31-dashboard-pages.md @@ -1,10 +1,15 @@ -# 31 - Dashboard Pages +# 31 - Dashboard Page Contract -Dashboard pages are discovered from modules under -[`dashboard/pages`](../dashboard/pages). Each leaf module contains one -`DashboardPage` subclass decorated with `@dashboard_page(...)`; page packages +The visualizer discovers dashboard pages in modules under +[`dashboard/pages`](../dashboard/pages). Each final module contains one +`DashboardPage` subclass with a `@dashboard_page(...)` decorator. Page packages export a `DashboardGroupDefinition` as `GROUP`. +For descriptions of the analyses and advice about interpreting their controls, +see [16 - Dashboard User Guide](16-dashboard-user-guide.md). This chapter is +the authoritative reference for page IDs, data prerequisites, and extension +contracts. + ## Page Definition Contract Important fields: @@ -20,21 +25,21 @@ Important fields: | `optional_summary_ids` | Independent add-on summaries that may be absent. | | `required_prepared_tables` | Prepared tables required by the page. | -These declarations control dashboard cache loading, pruning, availability -diagnostics, and prepared-table loading. They do **not** select which generated -summaries the summarize workflow builds; ordinary summarize runs build every -`build_by_default=True` declaration. +These declarations control dashboard cache loads, removal of unused data, +availability diagnostics, and prepared-table loads. They do not select which +summaries the summarize workflow builds; standard summarize runs build every +declaration with `build_by_default=True`. -`required_summary_ids` marks the page's primary data. If no run has a usable +`required_summary_ids` identifies the primary page data. If no run has a usable required table, `self.data.summary(...)` records a required-data warning and the -page should render a standard unavailable card. `optional_summary_ids` declares -an independent add-on: its absence should hide or replace only that feature. -Neither declaration crashes the whole dashboard, and both can be partially -available when some runs are usable and others are excluded. +page shows a standard unavailable card. `optional_summary_ids` identifies data +for an independent feature; if that data is absent, hide or replace only the +feature. Missing declared data does not stop the entire dashboard, and data can +be available for only some runs. ## Enabling Pages -Live pages are selected in config: +Select live pages in the configuration: ```yaml dashboard: @@ -55,15 +60,15 @@ Group selection modes are: | `trip_summaries: all` | Every registered child, including children with `default_enabled=False`. | | `trip_summaries: [trip_mode, trip_stop_distance]` | Exactly the listed children in that order. | -When `dashboard.live.pages` is omitted, standalone pages and groups must be -default-enabled, and grouped children must also be default-enabled. A group's -`default_page_id` selects the initially visible tab/fallback; it does not by -itself enable every child. +If you omit `dashboard.live.pages`, the visualizer selects default-enabled +standalone pages and groups, along with each group's default-enabled children. +A group's `default_page_id` selects the first visible or fallback tab; it does +not enable every child. -`dashboard.export.pages` modifies matching pages in the resolved live page set; -it is not an allow-list. Unmentioned live pages keep their default export -behavior. Use `enabled: false`, `exclude_pages`, or `exclude_groups` to narrow -the export. Export cannot add a page omitted from `dashboard.live.pages`. +`dashboard.export.pages` changes matching pages in the resolved live page set. +It is not an allow-list. Live pages without an entry keep their default export +behavior. Use `enabled: false`, `exclude_pages`, or `exclude_groups` to remove +pages. Export cannot add a page that `dashboard.live.pages` omits. For example, enable only two trip-summary children: @@ -79,9 +84,9 @@ dashboard: ## Prepared-Data Pages -Most pages are summary-backed. A prepared-data page declares -`prepared_data_mode` and `required_prepared_tables`. Use prepared data only when -the page truly needs disaggregate records. +Most pages use summary data. A prepared-data page declares `prepared_data_mode` +and `required_prepared_tables`. Use prepared data only when the page requires +disaggregate records. Current runtime behavior is: @@ -89,18 +94,43 @@ Current runtime behavior is: |---|---| | `none` | Prepared caches are not requested for the page. `required_prepared_tables` must be empty. | | `optional` | Prepared caches are requested, but the page's primary summary-backed workflow should remain useful when they are unavailable. | -| `required` | Prepared caches are requested and the page should present an unavailable state when they cannot be loaded. | +| `required` | The runtime requests prepared caches. The page must show an unavailable state if it cannot load them. | -Both `optional` and `required` trigger loading; the distinction communicates -feature criticality and contributes to the strongest requirement across enabled -pages. Page render code remains responsible for the fallback. Standalone HTML -export does not load prepared tables; see chapter 34 for section-level export +Both `optional` and `required` cause a data load. The value identifies whether +the feature requires the data. It also contributes to the strongest requirement +for all enabled pages. Page render code must supply the fallback. Standalone +HTML export does not load prepared tables. See chapter 34 for section export rules. +## Availability And Validation Features + +Page selectors reflect the available data, and option providers list values +from usable runs. If an earlier choice makes a selection invalid, the page +lifecycle repairs it. A value should appear only when its dependent section has +data, not simply because it belongs to a fixed domain. + +When no usable run remains, the page shows the standard data-unavailable card +for the affected feature. Missing required data can make the primary page +workflow unavailable. Missing optional data replaces only its independent +feature. Set `display.missing_data_display: blank` to hide all these cards. + +The validation group provides: + +| Page | Current behavior | +|---|---| +| Traffic Validation | Observed-versus-modeled count-location fit, traffic volume summaries, top modeled count locations, link tables, and screenline flow comparison. Count-location diagnostics report location count, RMSE, RMSPE, and R-squared by facility group. Scatterplots include a 1:1 line. Fit-line hover shows the fitted equation, R-squared, and sample size. Filter screenlines by time period and facility type before the system calculates a fit for each run. RMSPE is blank for a group that contains a zero observed count. | +| Transit Validation | Boardings by operator and technology.
    Transfer rates by operator, technology, and access mode.
    The page shows notes and unavailable states if it cannot use the supplied contracts. | +| VMT Validation | Overview comparisons plus selector-driven personal-auto and non-motorized VMT. Optional outside tables add external travel/VMT, commercial travel/VMT, and bicycle facility summaries; each optional feature gets its own unavailable state. | +| Regional Validation | Optional district or county observed flow matrices, modeled `commuting_flows`, and aligned heatmaps. Heatmaps can show modeled, observed, difference, percent difference, or absolute percent difference. You can include or exclude totals. The selector shows only flow types that have available input. | + +Expandable calculation notes appear below the related output and identify +source summary IDs, filters, formulas, and aggregation details. They are shown +by default; set `dashboard.include_notes: false` to hide them. + ## Generated Page Catalog -The catalog below is generated from the dashboard page registry. Regenerate it -with: +The dashboard page registry generates the catalog below. Use this command to +regenerate it: ```bash uv run python scripts/generate_wiki_catalogs.py @@ -115,7 +145,7 @@ Total registered pages: **27** |---|---|---|---|---|---|---|---| | `overview` | Overview | - | yes | `none` | `population_totals`, `person_type_distribution`, `household_size_distribution`, `auto_vmt_totals` | - | - | | `daily_activity_pattern` | Daily Activity Pattern | Daily Travel | yes | `none` | `daily_activity_pattern_by_person_type`, `mandatory_tour_frequency_by_person_type`, `nonmandatory_tour_frequency_by_person_type`, `tour_rates_by_person_type_and_tour_purpose`, `trip_rates_by_person_type_and_trip_purpose` | - | - | -| `escorted_tours` | Escorted Tours | Daily Travel | yes | `none` | `escorted_tour_totals`, `school_escorted_tours_by_escort_type_and_direction`, `adult_escort_event_stop_distribution`, `adult_escorted_tours_by_person_type_and_direction`, `adult_escorted_tour_distance_distribution_by_direction`, `adult_escorted_trip_distance_distribution_by_direction`, `student_school_escort_status_by_direction`, `student_households_by_student_count`, `households_with_school_escorting_by_student_count_and_direction`, `schoolkids_per_escorted_tour_by_student_count_and_direction` | - | - | +| `escorted_tours` | Escorted Tours | Daily Travel | yes | `none` | `escorted_tour_totals`, `school_escorted_tours_by_escort_type_and_direction`, `adult_escort_event_stop_distribution`, `adult_escorted_tours_by_person_type_and_direction`, `adult_escorted_tour_distance_distribution_by_direction`, `adult_escorted_trip_distance_distribution_by_direction` | `student_school_escort_status_by_direction`, `student_households_by_student_count`, `households_with_school_escorting_by_student_count_and_direction`, `schoolkids_per_escorted_tour_by_student_count_and_direction` | - | | `joint_travel` | Joint Travel | - | yes | `none` | `jtf_distribution`, `joint_tours_by_household_size`, `joint_tour_party_size_distribution`, `joint_tour_composition_by_party_size`, `person_jtp_by_household_size`, `household_jtp_by_household_size_and_jtf` | - | - | | `individual_choices` | Individual Choices | Long-Term Choices | yes | `none` | `license_holding_status_distribution`, `bicycle_comfort_level_distribution`, `transit_pass_ownership_by_person_type`, `transit_subsidy_by_person_type` | - | - | | `vehicle_ownership_type` | Vehicle Ownership and Type | Long-Term Choices | yes | `none` | `auto_ownership_distribution`, `autonomous_vehicle_ownership_totals`, `vehicle_age_distribution`, `vehicle_fuel_type_distribution`, `vehicle_body_type_distribution` | - | - | @@ -138,7 +168,7 @@ Total registered pages: **27** | `traffic` | Traffic Validation | Validation Summaries | yes | `none` | `screenline_flow_comparisons` | `link_validation_summary`, `count_location_counts_validation_summary`, `count_location_volumes_validation_summary`, `count_location_scatter_validation_summary`, `count_location_fit_validation_summary` | - | | `transit` | Transit Validation | Validation Summaries | yes | `none` | `transit_boardings_by_operator_and_technology`, `transit_transfer_rate` | - | - | | `vmt` | VMT Validation | Validation Summaries | yes | `none` | `auto_vmt_by_home_geography_income_hhsize_time_period`, `non_motorized_vmt_by_home_geography_income_hhsize_time_period`, `bicycle_vmt_by_facility_type` | `commercial_vehicle_validation_summary`, `commercial_vehicle_vmt_validation_summary`, `external_trip_validation_summary`, `external_vmt_validation_summary` | - | -| `regional_validation` | Regional Validation | Validation Summaries | no | `none` | - | `county_flows_validation_summary`, `county_flows_joja_validation_summary`, `commuting_flows` | - | +| `regional_validation` | Regional Validation | Validation Summaries | no | `none` | - | `district_commuting_flows_validation_summary`, `county_commuting_flows_validation_summary`, `commuting_flows` | - | | `raw_trip_demo` | Prepared Trip Demo | - | no | `required` | - | - | `trips` | ## Registered Page Groups @@ -155,6 +185,7 @@ Total registered pages: **27** ## Related Chapters +- [16 - Dashboard User Guide](16-dashboard-user-guide.md) - [30 - Output Visualizer](30-output-visualizer.md) - [32 - Figures and Widgets](32-figures-and-widgets.md) - [33 - Dashboard Page Recipes](33-dashboard-page-recipes.md) diff --git a/wiki/32-figures-and-widgets.md b/wiki/32-figures-and-widgets.md index 06458f5..b9f972b 100644 --- a/wiki/32-figures-and-widgets.md +++ b/wiki/32-figures-and-widgets.md @@ -1,65 +1,61 @@ # 32 - Figures And Widgets -Current pages declare data access, selectors, independently refreshable -sections, and figures through one shared authoring model. Framework code owns -widget synchronization, query identity, missing-data diagnostics, and export -metadata. +Pages use one authoring model for data access, selectors, refreshable sections, +and figures. The framework controls widget synchronization, query identity, +missing-data diagnostics, and export metadata. ## Page Lifecycle -Every page subclasses `DashboardPage` and implements `build_page()`. That method +Each page subclasses `DashboardPage` and implements `build_page()`, which declares selectors and sections once and returns a stable Panel layout. -`DashboardPage.__init__()` calls `build_page()` after it creates `self.data`, -page state, and the component registries. Ordinary pages should therefore not -define their own `__init__`. If specialized initialization is unavoidable, it -must call `super().__init__(state, config)`, and attributes used by -`build_page()` must exist before that call. In practice, put declarations in -`build_page()` and keep implementation mixins free of `__init__` methods. +`DashboardPage.__init__()` creates `self.data`, page state, and the component +registries before calling `build_page()`. A standard page therefore does not +need its own `__init__`. If special initialization is necessary, create any +attributes needed by `build_page()` before calling +`super().__init__(state, config)`. Keep declarations in `build_page()` and never +put an `__init__` method in an implementation mixin. The main author-facing objects are: - `self.data` for summary and prepared `RunTables` -- `self.select(...)` for ordinary dropdowns, including dynamic options +- `self.select(...)` for standard selection lists, including dynamic options - `self.selector(...)` only for custom widgets - `self.section(...)` for refreshable visible regions - `self.feature(...)` for a namespaced group of selectors and sections - `self.query(...)` for repeated or expensive transformations - `self.plot` for figures and tables -Do not add routine `sync_controls()` or page-authored cache keys. Option -providers and section dependencies give the framework enough information to do -that work. ## Data And Figures -For end-to-end examples of an ordinary chart, a Plotly customization, a new -shared figure type, a custom widget, and a table, use the +For complete examples of a standard chart, a Plotly customization, a new shared +figure type, a custom widget, and a table, use the [Dashboard Extension Cookbook](45-dashboard-extension-cookbook.md). For the complete chart-method, count/share, table, and figure-testing API, see the [Plotting Reference](35-plotting-reference.md). -Load the narrowest useful data selection through `self.data.summary(...)` or -`self.data.summaries(...)`. `RunTables` applies the same Polars operation across -runs while retaining labels and availability issues; it supports operations +Load only the required data through `self.data.summary(...)` or +`self.data.summaries(...)`. `RunTables` applies the same Polars operation to +each run. It keeps labels and availability issues. It supports operations such as `where`, `with_columns`, `group`, `select`, `sort`, `join`, `map`, `requiring`, and `drop_empty`. -A `RunTables` value is truthy when at least one run has a non-empty compatible -table. Runs with a missing table, schema mismatch, failure, or empty input are -excluded from iteration and described in `data.issues`; consequently, -`data.partial` means there are both usable and excluded runs. Fluent operations -preserve those issues. Filtering can make a frame empty without removing it, so -call `.drop_empty()` when downstream code should ignore those runs. +A `RunTables` value is true when at least one run has a nonempty compatible +table. Iteration excludes runs with a missing table, schema mismatch, failure, +or empty input. `data.issues` describes these runs. `data.partial` means that +there are usable and excluded runs. Query operations keep these issues. A +filter can make a frame empty without removal. Call `.drop_empty()` when later +code must ignore these runs. The `columns=` argument to `summary()` and `prepared()` is a compatibility -check: a run missing any named column is excluded with a schema diagnostic. It -does **not** project the returned frames. Use `.select(...)` when a transform -needs a narrower schema. +check. The system excludes a run that is missing a named column. It also adds a +schema diagnostic. The argument does not select columns in the returned frames. +Use `.select(...)` to select a smaller schema. -Pass `RunTables` to `self.plot` methods where possible. Shared rendering lives -under `dashboard/rendering/`, including figures, tables, layout, and plotter -logic. Cross-page domain helpers live under `dashboard/helpers/`. +Pass `RunTables` to `self.plot` methods when possible. Figures, tables, layout, +and plotter logic are in `dashboard/rendering/`. Shared page helpers are in +`dashboard/helpers/`. ```python def render_mode_chart(self): @@ -77,9 +73,125 @@ def render_mode_chart(self): return self.plot.bar(chart_data, x="trip_mode", y="trip_count") ``` +The page-facing data API is: + +| API | Result | +|---|---| +| `self.data.summary(id, weighting=None, columns=(), required=None)` | One summary across usable runs. `columns` performs a schema compatibility check. | +| `self.data.summaries(*ids, columns=None, required=None)` | A dictionary of summary ID to `RunTables`. | +| `self.data.prepared(table, columns=(), weighting_mode=None)` | One declared prepared table across loaded runs. | +| `self.data.prepared_runs(weighting_mode=None)` | Direct `RunData` access for features that require matrices or other non-table state. | +| `self.data.summary_series(id, weighting=None)` | Specialized skim-summary view that retains summary-series metadata. | + +For `summary()` and `summaries()`, `required` has these exact meanings: + +| Value | Behavior when no run is usable | +|---|---| +| `None` | Required when the ID appears in the page definition's `required_summary_ids`; optional otherwise. | +| `True` | Record the selection and emit the page's required-summary warning even when the decorator did not declare it. | +| `False` | Record diagnostics but suppress the required-summary warning. Use this for an independent optional feature. | + +`required` does not make the lookup raise and does not render a card by itself. +The section must still test the returned `RunTables` and choose its standard +unavailable or optional-feature fallback. `columns=` is evaluated per run, so +one compatible run can render while other runs appear in `data.issues`. + +`RunTables` is iterable and indexable as `(run_label, DataFrame)` pairs. Its +public query interface is: + +| API | Behavior | +|---|---| +| `.where(column=value, ...)` | Equality filter; list, tuple, set, or frozenset values use membership. | +| `.with_columns(*exprs)` / `.select(*exprs)` / `.sort(*by)` | Apply the corresponding Polars operation to every run. | +| `.group(by, *aggs, **named_aggs)` | Group and aggregate every run. | +| `.join(other, on=..., how="left", coalesce=None)` | Join matching run labels and merge availability issues/source IDs. | +| `.map(transform)` | Apply a DataFrame-to-DataFrame transform to every run. | +| `.requiring(*columns)` | Keep frames containing all named columns. Prefer the lookup `columns=` check when exclusions should produce schema diagnostics. | +| `.drop_empty()` | Remove frames made empty by a previous operation. | +| `.values(column)` | Distinct non-null values in first-seen run order. | +| `.scalar(column, default=None)` | First value for each usable run. | +| `.to_list()` | Materialize tuples for an external API that cannot consume `RunTables`. | +| `.available`, `.partial`, `.issues`, `.source_ids` | Availability and provenance metadata retained through fluent operations. | + +Each issue contains `label`, `status`, `detail`, `source_kind`, `source_id`, +`missing_columns`, and available run/cache identity. Plotting a partial +`RunTables` value renders only its usable runs and keeps the exclusions in page +and export diagnostics. Do not replace it with `.to_list()` before the normal +render boundary unless an external API requires tuples; that discards the +structured availability object from subsequent fluent operations. + +## Query Cache Contract + +`self.query(factory)` accepts one zero-argument callable and returns the +callable's result. On a cache miss it executes `factory`; on a hit it returns +the stored value. Its identity contains: + +- page ID; +- current global dashboard state, including weighting, Values, and segment + presentation state; +- active section ID; +- current values of selectors declared by that section; +- callable module, qualified name, file, and first line; and +- simple closure/default/keyword-default values. Complex captured objects are + represented by type, so capture the scalar values that actually change the + calculation. + +A selector affects query identity only when its ID appears in the active +section's `selectors=(...)` declaration. Always declare every selector that +changes the renderer. Global state or a declared selector change creates a new +identity automatically. Call `self.clear_query_cache()` only after mutable +external state changes outside those declared inputs; it clears this page's +memoized queries. + +```python +def render_body(self): + purpose = self._purpose_by_label[self.purpose.value] + data = self.data.summary( + "trip_mode_by_tour_purpose_and_tour_mode", + columns=("tour_purpose", "trip_mode", "trip_count"), + ) + return self.query( + lambda: data.where(tour_purpose=purpose) + .group("trip_mode", pl.col("trip_count").sum()) + .drop_empty() + ) +``` + +## Calculation Notes + +Calculation notes are expandable HTML details below annotated charts and +tables. They have no external dependencies. To hide all notes, use: + +```yaml +dashboard: + include_notes: false +``` + +`dashboard/calculation_notes.yaml` contains the content. The top-level `methods` +mapping contains reusable method explanations. `notes` contains stable note +IDs. Each note requires `summary`, `method`, and a nonempty `sources` list. A +note can also contain `label`, `method_text`, `formula`, `source_filters`, and +grouped `details`. The loader validates unknown fields and method references. + +Page authors attach a note to a registered selector-driven section with: + +```python +body = self.section( + "comparison", + selectors=("facility_type",), + render=self.render_comparison, +) +return self.noted_section("traffic.observed_model_fit", body) +``` + +Use `self.noted_view(note_id, view)` for an individual plot or table outside the +registered section container. `self.section_note(...)` is the low-level helper. +It rejects unregistered sections. Notes use the same page layout in live mode +and HTML export. + ## Selectors -Declare a normal dropdown with its option domain in one place: +Declare a standard selection list and its option domain in one place: ```python self.purpose = self.select( @@ -90,10 +202,10 @@ self.purpose = self.select( ) ``` -An option provider is called before dependent sections render. The framework -repairs stale values. `default` may be `"first"`, `"last"`, or a callable. -Use `self.selector(...)` only when wrapping a custom checkbox, numeric input, or -another widget that `select(...)` cannot express. +Before rendering dependent sections, the framework calls an option provider and +repairs stale values. `default` can be `"first"`, `"last"`, or a callable. +Use `self.selector(...)` only for a custom checkbox, numeric input, or other +widget that `select(...)` cannot define. ## Sections And Features @@ -107,20 +219,20 @@ chart = self.section( ) ``` -A section renderer may return one Panel `Viewable`, or a list/tuple of -`Viewable` objects. It should not mutate the stable section container itself; -the lifecycle replaces that container's contents after each render. +A section renderer can return one Panel `Viewable`, or a list or tuple of +`Viewable` objects. The lifecycle replaces the container content after each +render, so the renderer must not replace the stable section container itself. -For a large page, use `self.feature("comparison")` to namespace a coherent +For a large page, use `self.feature("comparison")` to give a name to one workflow. Feature component IDs become `comparison.metric`, `comparison.body`, -and so on. Features participate in the same lifecycle and export behavior as -the parent page. +and similar names. Features use the same lifecycle and export behavior as the +parent page. -Large controllers may also use private implementation mixins under a -`_/` package. Mixins organize source responsibilities; `PageFeature` -organizes live components. A refactored page commonly uses both. Keep mixins -focused, do not give them `__init__` methods, keep pure transforms as functions, -and preserve page/component IDs during source-only refactors. +Large controllers can also use private implementation mixins in a `_/` +package. Mixins organize source responsibilities, while `PageFeature` organizes +live components; a page can use both. Give each mixin one purpose and no +`__init__` method. Keep pure transforms as functions, and preserve page and +component IDs during a source-only refactor. ### Large-Page Implementation Mixins @@ -157,9 +269,9 @@ class ExamplePage( pass ``` -Every mixin method receives the final `ExamplePage` instance. Python resolves -methods left to right through the declared bases and then `DashboardPage`. -Mixins are not standalone pages and must not be instantiated. +Each mixin method receives the final `ExamplePage` instance. Python resolves +methods from left to right through the declared bases, with `DashboardPage` +last. Mixins are not standalone pages and should not be instantiated. Keep this pattern narrow: @@ -170,14 +282,13 @@ Keep this pattern narrow: - keep stateless pure functions outside mixins - preserve page, selector, section, and export IDs during source-only refactors -Mixins organize Python source; `PageFeature` organizes registered live -components. One does not replace the other. Prefer one page class until stable -composition, domain, transformation, and rendering boundaries make the split -easier to understand. +Mixins organize Python source, while `PageFeature` organizes registered live +components. Use one page class until the composition, domain, transformation, +and rendering boundaries are stable. ## Shared Helpers -Check these before adding page-local utilities: +Before you add page-local utilities, examine these modules: | Module | Use | |---|---| @@ -190,14 +301,14 @@ Check these before adding page-local utilities: ## Export Considerations -Export behavior derives from the same selectors and sections used live. Keep -render methods deterministic for each selector state and avoid unregistered -live-only callbacks. Export can only include selector values generated at -export time. +The same selectors and sections control live and export behavior. Make sure +that render methods give the same result for each selector state. Do not use +unregistered live-only callbacks. Export can include only selector values that +exist at export time. ## Related Chapters -- [31 - Dashboard Pages](31-dashboard-pages.md) +- [31 - Dashboard Page Contract](31-dashboard-pages.md) - [33 - Dashboard Page Recipes](33-dashboard-page-recipes.md) - [34 - HTML Export](34-html-export.md) - [35 - Plotting Reference](35-plotting-reference.md) diff --git a/wiki/33-dashboard-page-recipes.md b/wiki/33-dashboard-page-recipes.md index ed8af47..c8157cd 100644 --- a/wiki/33-dashboard-page-recipes.md +++ b/wiki/33-dashboard-page-recipes.md @@ -1,7 +1,8 @@ # 33 - Dashboard Page Recipes -Use the smallest page shape that fits the behavior. Each discoverable page -module contains one class decorated with `@dashboard_page(...)`. +Use the smallest page structure that provides the required behavior. Each +discoverable page module contains one class with a `@dashboard_page(...)` +decorator. ## Recipe 1: Simple Summary Page @@ -30,8 +31,8 @@ class MySummaryPage(DashboardPage): return data_table(data, title="My Summary") ``` -Use `required_summary_ids` for the page's primary workflow and -`optional_summary_ids` for independent add-on features. +Use `required_summary_ids` for the primary page workflow. Use +`optional_summary_ids` for independent optional features. ## Recipe 2: Dynamic Selector @@ -79,13 +80,13 @@ def render_chart(self): ) ``` -The framework refreshes options and dependent sections. Use -`self.selector(...)` only for a genuinely custom widget. Keep the label-to-raw -mapping so display labels do not leak into data filters. +The framework refreshes both options and dependent sections. Use +`self.selector(...)` only for a custom widget, and keep the label-to-raw mapping +so display labels do not enter data filters. ## Recipe 3: Multi-Workflow Page -Create one `PageFeature` per coherent user workflow: +Create one `PageFeature` for each user workflow: ```python comparison = self.feature("comparison") @@ -95,10 +96,10 @@ comparison_body = comparison.section( ) ``` -When the Python controller itself becomes difficult to navigate, keep the -registered page as a small facade and split implementation mixins into a -private `_/` package. Current examples include tour mode, mandatory -location choice, escorted tours, VMT, and traffic validation. +If the Python controller becomes difficult to read, keep the registered page as +a small facade. Put implementation mixins in a private `_/` package. +Examples include tour mode, mandatory location choice, escorted tours, VMT, +and traffic validation. ## Recipe 4: Prepared-Data Page @@ -137,21 +138,20 @@ class RawTripDemoPage(DashboardPage): ) ``` -Load prepared data through `self.data`, handle an unavailable selection with a -standard card, and keep disaggregate use limited. Prefer summaries for repeated -aggregate views. `raw_trip_demo.py`, the skim pages, and parking location show -the current required/optional patterns. +Load prepared data through `self.data`, and show a standard card when it is +unavailable. Reserve disaggregate data for cases that need it; use summaries +for repeated aggregate views. See `raw_trip_demo.py`, the skim pages, and +parking location for current required and optional patterns. -Mark every section that reads prepared data with +Mark each section that reads prepared data with `export_data_mode="optional"` or `"required"`. Standalone export does not load -prepared tables and skips those sections. If the page also has a summary-backed -view that should export, place it in a separate section whose -`export_data_mode` remains `"none"`. +prepared tables. It omits these sections. Put an exportable summary view in a +separate section. Keep `export_data_mode="none"` for that section. ## Adding A New Page Group -For a complete file layout, config example, discovery explanation, and tests, -see [Add A New Page Group](45-dashboard-extension-cookbook.md#add-a-new-page-group). +For a complete file layout, configuration example, discovery description, and +tests, see [Add A New Page Group](45-dashboard-extension-cookbook.md#add-a-new-page-group). Create a package under `dashboard/pages/` and define `GROUP` in `__init__.py`: @@ -166,9 +166,8 @@ GROUP = DashboardGroupDefinition( ) ``` -Every child decorator sets `group_id="my_group"`. Discovery rejects duplicate -IDs, missing definitions, unknown groups, and invalid summary or prepared-table -requirements. +Set `group_id="my_group"` in each child decorator. Discovery rejects duplicate +IDs, missing definitions, unknown groups, and invalid data requirements. ## Page Review Checklist @@ -184,7 +183,7 @@ requirements. ## Related Chapters -- [31 - Dashboard Pages](31-dashboard-pages.md) +- [31 - Dashboard Page Contract](31-dashboard-pages.md) - [32 - Figures and Widgets](32-figures-and-widgets.md) - [34 - HTML Export](34-html-export.md) - [45 - Dashboard Extension Cookbook](45-dashboard-extension-cookbook.md) diff --git a/wiki/34-html-export.md b/wiki/34-html-export.md index 1adae1f..70a2868 100644 --- a/wiki/34-html-export.md +++ b/wiki/34-html-export.md @@ -1,30 +1,29 @@ # 34 - HTML Export -HTML export writes a standalone dashboard file that can be opened without a -Python server. +HTML export writes a standalone dashboard that opens without a Python server. ```text registered dashboard pages -> export payload -> serialized Panel nodes -> embedded CSS, Plotly, and runtime JS - -> one HTML file + -> one HTML file + diagnostics JSON sidecar ``` ## When To Use Export -Use export when you need: +Use export for these requirements: - an offline deliverable -- a dashboard that can be emailed or archived -- a frozen set of run comparisons +- a dashboard that you can send or archive +- a fixed set of run comparisons - no Python server dependency for viewers -Use live mode when you need: +Use live mode for these requirements: - full Python-backed interactivity - exploratory pages that are not export-ready -- development/debugging feedback +- development or debug feedback ## Export Configuration @@ -42,26 +41,43 @@ dashboard: output_path: exports/dashboard.html ``` -Run the same command used for every configured workflow: +Use the standard command for a configured workflow: ```bash uv run activitysim-viz --config local_config.yaml ``` -This writes `artifacts/exports/dashboard.html`. Relative export paths resolve -below `root`; an absolute path writes elsewhere. Change -`pipeline.dashboard_mode` back to `live` when the same config should serve the -dashboard instead. +This command writes `artifacts/exports/dashboard.html`. Relative export paths +start from `root`; use an absolute path for a different location. To start the +live dashboard from the same configuration, set `pipeline.dashboard_mode` to +`live`. -Export begins with the pages resolved by `dashboard.live.pages`. The -`dashboard.export.pages` mapping modifies matching page selectors and parts; it -does not select the included page set. Use a page override with `enabled: false`, -`exclude_pages`, or `exclude_groups` to narrow the live set. Export cannot add a -page that live configuration omitted. +The command also writes `artifacts/exports/dashboard.diagnostics.json`, a +sidecar file that records export warnings and size or state analysis. The HTML +file does not depend on this sidecar. + +After you verify the exported HTML, you can publish it with the free public +workflow in +[17 - Publish An Export With Posit Connect Cloud](17-posit-connect-cloud.md). + +The sidecar distinguishes rendered, partial, and skipped visualization inputs +for every exported dashboard state and region variant. It also reports raw, +valid, aliased, and pruned selector combinations plus estimated JSON bytes by +state, page, and region. See +[36 - HTML Export Schema](36-html-export-schema.md#diagnostics-sidecar-schema) +for every field and current warning threshold. + +For one override, use `--export-html [PATH]`. If you do not give a path, the +command uses the configured output path. If that path is absent, it uses +`/exported_dashboard.html`. You must also select the dashboard step. Add +`--dashboard` if `pipeline.steps` does not contain it. + +For the live page set and export override/exclusion behavior, see +[Enabling Pages](31-dashboard-pages.md#enabling-pages). ## Supported Runtime Behavior -The export runtime supports a deliberately small set of rendered objects: +The export runtime supports these rendered objects: - containers - cards @@ -72,52 +88,52 @@ The export runtime supports a deliberately small set of rendered objects: - registered regions - registered selector widgets +Use the header button to close or open the export sidebar. Plotly charts change +size after the layout changes. Long run names use short, unique tab and legend +labels. Tab tooltips and chart hover text show the full names. + The Python-to-JavaScript contract lives in `dashboard/export/types.py`, and the browser runtime lives under `dashboard/export/js_runtime/`. ## Selector Variants -Page-local export interactivity is pre-rendered. During export, the runtime -walks configured selector values, renders page regions, serializes them, and -stores them as variants. +Before writing the file, the exporter processes configured selector values, +renders page regions, and stores those regions as export-data variants. -That means: +These rules apply: - exported selectors can only switch among values generated at export time - large selector domains can make export files large - pages must register selectors and sections through the page API - live-only callbacks do not automatically work in export -Selector and part names are author-defined IDs, not widget labels or section -titles. Find selector IDs in a page's `self.select(...)` and -`self.selector(...)` calls, and part IDs in `self.section(...)` calls. Feature -IDs prefix their components (for example, `comparison.metric` and -`comparison.body`). The page/group IDs are listed in the generated catalog in -chapter 31, and chapter 13 contains a complete override example. Invalid page, -selector, part, or selector-value entries fail or produce a targeted warning -rather than being silently guessed. +Selector and part names are IDs from the author. They are not widget labels or +section titles. Find selector IDs in `self.select(...)` and +`self.selector(...)` calls. Find part IDs in `self.section(...)` calls. Feature +IDs are prefixes for their components. Examples are `comparison.metric` and +`comparison.body`. The generated catalog in chapter 31 lists page and group +IDs. Chapter 13 contains a complete override example. An invalid page, +selector, part, or selector value causes an error or a specific warning. For a concrete selector/section declaration that works in both modes, see the [Dashboard Extension Cookbook](45-dashboard-extension-cookbook.md#add-a-dynamic-selector). ## Page Authoring Contract -Export metadata comes from the same page registration graph used by the live -dashboard: +Live mode and export use the same page registration graph for metadata: - `@dashboard_page(...)` owns page identity, grouping, order, and data requirements. - `build_page()` creates stable widgets, sections, features, and layout. -- `self.select(...)` registers ordinary dropdowns and their option/default +- `self.select(...)` registers standard selection lists and their option/default policy. - `self.selector(...)` registers custom widgets. - `self.section(...)` defines refresh and export-region boundaries. -Keep section renderers deterministic for a given selector state. Set -`export=False` on a section that should remain static in the exported shell, -and `exportable=False` on a selector that should remain live-only. Do not add a -second export-only registry or duplicate selector metadata on the page -definition. +Make sure a section renderer gives the same result for a specified selector +state. Set `export=False` on a section that must stay static in the +exported shell. Set `exportable=False` on a live-only selector. Do not add an +export-only registry. Do not copy selector metadata to the page definition. Grouped export configuration addresses children by their leaf `page_id`: @@ -131,13 +147,13 @@ dashboard: tour_purpose: all ``` -Validation rejects unknown page, group, selector, and part IDs against this -shared runtime graph. +Validation compares page, group, selector, and part IDs with this shared runtime +graph. It rejects unknown IDs. ## Prepared Data Is A Live-Only Boundary -The export workflow loads summary caches but does not load prepared runs. A -section that reads prepared data must declare that boundary: +The export workflow loads summary caches. It does not load prepared runs. A +section that reads prepared data must declare this limit: ```python trip_table = self.section( @@ -147,12 +163,11 @@ trip_table = self.section( ) ``` -During HTML export, any section whose `export_data_mode` is `optional` or -`required` is skipped. The distinction still documents whether the feature is -optional or essential in live mode. Summary-only sections use the default -`export_data_mode="none"` and remain eligible for export. Split mixed pages -into separate prepared-backed and summary-backed sections so the latter can be -exported safely. +HTML export omits sections whose `export_data_mode` is `optional` or `required`; +these values indicate whether a prepared-data feature is optional or required +in live mode. Summary-only sections use the default +`export_data_mode="none"` and can be exported. On a mixed page, separate the +prepared-data and summary-data sections so the latter can remain in the export. ## Important Files @@ -164,11 +179,28 @@ exported safely. | `dashboard/export/selector_states.py` | Resolves selector domains, configured values, and scoped widget state. | | `dashboard/export/traversal.py` | Projects registered page components onto the export traversal contract. | | `dashboard/export/serializer.py` | Converts Panel objects to export nodes. | -| `dashboard/export/types.py` | Defines payload and node dataclasses. | +| `dashboard/export/types.py` | Defines payload and node `TypedDict` contracts. | | `dashboard/export/runtime_assets.py` | Loads CSS and JavaScript runtime assets. | | `dashboard/export/js_runtime/` | Readable browser runtime source. | | `dashboard/export/assets/export_runtime.js` | Built browser runtime embedded in exports. | +## Export Write And Python APIs + +`dashboard.export` exposes two entry points: + +| API | Behavior | +|---|---| +| `build_export_html_document(runs, config, summary_runs=None) -> str` | Build, serialize, and validate a complete HTML document in memory. Useful for tests and callers that need the string. | +| `write_export_html_document(output_path, runs, config, summary_runs=None) -> Path` | Build the payload and stream JSON into a temporary HTML file.
    Write the diagnostics sidecar through a temporary file.
    Replace each destination only after the temporary file is complete.
    This is the standard workflow method. | + +Payload construction cleans NumPy and Pandas values before JSON encoding: +nonfinite numbers become JSON `null`, timestamps become ISO strings, and closing +script tags are escaped. The writer streams the JSON without creating a second +payload or final HTML string, which reduces peak memory use for exports with +many selector states. If conversion, shell creation, writing, or finalization +fails, an `ExportBuildError` identifies the phase and the writer removes its +temporary files. + ## Changing Export Runtime Behavior Checklist: @@ -188,12 +220,15 @@ Checklist: 1. Open the exported HTML in a browser. 2. Open developer tools and check the console. -3. Look for `ExportRuntimeError` messages. -4. Try `?debug_export=1` in the URL. -5. Compare live mode to export mode with the same config and summary caches. +3. Inspect the adjacent `.diagnostics.json` file for build warnings and + size/state analysis. +4. Look for `ExportRuntimeError` messages. +5. Try `?debug_export=1` in the URL. +6. Compare live mode to export mode with the same config and summary caches. ## Related Chapters +- [17 - Publish An Export With Posit Connect Cloud](17-posit-connect-cloud.md) - [30 - Output Visualizer](30-output-visualizer.md) - [32 - Figures and Widgets](32-figures-and-widgets.md) - [90 - Troubleshooting](90-troubleshooting.md) diff --git a/wiki/35-plotting-reference.md b/wiki/35-plotting-reference.md index 6cc4874..33f4e9c 100644 --- a/wiki/35-plotting-reference.md +++ b/wiki/35-plotting-reference.md @@ -1,27 +1,27 @@ # 35 - Plotting Reference -Dashboard pages use one plotting surface: `self.plot`. It accepts the same -`RunTables` object returned by `self.data`, applies the session's run colors and -count/share mode, and returns a Panel view ready for a section. +Dashboard pages use one plotting interface, `self.plot`. It accepts the +`RunTables` object returned by `self.data`, applies the session run colors and +count or share mode, and returns a Panel view for a section. -## The normal path +## Standard method Fetch, query, and plot without converting the data to tuple lists: ```python data = ( self.data.summary( - "trip_mode_by_purpose", - columns=("purpose", "mode", "trip_count"), + "trip_mode_by_tour_purpose_and_tour_mode", + columns=("tour_purpose", "trip_mode", "trip_count"), ) - .where(purpose=self.purpose_sel.value) - .group("mode", pl.col("trip_count").sum()) - .sort("mode") + .where(tour_purpose=self.purpose_sel.value) + .group("trip_mode", pl.col("trip_count").sum()) + .sort("trip_mode") ) return self.plot.bar( data, - x="mode", + x="trip_mode", y="trip_count", title="Trip Mode", x_title="Mode", @@ -29,9 +29,9 @@ return self.plot.bar( ) ``` -Every chart argument after `data` is keyword-only. The short names (`x`, `y`, -`x_title`, and `y_title`) are the complete public vocabulary; the former -`x_col`, `y_col`, and `xaxis_title` aliases are not supported. +Each chart argument after `data` is keyword-only. The public names are `x`, `y`, +`x_title`, and `y_title`. The interface does not support the former `x_col`, +`y_col`, and `xaxis_title` aliases. ## Chart types @@ -42,8 +42,8 @@ Use: - `self.plot.line(...)` for an unfilled profile; and - `self.plot.scatter(...)` for observed-versus-modeled comparisons. -All four validate their required columns before calling Plotly. An error names -the chart type, run, and missing columns. +Before calling Plotly, all four methods validate their required columns. Any +error identifies the chart type, run, and missing columns. ```python return self.plot.density( @@ -58,8 +58,34 @@ return self.plot.density( ``` Sort ordered data in the query. For categorical bars, pass -`category_order=[...]` when the configured display order matters or missing -categories must keep a stable axis position. +`category_order=[...]` to use the configured display order. Also use it when +missing categories must keep a stable axis position. + +### Keyword Reference + +All figure builders accept `x`, `y`, `title`, `x_title`, `y_title`, and +`height`. Additional chart-specific keywords are: + +| Chart | Keywords | +|---|---| +| `bar` | `barmode="group"`, `share_y=None`, `value_mode="dashboard"`, `category_order=None`, `show_legend=None` | +| `line` | `value_mode="dashboard"` | +| `density` | `value_mode="dashboard"`, `x_range=None`, `category_order=None`, `tick_values=None`, `tick_text=None`, `hover_x_title=None` | +| `scatter` | `drop_zero_y=False`, `fit_overlays=None`, `fit_annotation="annotation"`, `one_to_one=False`, `legend_on_right=False` | + +`self.plot.scatter(...)` also accepts `panel_aspect_ratio`. This argument sets +the size of the returned Panel pane. It does not go to the Plotly figure builder. + +For fitted scatterplots, `fit_overlays` is another `RunTables` object or an +iterable of run and frame pairs. Each fit frame must contain the scatter `x` +and `y` columns. It can contain the column specified by `fit_annotation`. +Hovering on the fitted line shows this text. `one_to_one=True` adds a dashed 1:1 +line. It gives both axes the same range and locks their scale. Validation pages +use this API for run equations, R-squared values, and sample sizes. + +The interface can shorten long run labels for legends and tabs without changing +their identity. Plotly hover text and exported tab tooltips keep the full label, +and scatter point and fit hover text also include the run name. ## Count and share behavior @@ -78,8 +104,8 @@ return self.plot.bar( ) ``` -If a summary already contains a specifically defined share, provide that -column with `share_y`. The renderer selects it only in share mode: +If a summary contains a defined share, supply that column with `share_y`. The +renderer selects it only in share mode: ```python return self.plot.bar( @@ -90,15 +116,15 @@ return self.plot.bar( ) ``` -Use `share_y` when the denominator has domain meaning that cannot be recovered -by summing `y`. Do not select between count and percent columns in the page just -to follow the global toggle. There are no `as_percent`, `normalize`, -`percent_y_col`, or `pct_col` plotting arguments. +Use `share_y` when its denominator has a special meaning that a sum of `y` +cannot reproduce. Do not select count or percentage columns in the page merely +to follow the global control. The plotting interface does not have +`as_percent`, `normalize`, `percent_y_col`, or `pct_col` arguments. -## Figure-first escape hatch +## Direct figure API -The core builders return `plotly.graph_objects.Figure`, which is useful for -testing or for adding a genuinely page-specific annotation: +The core builders return `plotly.graph_objects.Figure`. Use this result for +tests or a page-specific annotation: ```python figure = self.plot.figure.scatter( @@ -111,10 +137,10 @@ figure.add_vline(x=1000, line_dash="dot") return self.plot.panel(figure) ``` -Prefer the normal `self.plot.*` methods when no figure customization is needed. -They use the same immutable `RenderContext` as export, so live and exported -charts receive identical colors, labels, hover policy, and value mode without -module-global setup. +Use the standard `self.plot.*` methods when a figure does not require a custom +change. They use the same fixed `RenderContext` as export, so live and export +charts get identical colors, labels, hover policy, and value mode. They do not +require module-global setup. ## Tables and layout @@ -124,12 +150,28 @@ Display helpers are grouped by responsibility under `dashboard.rendering`: from dashboard.rendering import data_table, selector_row ``` -`data_table(data, title)` accepts `RunTables` directly. Page KPI values should -use `self.plot.kpi(...)`, which shares the same run context as charts. Selector -rows, missing-data cards, legends, and other layout helpers live in -`dashboard.rendering.layout`; numeric and column formatting lives in +`data_table(data, title)` accepts `RunTables` directly. Use `self.plot.kpi(...)` +for page KPI values. It uses the same run context as charts. Selector rows, +missing-data cards, legends, and layout helpers are in +`dashboard.rendering.layout`. Numeric and column formatting are in `dashboard.rendering.tables`. +The `dashboard.rendering` facade exports these non-plot helpers: + +| API | Purpose | +|---|---| +| `data_table()`, `to_pandas()` | Render run-aware tables or convert supported Polars/Pandas input at the presentation boundary. | +| `format_numeric()`, `format_numeric_frame()` | Apply display-only numeric precision. | +| `drop_index_columns()`, `column_titles()` | Remove serialized index artifacts and create human-readable column titles. | +| `standardize_keys()` | Normalize a table iterable to common key/value column names. | +| `selector_row()`, `control_row()`, `control_row_spacer()` | Build consistent page control layouts. | +| `data_unavailable_card()` | Render the standard missing-data diagnostic card. | +| `run_legend_entries()`, `run_legend_panes()` | Build run/color legend metadata or panes. | + +`column_title_metadata()` is available from +`dashboard.rendering.tables` for serializer-aware title metadata, but is not +part of the package-level facade. + ## Testing charts Test the figure instead of constructing a full Panel layout: @@ -145,16 +187,17 @@ assert figure.data[0].name == "Base" assert list(figure.data[0].x) == ["Walk", "Bike"] ``` -This keeps plot tests fast and isolates data/query behavior from Panel. +This approach keeps plot tests fast by separating data and query behavior from +Panel. Use the focused plotting target during development: ```bash -pytest tests/test_figure_builders.py +uv run pytest --basetemp .pytest_tmp tests/test_figure_builders.py ``` -Page query behavior belongs in `tests/test_page_authoring.py`; the complete -HTML export suite is a separate release-boundary check. +Test page query behavior in `tests/test_page_authoring.py`. Run the complete +HTML export suite as a separate release check. ## Related Chapters diff --git a/wiki/36-html-export-schema.md b/wiki/36-html-export-schema.md index acb1f9d..9ab2b8d 100644 --- a/wiki/36-html-export-schema.md +++ b/wiki/36-html-export-schema.md @@ -1,10 +1,12 @@ # 36 - HTML Export Schema -This document defines the Python-to-JavaScript contract used by the standalone offline dashboard export. +This document defines the Python-to-JavaScript contract for the standalone +dashboard export. The implementation lives under `dashboard/export/`: -- `dashboard/export/html.py`: entry points that build and write the final self-contained HTML document +- `dashboard/export/html.py`: entry points that build and write the final + self-contained HTML document - `dashboard/export/payload.py`: dashboard-state and top-level payload composition - `dashboard/export/traversal.py`: page-tree and export-region resolution - `dashboard/export/selector_states.py`: selector request and canonical-state enumeration @@ -14,17 +16,18 @@ The implementation lives under `dashboard/export/`: - `dashboard/export/types.py`: typed payload and node definitions - `dashboard/export/js_runtime/`: readable browser-runtime source split into small files - `dashboard/export/assets/export_runtime.js`: client runtime that validates and renders the payload -- `dashboard/export/build_export_runtime.py`: concatenates `js_runtime/` into the shipped runtime asset +- `dashboard/export/build_export_runtime.py`: concatenates `js_runtime/` into + the shipped runtime asset ## Top-Level Payload -The exported HTML embeds one JSON payload inside: +The exported HTML contains one JSON payload in: ```html ``` -The payload shape is defined in `dashboard/export/types.py` as `ExportPayload`. +`dashboard/export/types.py` defines the payload as `ExportPayload`. Top-level fields: @@ -32,6 +35,7 @@ Top-level fields: |---|---|---| | `schema_version` | `str` | Versioned schema identifier checked by the browser runtime before rendering | | `title` | `str` | Dashboard title shown in the export header | +| `logo` | `str \| None` | Configured dashboard logo embedded as a data URI | | `runs_loaded` | `list[dict[str, str]]` | Run labels and colors used for the export legend | | `chrome` | `ExportChrome` | Shell layout metadata and dashboard-control enablement flags | | `dashboard_controls` | `DashboardControlsPayload` | Exported dashboard-wide weighting and values options | @@ -41,7 +45,29 @@ Top-level fields: | `page_export_support` | `PageExportSupportPayload` | Metadata about export-enabled page selectors | | `client_runtime` | `str` | Runtime family identifier for diagnostic/debugging purposes | -`states` is keyed by the dashboard state key built in `dashboard/export/payload.py`: +Current protocol identifiers are `schema_version: "2.0"`, +`client_runtime: "region-swap-v1"`, and +`page_export_support.client_side_runtime: "dashboard-and-page-selectors"`. +Treat them as compatibility identifiers, not user-visible labels. + +### Dashboard Chrome And State Fields + +| Object | Complete fields | +|---|---| +| `runs_loaded[*]` | `label`, `color` | +| `chrome` | `layout`, `rail_sections`, `controls_enabled` | +| `chrome.controls_enabled` | Boolean `weighting`, Boolean `values` | +| `dashboard_controls` | `weighting` list and `values` list | +| `default_state` | `weighting`, `values` | +| `page_export_support` | `client_side_runtime`, `enabled_page_selectors` | +| `enabled_page_selectors[*]` | `page_id`, `selector_id` | + +The runtime uses the options in `dashboard_controls` to validate +`default_state` and to form state keys. A control can remain in the payload +while `controls_enabled` disables switching because only one value was +exported. + +`dashboard/export/payload.py` builds the dashboard state key for `states`: ```text || @@ -65,7 +91,9 @@ Each `PageDescriptorPayload` contains: | `children` | `list[PageDescriptorPayload]` | Child page descriptors when this entry is a grouped top-level page | | `default_page_id` | `str \| None` | Default leaf page used when a grouped export page first loads | -The top-level page order is resolved through the shared page registry. Grouped pages keep their child pages nested under a single top-level export tab, while serialized page content in `states` remains keyed by leaf page id. +The shared page registry sets the top-level page order. Grouped pages keep their +child pages under one top-level export tab. Content in `states` uses the final +page ID as its key. ## Selector Metadata @@ -84,7 +112,7 @@ Each `SelectorMetadataPayload` contains: | `export_enabled` | `bool` | Whether the selector is interactive in export or rendered as a disabled/static control | | `parent_selector_id` | `str` (optional) | Parent selector for a dependent option domain | | `options_by_parent_value` | `dict[str, list[str]]` (optional) | Child options keyed by parent value | -| `disabled_parent_values` | `list[str]` (optional) | Parent values for which the dependent selector is disabled | +| `disabled_parent_values` | `list[str]` (optional) | Parent values that disable the dependent selector | Selector config is driven from: @@ -96,7 +124,7 @@ dashboard: : ... ``` -Grouped child pages may also be configured as: +You can also configure grouped child pages as follows: ```yaml dashboard: @@ -108,26 +136,28 @@ dashboard: : ... ``` -Validation comes from the shared page registry: +The shared page registry supplies these validation rules: - unknown page ids fail in `validate_page_export_config()` - unknown selector ids fail in `validate_page_export_config()` -- unavailable configured selectors log a warning once and fall back to non-interactive region/static page behavior +- unavailable configured selectors write one warning to the log and use a static region or page ## Page Content Shape -`PageContentPayload` is always: +`PageContentPayload` always has these fields: | Field | Type | Purpose | |---|---|---| | `kind` | `"page"` | Discriminator | -| `content` | `ExportNode` | Serialized page shell rooted at a normal export node tree | +| `content` | `ExportNode` | Serialized page shell that starts with a standard export node tree | -Pages without export-enabled selectors serialize as a normal page shell whose tree contains no `region` nodes. Pages with export-enabled selectors serialize one stable page shell with one or more embedded `region` nodes. +Pages without export-enabled selectors create a standard page shell whose tree +has no `region` nodes. Pages with export-enabled selectors create one stable +page shell with one or more `region` nodes. ## Region Nodes -`region` is a first-class `ExportNode` kind used for subtree-level switching. +`region` is an `ExportNode` kind that changes one part of a node tree. Fields: @@ -142,7 +172,7 @@ Fields: | `variants` | `dict[str, ExportNode]` | Mapping from selector-combination key to serialized subtree | | `variant_aliases` | `dict[str, str]` | Alternate selector keys mapped to a canonical rendered variant | -Variant keys are JSON strings generated by `dashboard.export.serializer.variant_key()`. +`dashboard.export.serializer.variant_key()` generates JSON variant-key strings. Example: @@ -150,13 +180,14 @@ Example: ["All","DRIVE"] ``` -The order of values in the key must match `selector_ids`. +The value order in the key must agree with `selector_ids`. -If a configured selector is unavailable for a region at export time, that region serializes with empty `selector_ids`, `default_content`, and no interactive variants. +If a configured selector is unavailable at export time, the region has empty +`selector_ids` and `default_content`. It has no interactive variants. ## Supported Node Kinds -The browser runtime only supports the node kinds declared in `dashboard/export/types.py`. +The browser runtime supports only the node kinds in `dashboard/export/types.py`. | Kind | Produced from | Important fields | |---|---|---| @@ -170,15 +201,92 @@ The browser runtime only supports the node kinds declared in `dashboard/export/t | `html` | `pn.pane.Markdown`, `pn.pane.HTML`, plain strings, unsupported fallback markup | `html` | | `spacer` | `pn.Spacer` | no extra fields | -Unsupported objects currently serialize to an `html` node containing a visible fallback panel. The runtime itself treats unknown node kinds as an error and shows an error panel. +An unsupported object becomes an `html` node with a visible fallback panel. An +unknown node kind is an error, which the runtime shows in an error panel. -Supported widget types are `select`, `radio_button_group`, `float_input`, +The supported widget types are `select`, `radio_button_group`, `float_input`, `checkbox`, and `button`. `SelectorMetadataPayload.default_value` and widget -values are JSON-compatible values and are not limited to strings. +values can be all JSON-compatible values. They are not limited to strings. + +### Complete Node Fields + +Every node has a `kind` discriminator and only the fields for that kind: + +| Kind | Required fields | Optional fields | +|---|---|---| +| `container` | `layout`, `children`, `child_count`, `styles`, `css_classes` | none | +| `card` | `title`, `children` | none | +| `tabs` | `tabs`; each tab has `title`, `content` | tab `full_title` | +| `plotly` | `figure` | `height`, `aspect_ratio` | +| `table` | `columns`, `rows` | `column_tooltips` | +| `widget` | `widget_type`, `name`, `value`, `options`, `step`, `disabled`, `selector_id`, `export_enabled` | `parent_selector_id`, `options_by_parent_value`, `disabled_parent_values` | +| `html` | `html` | none | +| `spacer` | none | none | +| `region` | `region_id`, `selector_ids`, `content_mode`, `default_key`, `default_content`, `variants`, `variant_aliases` | none | + +`container.layout` is `row` or `column`. `widget_type` is one of +`radio_button_group`, `select`, `float_input`, `checkbox`, or `button`. +`region.content_mode` is currently `snapshot`. + +Plotly's `figure` field is its JSON-compatible figure dictionary. Table rows +are dictionaries keyed by the ordered `columns`. HTML is already serialized +markup; the browser runtime does not execute Python pane logic. + +## Diagnostics Sidecar Schema + +`write_export_html_document()` writes `.diagnostics.json` beside the +HTML. It is build-time diagnostic data and is not required to open the HTML. +The top-level shape is: + +| Field | Type | Meaning | +|---|---|---| +| `schema_version` | integer, currently `1` | Diagnostics format version; separate from export payload `2.0`. | +| `title` | string | Dashboard title. | +| `states` | mapping | Page and region diagnostics keyed by `||`. | +| `size_analysis` | mapping | Estimated compact-JSON bytes by state, page, and region. | + +For each dashboard state, `states[state_key][page_id]` is a mapping containing: + +- `default`: visualization diagnostics for the default page state; +- `export_region:`: selector enumeration counts; and +- `region::`: visualization diagnostics captured for + one rendered region variant. + +A visualization diagnostic has these fields: + +| Field | Meaning | +|---|---| +| `visualization_id` | Summary or prepared input used as the diagnostic boundary. | +| `render_state` | `rendered`, `partial`, or `skipped`. | +| `input_kind` | `summary`, `prepared`, or `mixed`. | +| `input_ids` | Source summary/prepared IDs. | +| `usable_run_labels` | Runs included in that output. | +| `excluded_runs` | Per-run exclusions. | + +Each excluded run contains `label`, `status`, `detail`, `source_kind`, +`source_id`, and `missing_columns`. An export-region enumeration record +contains `selector_ids`, `selector_counts`, `raw_state_count`, +`valid_state_count`, `alias_count`, and `pruned_state_count`. + +`size_analysis` contains: + +| Field | Contents | +|---|---| +| `warning_thresholds` | Byte thresholds for total, strong-total, page, static-region, and selector-region warnings. | +| `total_payload_bytes`, `state_count` | Whole payload estimate and number of dashboard states. | +| `states` | Per-state `payload_bytes`; each page has `payload_bytes` and region metrics. | +| `page_peaks` | Largest state for each page. | +| `region_peaks` | Largest state for each page/region. | + +Region size metrics contain `selector_ids`, `variant_count`, +`default_content_bytes`, `variants_bytes`, and `total_bytes`. Current warning +thresholds are 100 MiB total, 250 MiB strong total, 10 MiB per page, 5 MiB for +a static region, and 1 MiB for a selector region. These are warnings, not hard +limits. ## Runtime Validation Rules -The embedded runtime validates: +The embedded runtime validates these items: - payload presence and JSON parseability - `schema_version` compatibility @@ -187,7 +295,7 @@ The embedded runtime validates: - presence of `states` - presence of `dashboard_controls` -At render time it also fails visibly on: +At render time, it shows an error for these conditions: - unknown rail sections - unknown widget types @@ -196,7 +304,8 @@ At render time it also fails visibly on: - missing region state for the active selector combination - Plotly runtime failures -Failures are shown in the HTML via a visible error panel and also logged to the browser console. +The HTML shows failures in a visible error panel and also writes them to the +browser console. ## Schema Versioning Policy @@ -204,20 +313,26 @@ Failures are shown in the HTML via a visible error panel and also logged to the Rules: -1. Change `schema_version` whenever the browser runtime can no longer safely consume payloads emitted by older Python code. -2. Keep the runtime check strict. A mismatch should fail loudly instead of rendering incorrect content. -3. Update this document, `dashboard/export/assets/export_runtime.js`, and the export payload tests in the same change. +1. Change `schema_version` when the browser runtime cannot safely use payloads + from older Python code. +2. Keep the runtime check strict. A mismatch must show an error and must not + render incorrect content. +3. Update this document, the readable files under + `dashboard/export/js_runtime/`, rebuild the generated asset, and update the + export payload/runtime tests in the same change. ## Checklist for Adding a New Node Kind -When adding a new serialized node kind: +To add a serialized node kind: 1. Add the new typed shape to `dashboard/export/types.py`. 2. Emit it from `dashboard/export/serializer.py`. -3. Render it in `dashboard/export/assets/export_runtime.js`. -4. Add serializer coverage in `tests/test_export_serializer.py`. -5. Add or update payload/smoke assertions if the new node can appear in representative exports. -6. Update this document. +3. Render it in `dashboard/export/js_runtime/`. +4. Run `uv run python dashboard/export/build_export_runtime.py` to rebuild + `dashboard/export/assets/export_runtime.js`; do not edit the built asset. +5. Add serializer coverage in `tests/test_export_serializer.py`. +6. Add or update payload/smoke assertions if the new node can appear in representative exports. +7. Update this document. ## Related Chapters diff --git a/wiki/40-developer-workflows.md b/wiki/40-developer-workflows.md index a6438a5..577b2ce 100644 --- a/wiki/40-developer-workflows.md +++ b/wiki/40-developer-workflows.md @@ -1,6 +1,6 @@ # 40 - Developer Workflows -This chapter is for contributors changing code or documentation. +Use this chapter to find the right workflow when changing code or documentation. ## Codebase Map @@ -40,6 +40,8 @@ activitysim_visualizer/ | New raw-output normalization | [21 - Prepared Tables](21-prepared-tables.md) | | New prepared column | [21 - Prepared Tables](21-prepared-tables.md#adding-a-prepared-column) | | New skim-derived output | [22 - Skimjoin](22-skimjoin.md#adding-a-skim-output) | +| New segmentation source or relationship | [24 - Segmentation](24-segmentation.md#implementation-and-extension-points) | +| New custom geography behavior | [27 - Geography](27-geography.md#implementation-and-extension-points) | | New generated summary function/table | [44 - Summary Function Cookbook](44-summary-function-cookbook.md) | | New figure or table on existing page | [32 - Figures and Widgets](32-figures-and-widgets.md) | | New dashboard page | [33 - Dashboard Page Recipes](33-dashboard-page-recipes.md) | @@ -54,7 +56,7 @@ activitysim_visualizer/ ## Testing Guidance -Use focused tests for the subsystem you changed: +Run focused tests for the subsystem you changed: - prepare changes: minimal raw/prepared input tests and cache identity tests - skimjoin changes: config normalization, lookup behavior, reports @@ -65,19 +67,22 @@ Use focused tests for the subsystem you changed: Common command: ```bash -uv run --with pytest pytest --basetemp .pytest_tmp +uv run pytest --basetemp .pytest_tmp ``` -Run narrower tests while iterating when possible. -The [Testing](46-testing.md) chapter documents the fast/full marker split and -the required release-boundary commands. +During development, use the smallest relevant test group. The +[Testing](46-testing.md) chapter describes the fast and full markers and gives +the required release test commands. -## Generated Wiki Catalogs +## Generated Documentation Catalogs -Regenerate catalogs after changing: +Regenerate the catalogs after you change: - `@summary(...)` declarations and contracts - `processor/summarize/catalog.py` +- prepared-cache and hypothetical-sidecar contracts +- `scripts/processor_output_catalog_metadata.yaml` +- `scripts/summary_catalog_metadata.yaml` - dashboard page definitions - page data requirements @@ -87,19 +92,22 @@ Command: uv run python scripts/generate_wiki_catalogs.py ``` -Generated sections are marked with comments. Do not edit inside generated -markers by hand. +Comments identify generated sections. Do not edit the text between those +markers by hand. Processor contracts and metadata generate both the detailed +wiki catalog and `reference/processor-output-table-reference.md`. ## Documentation Maintenance -When behavior changes, update docs in the same change: +When behavior changes, update the documentation in the same change: | Change | Wiki updates | |---|---| | Config behavior | `11-configuring-your-data.md` and `13-configuration-reference.md` | | Prepare behavior | `21-prepared-tables.md` | | Skimjoin behavior | `22-skimjoin.md` | -| Summary contract or registration | `23-summary-functions.md`, then regenerate catalogs | +| Segmentation behavior | `24-segmentation.md` and `13-configuration-reference.md` | +| Geography behavior | `27-geography.md` and `13-configuration-reference.md` | +| Summary contract or registration | `25-summary-functions.md`, then regenerate catalogs | | Dashboard page API | `31-dashboard-pages.md`, `32-figures-and-widgets.md`, `33-dashboard-page-recipes.md` | | Export payload/runtime | `34-html-export.md` | | Export payload schema | `36-html-export-schema.md` | @@ -110,8 +118,8 @@ When behavior changes, update docs in the same change: - The change follows the owning subsystem's existing patterns. - Config and cache behavior are explicit. -- Missing optional inputs fail gracefully. -- Summary/page requirements are declared where the runtime can see them. +- Missing optional input gives a controlled result. +- Declare summary and page requirements where the runtime can use them. - Tests cover the behavior rather than only the implementation detail. - Generated wiki catalogs are current. - The fast suite passes, and the `full_export` boundary passes when the change diff --git a/wiki/41-data-extension-cookbook.md b/wiki/41-data-extension-cookbook.md index 295247e..df32bee 100644 --- a/wiki/41-data-extension-cookbook.md +++ b/wiki/41-data-extension-cookbook.md @@ -1,7 +1,7 @@ # 41 - Data Extension Cookbook -This chapter contains end-to-end examples for extending the data that reaches -the dashboard. Each recipe starts at the narrowest supported boundary. +This chapter gives complete examples of data extensions, starting at the +smallest supported boundary for each one. ## Choose The Smallest Extension @@ -9,14 +9,14 @@ the dashboard. Each recipe starts at the narrowest supported boundary. |---|---| | Load a dashboard-ready file produced elsewhere | Register an external summary schema and use `summary_table_map`. | | Reuse one derived value in several summaries | Add a column to an existing prepared table. | -| Carry a genuinely new row grain through the whole application | Add a prepared table. | +| Carry a new row type through the complete application | Add a prepared table. | -Adding a prepared table is much more invasive than adding a column. Prefer a -column unless the new data has its own stable row grain and lifecycle. +A prepared table changes more interfaces than a prepared column. Add a column +unless the new data has its own stable row type and lifecycle. ## Worked Example: Add An Outside Summary Table -Suppose another process writes `regional_emissions.csv`: +In this example, a different process writes `regional_emissions.csv`: ```csv pollutant,tons @@ -24,10 +24,10 @@ CO2,1250.5 NOX,18.2 ``` -The visualizer only accepts registered summary IDs with exact schemas. Register -the outside table with a no-op builder in an owning summary module. For a group -of project-supplied tables, a module such as -`processor/summarize/summaries/external_project.py` is appropriate: +The visualizer accepts only registered summary IDs with exact schemas. Register +the external table with a builder that does not calculate values, and place it +in the relevant summary module. For multiple project tables, use a module such +as `processor/summarize/summaries/external_project.py`: ```python import polars as pl @@ -49,10 +49,10 @@ def regional_emissions(run: RunData, config: Config) -> pl.DataFrame: return regional_emissions.empty() ``` -`build_by_default=False` is important: raw ActivitySim runs cannot build this -table, but the ID and contract must exist so an outside file can be validated. +Set `build_by_default=False` because the standard ActivitySim workflow cannot +build this table. The ID and contract must exist to validate an external file. -If this is a new module, import it and add it to `SUMMARY_MODULES` in +If you add a module, import it and add it to `SUMMARY_MODULES` in `processor/summarize/catalog.py`: ```python @@ -64,7 +64,7 @@ SUMMARY_MODULES = ( ) ``` -Point a run at the file: +Add the file to a run: ```yaml runs: @@ -73,18 +73,18 @@ runs: regional_emissions: inputs/regional_emissions.csv ``` -Relative paths resolve from the main config file. CSV and Parquet are -supported. The loader: +Relative paths start from the main configuration file. For both CSV and Parquet, +the loader performs these checks and actions: 1. rejects unknown summary IDs; 2. rejects missing or unexpected columns; 3. casts to the declared dtypes and declared column order; and 4. exposes the same outside table under every configured weighting mode. -The fourth behavior matters: an outside table is assumed to be already -aggregated. Selecting Weighted or Unweighted does not recalculate it. +The loader treats an external table as already aggregated, so the Weighted and +Unweighted selections do not calculate it again. -Wire the table to a page as optional data: +Connect the table to a page as optional data: ```python @dashboard_page( @@ -103,10 +103,17 @@ class RegionalValidationPage(DashboardPage): return self.plot.bar(data, x="pollutant", y="tons") ``` -Use `required_summary_ids` only if the page has no meaningful primary view -without the table. +Use `required_summary_ids` only if the table is necessary for the primary page +view. -Tests should prove registration, strict schema validation, loading, and page +Add the table's category, analytical use, and output-field descriptions to +`scripts/summary_catalog_metadata.yaml` so chapter 26 and the standalone user +reference can combine them with the registered contract. Add shared input-field +metadata there if the declaration introduces a new prepared requirement. When +adding a prepared table or stable prepared field, also update +`scripts/processor_output_catalog_metadata.yaml`. + +Tests must verify registration, strict schema validation, loading, and page requirements: ```python @@ -130,20 +137,20 @@ def test_external_emissions_loads(tmp_path, config): assert run.summaries_by_mode["weighted"]["regional_emissions"].height == 1 ``` -Run: +Use these commands: ```bash -uv run --with pytest pytest --basetemp .pytest_tmp tests/test_summary_declarations.py tests/test_runtime_workflows.py +uv run pytest --basetemp .pytest_tmp tests/test_summary_declarations.py tests/test_runtime_workflows.py uv run python scripts/generate_wiki_catalogs.py ``` ## Worked Example: Add A Column To An Existing Prepared Table -Suppose several summaries need a canonical household field named -`area_type`. The raw table already contains enough information to derive it. +In this example, several summaries require a canonical household field named +`area_type`. The raw table contains the information to calculate it. -Put the transformation in the enrichment module that owns the domain. For a -household field, that is normally +Put the transformation in the enrichment module for the domain. For a +household field, this module is usually `processor/prepare/enrichment/households_persons.py`: ```python @@ -169,7 +176,7 @@ def enrich_people_and_places_domain(state, config): return state ``` -Then declare the prepared dependency where it is consumed: +Then declare the prepared dependency where the summary uses it: ```python @summary( @@ -192,18 +199,19 @@ def households_by_area_type(run, config): ) ``` -Add a prepare test with the source column present and another with it absent. -Optional source data should leave the table usable; the summary contract will -record the new summary as unavailable when `area_type` is absent. +Add one prepare test with the source column and one without it. If the optional +source data is absent, the table must stay usable. The summary contract records +the new summary as unavailable when `area_type` is absent. -If config affects the derived value, also add that config value to -`prepare_signature_payload()` in `runtime/config/signatures.py`. Otherwise a -prepared cache built with old config could be reused incorrectly. +If configuration changes the derived value, add that configuration value to +`prepare_signature_payload()` in `runtime/config/signatures.py`. Without this +change, the visualizer can incorrectly use a cache from an old configuration. ## Worked Example: Add A Prepared Table -Assume ActivitySim now emits one row per zone in `final_accessibility.csv`, and -the table cannot sensibly be represented as columns on `land_use`. +In this example, ActivitySim writes one row for each zone in +`final_accessibility.csv`. Columns on `land_use` cannot correctly represent +this table. ### 1. Define Names And Runtime Storage @@ -235,8 +243,8 @@ class RunData: ``` Also update `PREPARED_TABLE_NAMES`, `prune_prepared_run()`, and every explicit -`RunData(...)` copy constructor. Copy constructors are intentionally explicit; -missing one is a common source of a table disappearing between workflows. +`RunData(...)` copy constructor. Because the copy constructors list fields +explicitly, missing one can cause a workflow to omit the table. ### 2. Read It And Track Availability @@ -272,16 +280,16 @@ PREPARED_TABLE_ATTRS = ( ) ``` -That one tuple drives prepared filenames, manifest entries, writes, and most +This tuple controls prepared file names, manifest entries, writes, and most loads. Because it changes the prepared cache contract, increment -`SCHEMA_VERSION` and decide whether old schema versions remain readable. +`SCHEMA_VERSION` and decide whether the reader can read older schema versions. ### 4. Decide Segmentation And Dashboard Behavior -If segmentation must filter or anchor on the new table, add explicit rules in -`processor/segmentation.py` and aliases in -`runtime/config/normalize_segmentation.py`. Do not silently copy the full table -into every segment unless that is correct for its row grain. +If segmentation must filter or use the new table as an anchor, add rules in +`processor/segmentation.py`. Add aliases in +`runtime/config/normalize_segmentation.py`. Do not copy the complete table to +each segment unless this is correct for its row type. Pages can now declare: @@ -296,7 +304,7 @@ Pages can now declare: ### 5. Test Every Boundary -At minimum, cover: +At a minimum, test these items: - config filename and `prepared_table_map` acceptance; - raw reader success and optional-file absence; @@ -309,8 +317,8 @@ At minimum, cover: Useful suites: ```bash -uv run --with pytest pytest --basetemp .pytest_tmp tests/test_processor_prepare.py tests/test_prepare_cache.py -uv run --with pytest pytest --basetemp .pytest_tmp tests/test_runtime_workflows.py tests/test_page_registry_contract.py +uv run pytest --basetemp .pytest_tmp tests/test_processor_prepare.py tests/test_prepare_cache.py +uv run pytest --basetemp .pytest_tmp tests/test_runtime_workflows.py tests/test_page_registry_contract.py ``` ## Completion Checklist @@ -319,12 +327,12 @@ uv run --with pytest pytest --basetemp .pytest_tmp tests/test_runtime_workflows. - IDs are stable across config, runtime, cache, and dashboard declarations. - Cache identity changes whenever config changes data content. - Missing optional input produces typed empty/unavailable state, not a crash. -- External schemas reject extra as well as missing columns. +- External schemas reject extra and missing columns. - Generated catalogs have been refreshed. ## Related Chapters - [21 - Prepared Tables](21-prepared-tables.md) -- [23 - Summary Functions](23-summary-functions.md) -- [24 - Summary Catalog](24-summary-catalog.md) +- [25 - Summary Functions](25-summary-functions.md) +- [26 - Summary Catalog](26-summary-catalog.md) - [42 - Config, Columns, and Labels](42-config-column-label-cookbook.md) diff --git a/wiki/42-config-column-label-cookbook.md b/wiki/42-config-column-label-cookbook.md index 02a658a..14fb81c 100644 --- a/wiki/42-config-column-label-cookbook.md +++ b/wiki/42-config-column-label-cookbook.md @@ -1,6 +1,6 @@ # 42 - Config, Columns, And Labels -This chapter shows how one YAML value travels through validation, typed config, +This chapter follows one YAML value through validation, typed configuration, cache identity, prepared data, and dashboard presentation. ## First Decide Which Boundary Owns The Setting @@ -12,13 +12,13 @@ cache identity, prepared data, and dashboard presentation. | labels, ordering, colors, or page appearance | `display` or `dashboard` | Presentation | | which workflow executes | `pipeline` | Runtime plan; include data effects in the owning signature too | -Do not add a setting only to `Config`. A complete setting has validation, -normalization, a typed field, cache/signature ownership, a consumer, an example, +A new setting needs more than a field on `Config`. Add validation, +normalization, a typed field, cache-signature ownership, a consumer, an example, and tests. ## Worked Example: Add A New Config Item -Suppose the dashboard needs a presentation-only switch: +In this example, the dashboard requires a presentation-only control: ```yaml display: @@ -62,12 +62,11 @@ class Config: show_zero_categories: bool ``` -Downstream code should read `config.show_zero_categories`, never the raw YAML -mapping. +Downstream code reads `config.show_zero_categories`, not the raw YAML mapping. ### 3. Put It In The Correct Signature -Because this switch changes only rendering, add it to +This control changes only rendering. Add it to `presentation_signature_payload()` in `runtime/config/signatures.py`: ```python @@ -77,32 +76,32 @@ return { } ``` -Do not add it to the prepare or summary signatures. That would cause expensive -cache rebuilds for a display-only change. +Do not add it to the prepare or summary signatures. If you add it, a display +change causes unnecessary cache rebuilds. ### 4. Consume It At The Presentation Boundary -For example, a shared category helper can choose whether to complete absent -categories: +For example, a shared category helper can add absent categories when the value +is true: ```python if config.show_zero_categories: chart_data = complete_category_rows(chart_data, expected_categories) ``` -Prefer a shared helper if several pages need the setting. Keep one-off behavior -on the owning page. +Use a shared helper when several pages need the setting; keep page-specific +behavior on the page itself. ### 5. Document And Test It -Update `config.yaml` and chapter 13. Add tests for the default, explicit value, -wrong type, signature ownership, and visible consumer behavior: +Update `config.yaml` and chapter 13. Test the default, an explicit value, and an +incorrect type. Also test signature ownership and visible consumer behavior. -The snippets below use illustrative module-local helpers named -`_write_config()` and `_raw_run()`. They are not repository-wide pytest -fixtures: define the minimal helper in the owning test module, or adapt that -module's existing config/run factory. Likewise, `extra_lines` and -`column_lines` are example helper arguments rather than public config APIs. +The examples below use module-local helpers named `_write_config()` and +`_raw_run()`. These helpers are not repository-wide pytest fixtures. Define a +small helper in the relevant test module, or use its existing configuration +and run factory. `extra_lines` and `column_lines` are example helper arguments. +They are not public configuration APIs. ```python def test_show_zero_categories_is_presentation_only(tmp_path): @@ -118,8 +117,8 @@ def test_show_zero_categories_is_presentation_only(tmp_path): ## Worked Example: Wire A Configured Column Name Into Prepare -Suppose different models call household area type `area_type`, `ATYPE`, or -`area_class`. The prepared contract should expose one stable name: +In this example, models use `area_type`, `ATYPE`, or `area_class` for household +area type. The prepared contract must supply one stable name: `area_type`. ### 1. Add The Alias Setting @@ -136,7 +135,7 @@ _ALIAS_COLUMN_DEFAULTS = { } ``` -`CANONICAL_COLUMN_KEYS` is derived from this mapping, so +The loader gets `CANONICAL_COLUMN_KEYS` from this mapping, so `columns.area_type` becomes valid automatically. Add the typed field to `Config`: @@ -144,14 +143,14 @@ _ALIAS_COLUMN_DEFAULTS = { col_area_type: list[str] ``` -The user can now override precedence: +The user can now change the order of preference: ```yaml columns: area_type: [area_class, ATYPE] ``` -The first available candidate wins. +The loader uses the first available candidate. ### 2. Materialize The Canonical Column @@ -167,13 +166,13 @@ def _canonicalize_households(hh: pl.DataFrame, config: Config) -> pl.DataFrame: ) ``` -Keep the configured source candidates in config and the stable output name in -prepare. Summary builders should require `hh.area_type`; they should never -probe `ATYPE` or `area_class`. +Keep configured source candidates in the configuration. Keep the stable output +name in prepare. Summary builders must require `hh.area_type`. They must not +search for `ATYPE` or `area_class`. -Use `_materialize_preferred_column(...)` only when candidate selection needs -extra rules, such as rejecting numeric purpose codes. Use `overwrite=True` only -when prepare intentionally replaces an existing canonical column. +Use `_materialize_preferred_column(...)` only when candidate selection requires +more rules. One example is the rejection of numeric purpose codes. Use +`overwrite=True` only when prepare must replace an existing canonical column. ### 3. Add Cache Identity @@ -184,8 +183,8 @@ Add the candidate list to the `columns` mapping returned by "area_type": list(config.col_area_type), ``` -The summary signature currently incorporates the prepared column payload, so -this also invalidates affected summary caches. +The summary signature includes the prepared column payload, so this change also +invalidates the affected summary caches. ### 4. Test Precedence And Materialization @@ -214,10 +213,10 @@ Also test the default candidate list and missing-source behavior. ## Worked Example: Add A Label Mapping And Use It On A Page -Label mappings are presentation data. They do not change raw values used for -filtering or summary grouping. +Label mappings are presentation data. They do not change raw values for filters +or summary groups. -Suppose a summary contains `employment_status` values `0`, `1`, and `2`: +In this example, a summary contains `employment_status` values `0`, `1`, and `2`: ```yaml display: @@ -231,7 +230,7 @@ display: ``` New category IDs do not require a schema change. `normalize_categories()` loads -arbitrary category IDs into `config.dashboard_labels`. +all category IDs into `config.dashboard_labels`. ### Selector With Display-To-Raw Mapping @@ -257,9 +256,9 @@ def selected_employment_status_raw(self): return self._employment_status_by_label.get(self.employment_status.value) ``` -The widget shows `Full time`; the data filter still uses raw value `2`. This -avoids corrupting joins, selector state, or summary contracts with display -text. +The widget shows `Full time`, while the data filter continues to use raw value +`2`. Display text therefore does not change joins, selector state, or summary +contracts. ### Add A Label Column For A Figure @@ -283,11 +282,10 @@ return self.plot.bar( ) ``` -If many pages use the category, keep mapping mechanics in +If many pages use the category, put mapping logic in `dashboard/helpers/category_helpers.py`. If the mapping changes canonical -summary values rather than appearance, it belongs under -`summarize.category_normalization` and must be applied by the owning summary -logic. +summary values, put it under `summarize.category_normalization`. The relevant +summary logic must apply it. ### Test Raw And Display Behavior Separately @@ -298,13 +296,174 @@ assert config.ordered_values( ) == ["0", "1", "2"] ``` -Add a page/helper test proving that selection of `Full time` filters raw `2`. -This catches the most common label-wiring regression. +Add a page or helper test. Verify that a `Full time` selection filters raw value +`2`. This test identifies a common label connection error. + +## Worked Example: Segment Runs With An External Lookup + +Use a CSV-backed segment when membership does not belong in the canonical model +output. For example, classify households into planning markets without adding a +regional column to prepare. + +Create `lookups/household_market.csv`: + +```csv +household_id,market +1,Core +2,Suburban +3,Rural +``` + +Enable the segment step and join the lookup to prepared households: + +```yaml +pipeline: + steps: [segment, summarize, dashboard] + dashboard_mode: live + refresh: [] + +segment: + dashboard: + segmentation_type: market + visibility: full_and_segments + definitions: + market: + source: + type: csv_lookup + file: lookups\household_market.csv + join: + source_table: hh + source_key_column: household_id + csv_key_column: household_id + segment_value_column: market + allow_overlapping: false + on_empty_segment: error + segments: + - id: core + label: Core + values: [Core] + - id: suburban + label: Suburban + values: [Suburban] + - id: rural + label: Rural + values: [Rural] +``` + +The household anchor keeps each matched household and its related people, +tours, trips, days, vehicles, and joint tours. Every default summary then runs +for each market and weighting mode. Output appears below: + +```text +summary_tables//segments/market// +``` + +Before using a large lookup, check that every key and market value is nonblank +and that one key does not map to different values. Decide whether missing CSV +keys should remain only in the full run or indicate an incomplete lookup. The +runtime permits missing keys but rejects joins that duplicate anchor rows. + +See [24 - Segmentation](24-segmentation.md) before using a person-, trip-, or +other lower-level anchor; the selected relationship changes how population +totals should be interpreted. + +## Worked Example: Add A Custom Geography From CSV + +Use a named geography aggregation when several summaries or pages need the same +zone grouping. Suppose `lookups/maz_district.csv` contains: + +```csv +MAZ,district +101,North +102,North +201,South +``` + +Configure the lookup once: + +```yaml +zones: + use_maz: true + maz_col: [MAZ, zone_id] + taz_col: [TAZ, taz] + +summarize: + geography: + enabled: true + aggregations: + district: + source_zone_system: maz + file: lookups\maz_district.csv + zone_id_col: MAZ + geography_col: district + +display: + labels: + geography: + mapping: + district: Planning District +``` + +Prepare creates role-specific columns such as `home_geo__district`, +`work_geo__district`, `origin_geo__district`, +`destination_geo__district`, and `land_use_geo__district`. Supporting summaries +emit `geography_type: district` and the mapped district label as +`geography_id`. + +The display mapping changes only the visible name of the geography type. It +does not change zone membership. Keep the zone join under +`summarize.geography.aggregations` and presentation text under +`display.labels.geography`. + +Test at least one zone from each district, an unmapped zone, and a conflicting +duplicate zone. Changing the CSV changes prepare and summary identity, so valid +old caches are not reused. See [27 - Geography](27-geography.md) for every +generated column and source-zone rule. + +## Worked Example: Share Skimjoin Rules Across Runs + +Keep skimjoin lookup behavior in one rules file and put model-specific data +paths in the main visualizer config. This avoids copying mode and component +rules for every scenario. + +Main config: + +```yaml +pipeline: + steps: [prepare, skimjoin, summarize, dashboard] + +skimjoin: + defaults: + config_path: configs\skimjoin_rules.yaml + skim_files: + - skims\base\*.omx + network_los_file: skims\base\network_los.yaml + +runs: + - dir: outputs\base + label: Base + - dir: outputs\build + label: Build + skimjoin: + skim_files: + - skims\build\*.omx + network_los_file: skims\build\network_los.yaml +``` + +`configs/skimjoin_rules.yaml` contains `activitysim`, `defaults`, `dimensions`, +and `modes`, but no `project` block. Integrated prepare supplies its trip and +tour tables, while the main config supplies the paths. + +Repeat all required path overrides in a run-specific block. Once an override +causes the selected rules file to be reloaded, an omitted skim or network path +comes from that rules file rather than from the other global path overrides. +See [22 - Skimjoin](22-skimjoin.md#where-path-settings-belong) for the full +precedence rules. ## Completion Checklist - Unknown keys and wrong types fail near the config boundary. -- Raw YAML is normalized once and represented by a typed `Config` field. +- Normalize raw YAML one time and represent it with a typed `Config` field. - The setting belongs to exactly the cache signatures it can affect. - Prepared code emits canonical names; summaries do not probe source aliases. - Dashboard filtering retains raw values and labels only at presentation time. @@ -314,6 +473,9 @@ This catches the most common label-wiring regression. - [13 - Configuration Reference](13-configuration-reference.md) - [21 - Prepared Tables](21-prepared-tables.md) +- [22 - Skimjoin](22-skimjoin.md) +- [24 - Segmentation](24-segmentation.md) +- [27 - Geography](27-geography.md) - [32 - Figures And Widgets](32-figures-and-widgets.md) - [41 - Data Extension Cookbook](41-data-extension-cookbook.md) - [43 - Weighting And Hosting Extensions](43-weighting-hosting-extensions.md) diff --git a/wiki/43-weighting-hosting-extensions.md b/wiki/43-weighting-hosting-extensions.md index 41538d6..3fb2a98 100644 --- a/wiki/43-weighting-hosting-extensions.md +++ b/wiki/43-weighting-hosting-extensions.md @@ -1,18 +1,18 @@ # 43 - Weighting And Hosting Extensions -Weighting and hosting both cross major runtime boundaries. Ordinary alternative -weights are configuration-driven. A Python registry remains available for -calculations that cannot be expressed as column selection, while hosting remains -a deliberately limited extension point. +Weighting and hosting cross several runtime boundaries. Configuration handles +standard alternative weights, while the Python registry supports calculations +that column selection cannot express. Hosting remains a limited extension point. ## Worked Example: Add A Weighting Mode -The built-in modes are `weighted` and `unweighted`. A named column mode adds -another complete set of summary tables, cache entries, dashboard selector state, -and export states without requiring Python code. +The built-in modes are `weighted` and `unweighted`. A named column mode adds a +set of summary tables, cache entries, dashboard selector state, and export +states. It does not require Python code. -Suppose ActivitySim writes `calibrated_hh_weight`, `calibrated_person_weight`, -and `calibrated_trip_weight` alongside its ordinary weights. +In this example, ActivitySim writes `calibrated_hh_weight`, +`calibrated_person_weight`, and `calibrated_trip_weight` with its standard +weights. ### 1. Define The Named Column Mode @@ -33,60 +33,68 @@ summarize: weighting_modes: [weighted, unweighted, calibrated] ``` -`label` is optional; an omitted label is generated from the mode ID. At least one -column must be configured. Supported source tables are `households`, `persons`, -and `trips`. +`label` is optional; if omitted, the loader creates one from the mode ID. You +must configure at least one column from a supported source table: +`households`, `persons`, or `trips`. -This differs from the three weight fields on a run. `hh_weight_col`, -`person_weight_col`, and `trip_weight_col` choose the one primary `weighted` -definition during prepare. `weighting.modes` preserves that primary definition -and adds named alternatives that can be compared in one dashboard. +This function differs from the four weight fields on a run. `hh_weight_col`, +`person_weight_col`, and `trip_weight_col` select the primary `weighted` +definition for those source levels during prepare. `day_weight_col` selects a +day-specific source and defaults to `day_weight`; null day values inherit from +persons or households. Declarative named modes accept household, person, and +trip sources and propagate those choices to days. `weighting.modes` keeps the +primary definition and adds named alternatives for comparison in one dashboard. ### 2. Understand Propagation -The configured source columns replace `finalweight` on their respective -prepared tables. Related tables then receive consistent weights: +The configured source columns replace `finalweight` on their prepared tables. +The system then supplies consistent weights to related tables: - a household source propagates to persons, trips, tours, days, and vehicles - unless a more specific source is configured; + unless you configure a more specific source; - a person source propagates to trips, tours, and days; - a trip source propagates to tours as the mean selected trip weight for each `tour_id`; and - trip and tour hypothetical-skim sidecars inherit the selected trip and tour weights. -You can configure only the levels that differ. For example, a mode containing -only `trips` changes trips and tours while leaving household, person, day, and -vehicle weights at their primary prepared values. +Configure only the levels that differ. For example, a mode that contains only +`trips` changes trips and tours. Household, person, day, and vehicle weights +keep their primary prepared values. -Source columns are validated on every prepared run before summaries begin. A -misspelling therefore produces an error naming the missing table and column -instead of silently reverting to another weight. Raw ActivitySim columns are -normally retained by prepare. When using `prepared_table_map`, include the named -source columns in those prepared files. +Before summaries start, the workflow validates source columns for each prepared +run. An incorrect name produces an error identifying the missing table and +column; it does not select a different weight. Prepare usually preserves raw +ActivitySim columns. If you use `prepared_table_map`, include the named source +columns in those prepared files. + +Column validation checks presence and castability during use; it does not +enforce finite, non-null, or nonnegative values. Validate those properties in +the producing workflow. Zero and negative values can produce zero denominators +or subtract from counts, and null source weights can be omitted by aggregation. +See [Summary Functions](25-summary-functions.md#weight-resolution-and-edge-cases) +for primary-mode fallback and sample-rate behavior. ### 3. Cache, Dashboard, And Outside-Summary Behavior -The mode ID, selected source columns, and column-mode implementation version are -part of summary cache identity. Changing a source column invalidates incompatible -summary caches. The configured label is used by live and exported dashboard -selectors. +The summary cache identity includes the mode ID, source columns, and column-mode +implementation version. A source column change invalidates incompatible summary +caches. Live and export dashboard selectors use the configured label. -Declarative column modes reject mode-independent `summary_table_map` inputs. -An already aggregated outside table does not contain enough information to -recalculate another weighting mode. Use generated summaries for these modes or -provide the outside data through a custom workflow that makes its weighting -semantics explicit. +Declarative column modes reject mode-independent `summary_table_map` input. An +aggregated external table does not contain enough information to calculate +a different weighting mode. Use generated summaries for these modes. Or supply +external data through a custom workflow that defines its weighting rules. ## Advanced: Custom Weight Calculations -Use a Python weighting module only when selecting columns is insufficient; for -example, when weights must be capped, scaled, joined from a control table, or -calculated from several prepared columns. +Use a Python weighting module only when column selection is insufficient. For +example, use one to limit, scale, join, or calculate weights from multiple +prepared columns. ### 1. Create An Importable Extension Module -This example adds a capped form of the primary prepared weights. Create +This example adds a limited form of the primary prepared weights. Create `my_project/weighting.py` in an installed package or another location on `PYTHONPATH`: @@ -130,9 +138,9 @@ def register_weighting_modes(registry: WeightingModeRegistry) -> None: ) ``` -`map_run_data_tables()` copies the complete `RunData`, transforms each DataFrame -table, and preserves availability metadata, diagnostics, skims, and skimjoin -artifacts. A transform must return a new `RunData` and must not mutate its input. +`map_run_data_tables()` copies the complete `RunData` and transforms each data +frame while preserving availability metadata, diagnostics, skims, and skimjoin +artifacts. A transform must return a new `RunData` without changing its input. The registration fields are: @@ -148,7 +156,7 @@ The registration fields are: ### 2. Load And Configure The Extension -Use `extensions.modules` for a project-local/importable module and keep plugin +Use `extensions.modules` for an importable project module. Put extension settings under `extensions.settings`: ```yaml @@ -163,9 +171,9 @@ summarize: weighting_modes: [weighted, unweighted, capped] ``` -Module imports are executable code, so configuration containing extensions is -trusted configuration. Extension settings and each selected definition's -version, requirements, and outside-summary policy enter summary cache identity. +Module imports execute code, so treat any configuration with extensions as +trusted. Summary cache identity includes the extension settings along with each +selected definition's version, requirements, and external summary policy. An installed package can advertise the same registration function with a Python entry point instead: @@ -175,14 +183,13 @@ Python entry point instead: capped = "my_project.weighting:register_weighting_modes" ``` -Use either the installed entry point or `extensions.modules`, not both for the -same definition. Duplicate IDs and labels fail during config loading. +For one definition, use the installed entry point or `extensions.modules`. Do +not use both. Duplicate IDs and labels cause an error during configuration load. ### 3. Runtime Behavior -The weighting definition contract is the single source for config validation, -summary transforms, prepared-data transforms, display labels, and cache -compatibility: +The weighting definition contract controls configuration validation, summary +transforms, prepared-data transforms, display labels, and cache compatibility: - config preserves the requested mode order and rejects unknown IDs; - the summary workflow applies each registered transform before running builders; @@ -193,7 +200,7 @@ compatibility: cache the result for the dashboard session; and - required source columns fail before a transform can silently fall back. -Ordinary pages do not branch on particular modes: +Standard pages do not branch on particular modes: ```python prepared = self.data.prepared("trips") @@ -204,17 +211,17 @@ weighted = self.data.prepared("trips", weighting_mode="weighted") ### 4. Outside Summary Tables -Built-in `weighted` and `unweighted` definitions explicitly use -`external_summary_policy="copy"`, preserving current behavior. A custom mode -defaults to `reject`: a run using `summary_table_map` then fails clearly because -the runtime cannot prove that an already-aggregated file represents that mode. +Built-in `weighted` and `unweighted` definitions use +`external_summary_policy="copy"`. A custom mode uses `reject` by default. A run +with `summary_table_map` then causes an error. The runtime cannot verify that an +aggregated file represents the custom mode. -Set the custom definition to `copy` only when the outside table is genuinely -mode-independent. Per-mode outside file maps are not currently supported. +Set the custom definition to `copy` only when the external table does not +depend on the mode. The system does not support file maps for each mode. ### 5. Test The Whole Mode -At minimum, prove: +At a minimum, verify these behaviors: - config accepts, orders, deduplicates, and rejects mode names correctly; - both module and installed-entry-point discovery use the registration contract; @@ -228,20 +235,25 @@ At minimum, prove: Useful suites: ```bash -uv run --with pytest pytest --basetemp .pytest_tmp tests/test_weighting_registry.py -uv run --with pytest pytest --basetemp .pytest_tmp tests/test_config_refactor_phase1.py tests/test_summary_cache.py -uv run --with pytest pytest --basetemp .pytest_tmp tests/test_dashboard_live.py tests/test_export_payload.py +uv run pytest --basetemp .pytest_tmp tests/test_weighting_registry.py +uv run pytest --basetemp .pytest_tmp tests/test_config_refactor_phase1.py tests/test_summary_cache.py +uv run pytest --basetemp .pytest_tmp tests/test_dashboard_live.py tests/test_export_payload.py ``` ## Worked Example: Connect A Hosting Script -The safest first hosting extension is a thin deployment entrypoint that uses -the existing config, cache loader, page requirements, and `build_dashboard()`. -It should not duplicate prepare or summarize logic. +This section is for developers who need to deploy the live Python/Panel +application. To publish an existing standalone HTML export instead, use +[17 - Publish An Export With Posit Connect Cloud](17-posit-connect-cloud.md). +Static publishing does not require a hosting adapter or server-side caches. + +The first hosting extension should be a small deployment entry point. Use the +existing configuration, cache loader, page requirements, and `build_dashboard()`. +Do not duplicate prepare or summarize logic. -The current `pipeline.dashboard_mode: host` is only a placeholder: `run.py` -logs a warning and falls back to live `pn.serve`. The `dashboard.host` keys are -validated but are not yet normalized into `Config` or consumed. +The current `pipeline.dashboard_mode: host` is a placeholder. `run.py` writes a +warning and uses live `pn.serve`. Validation accepts the `dashboard.host` keys. +The loader does not put them in `Config`, and the runtime does not use them. ## Option A: Provider Script Without Core Runtime Changes @@ -290,12 +302,32 @@ dashboard = build_dashboard( dashboard.servable() ``` -Panel-compatible hosts can launch this module with their normal command. A -provider SDK can instead receive `dashboard` from the same script. Keep secrets -and deployment IDs in environment variables or provider configuration, not the -main visualizer YAML. +Start it locally from the repository root: + +```powershell +$env:ACTIVITYSIM_VIZ_CONFIG = "C:\deploy\activitysim_viz\config.yaml" +uv run panel serve scripts/host_dashboard.py --address 127.0.0.1 --port 5006 +``` + +For a service behind a reverse proxy, bind the process to all container/host +interfaces and allow the public WebSocket origin: + +```powershell +uv run panel serve scripts/host_dashboard.py --address 0.0.0.0 --port 5006 --allow-websocket-origin dashboard.example.org +``` -This approach has useful properties: +Use the scheme/host value expected by the deployed Panel version when the +public URL is nonstandard, and repeat the origin option if the deployment has +several valid hosts. The proxy must forward WebSocket upgrade headers as well +as ordinary HTTP. Terminate TLS and enforce authentication in the proxy or the +chosen hosting provider unless the deployment deliberately adds those concerns +to the application. + +A provider SDK can instead receive `dashboard` from the same script. Put +secrets and deployment IDs in environment variables, a secret store, or +provider configuration. Do not put them in the main visualizer YAML. + +This approach has the following properties: - hosting imports a ready-to-serve object instead of calling blocking `pn.serve()`; @@ -303,15 +335,33 @@ This approach has useful properties: - enabled pages determine the data loaded; and - provider dependencies can live in an optional dependency group. -For a hosted service, caches must already exist or be available on persistent -storage. If startup should build them, call the public prepare/summarize -workflows before `build_dashboard()` and make the cost and write permissions -explicit. +For a hosted service, caches must exist in persistent storage. To build caches +at startup, call the public prepare and summarize workflows before +`build_dashboard()`. Make the runtime cost and write permissions explicit. + +### Deployment Requirements + +Before treating the command as a production service, verify: + +| Requirement | Deployment rule | +|---|---| +| Code/imports | Install the package or start from a working directory where `dashboard`, `processor`, and `runtime` are importable. Keep the deployed code version aligned with the cache schema. | +| Configuration | Set `ACTIVITYSIM_VIZ_CONFIG` to an explicit readable file. Resolve relative paths intentionally; absolute cache/input paths are safer in containers. | +| Summary caches | Mount `` as persistent readable storage. All enabled summary-backed pages need compatible run manifests. | +| Prepared caches | Mount them when any enabled live page has optional or required prepared data. HTML export alone cannot replace this live requirement. | +| Permissions | Read-only cache mounts are sufficient when artifacts are built before deployment. Grant writes only when startup intentionally builds or refreshes caches. | +| Network | Expose the selected port, configure the public WebSocket origin, and preserve WebSocket upgrades through the proxy/load balancer. | +| Sessions and memory | Panel creates server-side sessions. Size workers for the loaded summary/prepared data and expected concurrent sessions; do not assume a standalone HTML memory profile. | +| Security | Put TLS, authentication, secrets, and access logs in the provider/proxy boundary unless a reviewed adapter owns them. | +| Startup failure | Fail the deployment when config or required caches cannot load. Do not serve a process that silently has no configured runs. | + +The `panel serve` command above loads caches and serves the app. It does not run +prepare or summarize. Build and validate artifacts in a separate deployment +step unless startup generation is an explicit operational choice. ## Option B: Make `dashboard_mode: host` A Core Adapter -Use this only when the same hosting provider should be a supported runtime -mode. +Use this approach only when one hosting provider must be a supported runtime mode. 1. Add a typed `HostSettings` model in `runtime/config/models.py`. 2. Normalize `dashboard.host` in a focused parser and pass it into `Config`. @@ -325,7 +375,7 @@ mode. 5. Let `resolve_dashboard_execution_mode("host")` remain `host` instead of converting it to `live`. -6. Reuse the normal workflow loading and `build_dashboard()` path, then call +6. Reuse the standard workflow loading and `build_dashboard()` path, then call the adapter instead of `pn.serve()`. 7. Put provider SDKs in a `hosting` optional dependency group in `pyproject.toml`. @@ -335,15 +385,14 @@ The boundary should look like: ```text config + validated caches - -> normal dashboard data requirements + -> standard dashboard data requirements -> build_dashboard(...) -> provider adapter -> hosted application ``` -Avoid putting provider logic in pages, `dashboard/app.py`, or summary -workflows. Those layers should remain usable locally, in export, and with any -future host. +Do not put provider logic in pages, `dashboard/app.py`, or summary workflows. +These layers must operate locally, in export, and with a future host. ## Hosting Test Matrix @@ -358,6 +407,7 @@ future host. ## Related Chapters - [12 - Running Workflows](12-running-workflows.md) +- [17 - Publish An Export With Posit Connect Cloud](17-posit-connect-cloud.md) - [34 - HTML Export](34-html-export.md) - [40 - Developer Workflows](40-developer-workflows.md) - [41 - Data Extension Cookbook](41-data-extension-cookbook.md) diff --git a/wiki/44-summary-function-cookbook.md b/wiki/44-summary-function-cookbook.md index 8026eef..9c5ad84 100644 --- a/wiki/44-summary-function-cookbook.md +++ b/wiki/44-summary-function-cookbook.md @@ -1,20 +1,20 @@ # 44 - Summary Function Cookbook -This chapter follows one new summary from a question to a tested dashboard -dependency. Use it with the shorter contract reference in chapter 23. +This chapter shows how to create and test one dashboard summary. Use it with the +short contract reference in chapter 25. ## Worked Example: Trips By Mode -Suppose a page needs total trips by canonical `trip_mode`. The output grain is -one row per mode, per run, per weighting mode: +In this example, a page requires total trips by canonical `trip_mode`. The +output has one row for each mode, run, and weighting mode: | trip_mode | trip_count | |---|---:| | DRIVEALONE | 14230.0 | | WALK | 3180.0 | -Write the grain down first. It determines the grouping keys, schema, tests, and -figure axes. +First, define what one row represents because that decision controls grouping +keys, schema, tests, and figure axes. ## 1. Put Pure Calculation Before Registration @@ -39,8 +39,8 @@ def trips_by_mode_frame(trips: pl.DataFrame) -> pl.DataFrame: ) ``` -Keeping the transform pure makes the calculation easy to test without cache or -dashboard setup. Use canonical prepared columns; do not probe raw aliases here. +A pure transform is easy to test without setting up caches or a dashboard. Use +canonical prepared columns rather than searching for raw aliases here. ## 2. Declare The Runtime Contract @@ -66,21 +66,21 @@ def trips_by_mode(run: RunData, config: Config) -> pl.DataFrame: return trips_by_mode_frame(run.trips) ``` -The declaration does four jobs: +The declaration does four tasks: 1. gives the table a stable config/cache ID; 2. prevents the builder from running when inputs are unavailable; 3. supplies a correctly typed empty result; and 4. rejects successful results with wrong columns, order, or dtypes. -The unused `config` argument is still part of the uniform builder interface. If -config changes the calculation, use it here and ensure the setting belongs to -the summary signature. +The uniform builder interface includes `config` even when this example does not +use it. If configuration changes the calculation, use the argument and include +the setting in the summary signature. ## 3. Let The Workflow Handle Weighting -Always aggregate `finalweight`. The workflow supplies ordinary weights for the -weighted build and replaces them for the unweighted build. Do not add a +Always aggregate `finalweight`. The workflow supplies standard weights for the +weighted build. It replaces them for the unweighted build. Do not add a `weighted` branch to the builder. For an average, use a weighted numerator and denominator: @@ -94,21 +94,20 @@ For an average, use a weighted numerator and denominator: ) ``` -Decide how zero total weight should behave and test it explicitly. +Define the result for zero total weight and test it. ## 4. Register A New Owning Module Only Once -Adding a function to an existing module in `SUMMARY_MODULES` needs no catalog -edit. If you create `processor/summarize/summaries/emissions.py`, import that -module and add it to `SUMMARY_MODULES` in `processor/summarize/catalog.py`. +Adding a function to an existing module in `SUMMARY_MODULES` requires no catalog +change. If you create `processor/summarize/summaries/emissions.py`, import it and +add it to `SUMMARY_MODULES` in `processor/summarize/catalog.py`. -Do not maintain a second list of individual functions. Catalog discovery reads -decorated functions from the explicitly imported owning modules and rejects -duplicate IDs. +Do not keep a second list of functions. Catalog discovery reads decorated +functions from the imported modules. It rejects duplicate IDs. ## 5. Test Calculation And Contract Separately -Test the numbers with a tiny frame: +Test the numbers with a small frame: ```python def test_trips_by_mode_frame_uses_finalweight(): @@ -131,7 +130,7 @@ def test_trips_by_mode_frame_uses_finalweight(): } ``` -Then test the declaration boundary with a minimal `RunData`: +Then test the declaration boundary with a small `RunData`: ```python def test_trips_by_mode_preflights_missing_columns(): @@ -158,9 +157,9 @@ def test_trips_by_mode_preflights_missing_columns(): } ``` -Also add a catalog assertion when a new module is introduced. The shared -declaration tests already cover generic wrong-schema behavior; domain tests -should focus on your calculation and prerequisites. +Also add a catalog assertion when you add a module. The shared declaration +tests cover general incorrect-schema behavior. Domain tests must test the +calculation and requirements. ## 6. Wire It To A Page @@ -177,7 +176,7 @@ class TripModeTotalsPage(DashboardPage): ... ``` -Read the table through page data access and state the columns the view uses: +Read the table through page data access. Specify the columns that the view uses: ```python data = self.data.summary( @@ -196,59 +195,62 @@ return self.plot.bar( ) ``` -The page declaration controls cache pruning and startup requirements. The -`columns=` check provides a useful page-level diagnostic if an old or external -cache does not satisfy the view. +The page declaration controls the removal of unused cache data and startup +requirements. The `columns=` check gives a page diagnostic if a cache does not +supply the view. This can occur with an old or external cache. ## 7. Regenerate And Verify -Run: +Add the new summary's category, analytical use, and output-field descriptions to +`scripts/summary_catalog_metadata.yaml`. Add shared input-field metadata when +the declaration introduces a new prepared requirement. Update +`scripts/processor_output_catalog_metadata.yaml` for any new prepared output, +then use these commands: ```bash uv run python scripts/generate_wiki_catalogs.py -uv run --with pytest pytest --basetemp .pytest_tmp tests/test_summary_declarations.py tests/test_page_registry_contract.py +uv run pytest --basetemp .pytest_tmp tests/test_summary_declarations.py tests/test_page_registry_contract.py ``` -Confirm the new ID appears in chapter 24 and, once wired to a page, in the page -catalog in chapter 31. +Make sure the new ID appears in chapter 26. After connecting it to a page, make +sure it also appears in the chapter 31 page catalog. ## Variations ### Optional Summary -Use `optional_summary_ids` when the page retains a meaningful primary view -without the new table. Render an unavailable card only for the optional -section. +Use `optional_summary_ids` when the page has a useful primary view without the +new table. Show an unavailable card only for the optional section. ### External-Only Summary -Use `build_by_default=False` and a typed no-op builder for a registered table -that must come from `summary_table_map`. Follow the outside-table recipe in -chapter 41. +Use `build_by_default=False` and a typed builder that does not calculate values. +Use this configuration for a registered table from `summary_table_map`. Follow +the external-table procedure in chapter 41. ### Segmented Summary -Usually no builder change is needed. Segmentation slices prepared `RunData` -before invoking the same declaration. A summary that depends on a table or -column removed by segmentation should become unavailable through its declared -prerequisites, not fail inside the builder. +Usually, the builder does not require a change. Segmentation divides prepared +`RunData` before it calls the same declaration. Segmentation can remove a +required table or column. In this condition, the declared requirements must +make the summary unavailable. The builder must not fail. ## Review Checklist -- The row grain and value meaning are written down. +- The documentation defines the row type and value meaning. - Grouping uses canonical prepared fields. -- Counts, totals, and averages apply `finalweight` deliberately. +- Counts, totals, and averages apply `finalweight` correctly. - The schema is ordered and explicitly cast. - Mechanical prerequisites are in the decorator. - Domain-specific empty conditions return `builder.empty()`. - Pure calculation and declaration behavior have focused tests. - The consuming page declares the ID. -- Generated catalogs are current. +- Summary metadata is complete and generated catalogs are current. ## Related Chapters - [21 - Prepared Tables](21-prepared-tables.md) -- [23 - Summary Functions](23-summary-functions.md) -- [24 - Summary Catalog](24-summary-catalog.md) +- [25 - Summary Functions](25-summary-functions.md) +- [26 - Summary Catalog](26-summary-catalog.md) - [41 - Data Extension Cookbook](41-data-extension-cookbook.md) - [45 - Dashboard Extension Cookbook](45-dashboard-extension-cookbook.md) diff --git a/wiki/45-dashboard-extension-cookbook.md b/wiki/45-dashboard-extension-cookbook.md index f0f4452..f7d6641 100644 --- a/wiki/45-dashboard-extension-cookbook.md +++ b/wiki/45-dashboard-extension-cookbook.md @@ -1,20 +1,22 @@ # 45 - Dashboard Extension Cookbook -This chapter gives worked examples for adding a page, page group, selector, -custom widget, table, and reusable figure behavior. The examples use the -current declarative page lifecycle: selectors own option domains, sections own -refresh dependencies, and pages read data through `self.data`. +This chapter gives examples of a page, page group, selector, custom widget, +table, and reusable figure. They use the declarative page lifecycle: selectors +control option domains, sections control refresh dependencies, and pages read +data through `self.data`. ## Worked Example: Add A Page To An Existing Group -Assume the registered summary `trips_by_mode` has columns `trip_mode` and -`trip_count`. Create one discoverable leaf module: +In this example, `trips_by_mode` is the worked example summary created in +[44 - Summary Function Cookbook](44-summary-function-cookbook.md). It is not a +built-in summary until you add that declaration. It has `trip_mode` and +`trip_count` columns. Create one discoverable final module: ```text dashboard/pages/trip_summaries/trip_mode_totals.py ``` -The complete first version can stay small: +The complete first version can be small: ```python from __future__ import annotations @@ -56,8 +58,8 @@ class TripModeTotalsPage(DashboardPage): ) ``` -Discovery imports public child modules automatically. Do not edit a central -page list. The decorator is the single source for identity and data needs. +Discovery imports public child modules automatically, so there is no central +page list to edit. The decorator defines identity and data requirements. Enable the page explicitly while developing: @@ -71,8 +73,8 @@ dashboard: ## Add A Dynamic Selector -Suppose the summary instead contains `tour_purpose`, `trip_mode`, and -`trip_count`. Add a purpose dropdown whose options come from the loaded data: +In this example, the summary contains `tour_purpose`, `trip_mode`, and +`trip_count`. Add a purpose list with options from the loaded data: ```python from dashboard.helpers.category_helpers import column_options @@ -98,7 +100,7 @@ def build_page(self): def purpose_options(self): - data = self.data.summary("trips_by_mode_and_purpose") + data = self.data.summary("trip_mode_by_tour_purpose_and_tour_mode") options, self._purpose_by_label = column_options( data.to_list(), "tour_purpose", @@ -108,15 +110,14 @@ def purpose_options(self): return options ``` -The option provider runs before a dependent section renders. If available -options change, the framework repairs a stale selection using the selector's -`default` policy. +The option provider runs before the framework renders a dependent section. If +the available options change, the `default` policy repairs an invalid selection. -Filter with the raw value, not its display label: +Use the raw value for the filter. Do not use its display label: ```python raw_purpose = self._purpose_by_label[self.purpose.value] -data = self.data.summary("trips_by_mode_and_purpose") +data = self.data.summary("trip_mode_by_tour_purpose_and_tour_mode") chart_data = self.query( lambda: data.where(tour_purpose=raw_purpose).select( "trip_mode", "trip_count" @@ -124,8 +125,8 @@ chart_data = self.query( ) ``` -`self.query()` derives its cache identity from global state, active section, -declared selectors, callable location, and captured values. Do not invent a +`self.query()` gets its cache identity from global state, the active section, +declared selectors, callable location, and captured values. Do not create a page-local cache key. ## Add A Custom Widget @@ -147,7 +148,7 @@ body = self.section( ) ``` -Then apply its value inside the section query: +Then use its value in the section query: ```python if self.hide_auto.value: @@ -158,9 +159,9 @@ if self.hide_auto.value: ) ``` -Registration is what connects the widget to refresh and HTML export. A widget -created directly in the layout without `self.select()` or `self.selector()` is -not part of that lifecycle. +Registration connects the widget to refresh and HTML export. A widget created +directly in the layout is outside this lifecycle, so register it with +`self.select()` or `self.selector()`. ## Add A Figure With The Existing Plotter @@ -178,11 +179,11 @@ chart = self.plot.bar( ) ``` -This applies run colors, count/share state, layout conventions, and hover -behavior. Available shared types are `bar`, `line`, `density`, and `scatter`. +This method applies run colors, count or share state, layout rules, and hover +behavior. The shared types are `bar`, `line`, `density`, and `scatter`. -If one page needs a Plotly customization, build the figure through the escape -hatch, mutate it, and wrap it: +If one page requires a Plotly customization, build the figure through the +figure API. Change it and put it in a Panel pane: ```python figure = self.plot.figure.bar( @@ -195,13 +196,13 @@ figure.update_layout(legend_title_text="Model Run") return self.plot.panel(figure) ``` -Keep ordinary titles, axes, modes, category order, and sizing in the shared -arguments rather than post-processing every page. +Set standard titles, axes, modes, category order, and size with shared +arguments. Do not make these changes separately on each page. ## Add A Reusable Figure Type -When several pages need a genuinely new chart contract, add it to the shared -renderer instead of copying Plotly construction. +When several pages require a new chart contract, add it to the shared renderer. +Do not copy the Plotly construction. For an area chart: @@ -213,8 +214,8 @@ For an area chart: 4. validate required columns with the same clear errors as other builders; and 5. test the Plotly figure before testing Panel wrapping. -Here is a complete minimal builder for `dashboard/rendering/figures.py`. It -uses the existing internal helpers because it lives beside the other builders: +This is a complete small builder for `dashboard/rendering/figures.py`. It uses +the existing internal helpers with the other builders: ```python def area_figure( @@ -261,11 +262,10 @@ def area_figure( return figure ``` -`ChartTables`, `ChartValueMode`, `go`, and `np` are already used by that -module. The explicit `value_mode` keeps `"dashboard"`, forced count, and forced -share behavior consistent with the existing figure types. `_require_columns` -provides a run-specific error, while `RenderContext.color()` preserves the -configured run-color mapping. +That module already uses `ChartTables`, `ChartValueMode`, `go`, and `np`. The +explicit `value_mode` keeps `"dashboard"`, count, and share behavior consistent +with existing figure types. `_require_columns` gives an error for the affected +run. `RenderContext.color()` keeps the configured run-color mapping. The adapter shape is: @@ -280,7 +280,7 @@ class Plotter: return self.panel(self.figure.area(data, **kwargs)) ``` -A focused test should inspect traces and layout: +A focused test must examine traces and layout: ```python def test_area_figure_uses_run_labels_and_colors(): @@ -326,9 +326,9 @@ return data_table( ) ``` -It produces one run tab per frame and applies shared column titles and numeric -formatting. Use a page-local `Tabulator` only when the shared table contract -cannot express the required interaction. +It creates one run tab for each frame and applies shared column titles and +numeric formatting. Use a page-local `Tabulator` only when the shared table +contract cannot provide the required interaction. ## Add A New Page Group @@ -356,11 +356,11 @@ GROUP = DashboardGroupDefinition( ) ``` -Every child page declares `group_id="emissions"`. `default_page_id` must name -one of those children. Private helper packages and modules begin with `_` so -discovery ignores them. +Each child page declares `group_id="emissions"`. `default_page_id` must specify +one of these children. Start private helper package and module names with `_`. +Discovery ignores these names. -Users can enable the group's default pages or choose children: +Users can enable the default group pages or select child pages: ```yaml dashboard: @@ -374,8 +374,8 @@ dashboard: ## Test The Extension -Test pure transforms separately from lifecycle wiring. Then add focused checks -for declarations: +Test pure transforms separately from lifecycle connections. Then add focused +checks for declarations: ```python def test_trip_mode_page_declares_its_runtime_contract(): @@ -386,18 +386,18 @@ def test_trip_mode_page_declares_its_runtime_contract(): assert definition.required_summary_ids == ("trips_by_mode",) ``` -For selector behavior, instantiate a small test page with `DashboardState`, -change the option provider's domain, refresh, and assert that stale values are -repaired. For figures, test `Plotter(RenderContext()).figure` so failures are -independent of Panel. The full registry suites then prove discovery, unique -IDs, requirements, and export protocol support. +For selector behavior, create a small test page with `DashboardState`. Change +the option provider domain and refresh the page. Verify that the framework +repairs invalid values. For figures, test `Plotter(RenderContext()).figure`. +This keeps failures independent of Panel. The full registry tests verify +discovery, unique IDs, requirements, and export protocol support. -Run at least: +Use at least these commands: ```bash uv run python scripts/generate_wiki_catalogs.py -uv run --with pytest pytest --basetemp .pytest_tmp tests/test_page_authoring.py tests/test_page_registry_contract.py -uv run --with pytest pytest --basetemp .pytest_tmp tests/test_figure_builders.py tests/test_export_payload.py +uv run pytest --basetemp .pytest_tmp tests/test_page_authoring.py tests/test_page_registry_contract.py +uv run pytest --basetemp .pytest_tmp tests/test_figure_builders.py tests/test_export_payload.py ``` ## Review Checklist @@ -407,16 +407,16 @@ uv run --with pytest pytest --basetemp .pytest_tmp tests/test_figure_builders.py - Required and optional data match the visible workflows. - Summary reads declare the columns they consume. - Selectors own options; sections list every selector dependency. -- Custom widgets are registered rather than inserted raw. +- Register custom widgets. Do not insert them directly. - Pure transforms do not depend on Panel state. -- Existing shared figures and tables are used before adding new renderers. +- Use existing shared figures and tables before you add renderers. - Missing data produces a standard diagnostic card. - Live and export behavior use the same declarations. - Catalogs and focused tests are current. ## Related Chapters -- [31 - Dashboard Pages](31-dashboard-pages.md) +- [31 - Dashboard Page Contract](31-dashboard-pages.md) - [32 - Figures And Widgets](32-figures-and-widgets.md) - [33 - Dashboard Page Recipes](33-dashboard-page-recipes.md) - [34 - HTML Export](34-html-export.md) diff --git a/wiki/46-testing.md b/wiki/46-testing.md index 2a89ba7..e2f5245 100644 --- a/wiki/46-testing.md +++ b/wiki/46-testing.md @@ -1,40 +1,70 @@ # 46 - Testing -The default command runs every test, including the exhaustive offline HTML -export checks: +## Install Test Dependencies + +The project supports Python 3.10 or later. The checked-in GitHub Actions job +uses Python 3.12 on Windows. Install the locked runtime and `dev` dependency +group before running its commands: + +```powershell +uv sync --locked --group dev +``` + +The `dev` group contains `pytest` and `ruff`. The separate `notebooks` group is +not required for the test suite. Do not add an ad hoc `--with` dependency when +the locked development environment is already installed. + +The default command runs all tests, including the complete offline HTML export +checks: ```powershell uv run pytest --basetemp .pytest_tmp ``` -For a faster development loop, skip tests marked `full_export`: +For a faster development test, omit tests marked `full_export`: ```powershell uv run pytest --basetemp .pytest_tmp -m "not full_export" ``` -Run the exhaustive export boundary on its own before merging export, page, -plotting, or summary changes: +Run the complete export tests before merging export, page, plotting, or summary +changes: ```powershell uv run pytest --basetemp .pytest_tmp -m full_export ``` -The repository uses pytest's built-in `tmp_path` fixture with the workspace-local -`--basetemp` above. Tests must not create persistent UUID-named directories at -the repository root. +The repository uses pytest's built-in `tmp_path` fixture with the +workspace-local `--basetemp` value above. Tests must not create persistent +UUID-named directories in the repository root. + +Run this correctness check before pushing changes: + +```powershell +uv run ruff check . +``` + +## Match Continuous Integration -Run the configured correctness lint before pushing: +`.github/workflows/tests.yml` runs on pushes to `main`, pull requests, and +manual dispatch. It checks out the repository, selects Python 3.12, installs +`uv`, syncs the locked `dev` group, runs Ruff, and then runs the full pytest +command. To reproduce the CI work locally: ```powershell +uv sync --locked --group dev uv run ruff check . +uv run pytest --basetemp .pytest_tmp ``` -`full_export` is reserved for behavior that requires every default dashboard -page and all dashboard states. Tests of writing, validation, individual pages, -selectors, and diagnostics should configure the smallest page and state set -that exercises their contract. This keeps those tests focused without reducing -the end-to-end coverage provided by the full-export tests. +CI currently has no separate docs-link job. Documentation changes that add or +rename pages must therefore run the repository's catalog generator and a local +Markdown link/anchor check in addition to the relevant Python tests. + +Use `full_export` only for behavior that requires all default dashboard pages +and dashboard states. For writes, validation, pages, selectors, and diagnostics, +configure the smallest relevant page and state set. The full-export tests still +provide complete workflow coverage. ## Which Suite To Run @@ -45,10 +75,10 @@ the end-to-end coverage provided by the full-export tests. | Export serializer, payload, runtime, or state behavior | Focused export tests | Fast suite plus `-m full_export` | | Documentation only | Link/catalog checks and focused documentation tests | Fast suite if CI does not provide a docs-only path | -The full-export tests are slow because they render every default page and -dashboard state into a representative standalone HTML document. The shared -fixture builds that document once per test session, so running the marked group -together avoids repeating the expensive render. +The full-export tests take longer because they render every default page and +dashboard state in one representative standalone HTML document. A shared +fixture builds the document once per test session, so run the marked group +together to avoid repeated renders. ## Focused Commands @@ -58,7 +88,58 @@ uv run pytest --basetemp .pytest_tmp tests/test_figure_builders.py uv run pytest --basetemp .pytest_tmp tests/test_export_serializer.py tests/test_export_payload.py ``` -Use [Developer Workflows](40-developer-workflows.md) to choose tests by +## Generated Runtime And Catalog Checks + +The browser export runtime has readable source under +`dashboard/export/js_runtime/` and a generated artifact at +`dashboard/export/assets/export_runtime.js`. After changing the readable +source, rebuild it and run its contract test: + +```powershell +uv run python dashboard/export/build_export_runtime.py +uv run pytest --basetemp .pytest_tmp tests/test_export_runtime_build.py tests/test_export_runtime_contract.py +``` + +The build test checks that the tracked generated asset matches the runtime +source. If it fails, rebuild the asset and commit the generated change. Do not +edit the asset directly. + +After changing summary declarations, schemas, summary catalog metadata, page +definitions, groups, or page data requirements, regenerate and test the wiki +catalogs and standalone processor output reference. Changes to prepared-output +or sidecar contracts also require regeneration: + +```powershell +uv run python scripts/generate_wiki_catalogs.py +uv run pytest --basetemp .pytest_tmp tests/test_summary_declarations.py tests/test_page_registry_contract.py +``` + +Review the generated diff. A catalog change should follow the code contract +that caused it; do not hand-edit generated blocks. + +## Test Data Conventions + +- Build the smallest Polars frames that exercise the contract. Include only + the IDs, source fields, and `finalweight` needed by the behavior. +- Use `tmp_path` for files and `tmp_path_factory` only for intentionally shared + session fixtures. Never create persistent test output in the repository + root. +- Write CSV, Parquet, OMX, or YAML inputs inside that temporary directory. +- Give IDs explicit compatible types when a join or cache schema is under test. +- Test the complete case plus the relevant empty, unavailable, failed, orphan, + or partial-run case. +- Keep pure summary/transform assertions independent of Panel. Add lifecycle or + export tests only for behavior at those boundaries. +- Reuse the session-scoped `representative_full_export_html` fixture when a + test genuinely needs the full default export. Do not rebuild it in each + test. + +When a regression needs a large real data set, reduce it to a small synthetic +fixture or store only a reviewed stable fixture under `tests`. Tests must not +depend on a developer's model-output directory, network service, or existing +cache root. + +Use [Developer Workflows](40-developer-workflows.md) to select tests for a subsystem. ## Related Chapters diff --git a/wiki/90-troubleshooting.md b/wiki/90-troubleshooting.md index 6fdab0a..839c6a5 100644 --- a/wiki/90-troubleshooting.md +++ b/wiki/90-troubleshooting.md @@ -1,43 +1,98 @@ # 90 - Troubleshooting -Use this chapter when a run, cache, page, or export is not behaving as expected. +Use this chapter when a run, cache, page, or export does not behave as expected. -## Fast Triage +## Initial checks -1. Confirm the config path you ran. -2. Check the selected pipeline steps and dashboard mode in logs. -3. Check whether the issue appears in prepare, summarize, dashboard, or export. -4. Inspect `//manifest.json` for the affected run (see the - [cache layout](12-running-workflows.md#artifact-and-cache-paths)). -5. If cache reuse is suspect, temporarily set `pipeline.overwrite: true` for - the affected configured steps. +1. Make sure you used the correct configuration path. +2. Find the selected pipeline steps and dashboard mode in the log. +3. Identify whether the problem occurs in prepare, summarize, dashboard, or export. +4. Inspect `//manifest.json` for summary state and + `//prepared_tables/manifest.json` for final prepared/skimjoin + state (see the [cache layout](12-running-workflows.md#artifact-and-cache-paths)). +5. Use `--explain-cache` to examine reuse and rebuild decisions. If you + must rebuild, list only the relevant step in `pipeline.refresh`. ## Symptoms | Symptom | Likely causes | First checks | |---|---|---| | Run missing from dashboard | Missing summary cache, label mismatch, config run omitted | `runs`, cache directories, log run keys | -| Summary cache rebuilds unexpectedly | Input fingerprint changed, config digest changed, summary contract changed | the run manifest's summary-cache entries | +| Summary cache rebuilds unexpectedly | Input fingerprint changed, upstream prepared identity changed, summary config changed, summary declaration changed | run-level summary manifest and `--explain-cache` | | Page says data unavailable | Required summary missing, optional raw input absent, prepared column missing | page catalog and summary catalog | | Counts look wrong | Weighting mode, sample rate, explicit weight columns | `summarize.weighting_modes`, prepared `finalweight` | -| Geography options missing | Geography disabled, land-use columns missing, aggregation config wrong | `zones`, `summarize.geography` | +| Geography options missing | Geography disabled, zone columns missing, lookup config wrong | [Geography](27-geography.md) and `summarize.geography` | +| Segmented series missing | Segment step disabled, source values do not match, dashboard visibility hides them | [Segmentation](24-segmentation.md), `pipeline.steps`, and `segment.dashboard` | | Skim pages empty | Skimjoin disabled, no skim outputs, missing lookup rules | skimjoin manifest and reports | | Export differs from live | Widget/section not registered, selector values omitted, unsupported node | page selector/section registrations | | Dashboard-only run fails | Summary cache missing or prepared-data page needs prepared cache | `pipeline.steps`, page prepared-data mode | +## Configuration And Startup Problems + +| Symptom | Cause to distinguish | Action | +|---|---|---| +| Unknown top-level or section key | Typo, removed field, or field at the wrong nesting level | Use the replacement in the error and compare the field with chapter 13. Do not move it until you confirm its owning section. | +| Path exists in YAML but file is not found | Relative path uses a different base than expected | Use the path-resolution table in chapter 13. Raw `files` start from each run directory; most other main-config paths start from the config directory. | +| `skimjoin` or `segment` config appears ignored | The configuration block does not enable its logical step | Add the step and its prerequisite to `pipeline.steps`. Use canonical step order. | +| Dashboard starts instead of exporting | `dashboard_mode` is live or a CLI override selected live mode | Set `pipeline.dashboard_mode: export`, or use `--dashboard --export-html`. | +| `dashboard_mode: host` does not publish | Core host mode is a placeholder | Use the explicit Panel hosting script in chapter 43. | +| Port is already in use | Another server owns the selected port | Stop that process or use `--port `. | +| Another visualizer process is already using the output root | A live process holds that root's cross-process lock; the persistent `.activitysim_visualizer.lock` file alone is normal | Wait for or stop the owning process, or configure a different `root`. Deleting the file does not release a lock held by a live process. | + +Configuration validation is strict at documented typed boundaries but some +extension/nested mappings are intentionally free-form. If a nested setting has +no effect and no error, confirm its exact spelling and add a focused config +load test rather than assuming it was applied. + +## Raw Input And Prepare Problems + +| Symptom | Check | Interpretation | +|---|---|---| +| Raw table is unavailable | Effective `files` plus `runs[*].file_map`, run directory, extension, fallback path | A stem tries Parquet before CSV. An explicit extension tries only that file. | +| Entire run is skipped | Availability of households, persons, tours, and trips | The run is skipped when none of these four core tables is usable. One missing core table instead causes partial summary coverage. | +| Prepared column is missing | Source alias, owning raw table, enrichment prerequisites, prepared manifest | Prepare only materializes a canonical field when it finds the source needed for that field. | +| CSV reads with an unexpected type | Mixed values or inference | Prefer Parquet for controlled schemas or normalize the raw column before prepare. Prepared finalization casts only known canonical fields. | +| Relationship warning reports orphans | Source/target key values and types | Direct summaries can count orphan rows while joined summaries can drop them. Fix the relationship instead of comparing those totals as equivalent. | +| `prepared_table_map` lacks derived fields | External process supplied raw-like rather than canonical tables | That input bypasses all prepare enrichment, weighting, geography, and skimjoin. Materialize the prepared contract upstream. | +| Optional table has zero columns after cache load | Stored `empty`, `unavailable`, or `failed` state | Read `table_states` and `table_diagnostics`; the cache loader converted its sentinel back to an empty frame. | + +Use [14 - Input Data Contract](14-input-data-contract.md) to identify the +expected keys and relationship checks. Use chapter 26 to work backward from one +unavailable summary to its declared prerequisite columns and documented +builder-level conditions. + +## Weighting, Totals, And Units + +If counts differ from expectation, inspect the prepared `finalweight` values on +the table that the summary actually aggregates. Do not infer trip weights from +household weights without checking propagation. + +| Symptom | Likely reason | Check | +|---|---|---| +| Weighted equals unweighted | No source weight/sample rate, all source weights are one, or a mapped external summary was copied to both modes | Prepared `finalweight`, run weight fields, `summary_table_map` behavior | +| Household totals are not sample-expanded | Any explicit household, person, or trip run weight field disables automatic household sample-rate expansion | `hh_weight_col`, `person_weight_col`, `trip_weight_col`, `columns.sample_rate` | +| Person-day totals use an unexpected weight | `day_weight_col` defaults to `day_weight`; non-null day values take precedence over inherited person or household weights | Inspect the day source and set `day_weight_col: null` when it is only a placeholder. | +| Some rows disappear from weighted totals | Null source weights or builder filters | Null/nonfinite weight counts and summary requirements | +| Negative or infinite result | Negative weights, zero sample rate, or zero weighted denominator | Validate finite nonnegative weights and positive sample rates upstream | +| Distance/time/cost differs by a fixed factor | Runs or skims use different units | Prepared source columns and skim documentation; the visualizer does not convert units | +| Percent chart does not sum to 100 | Fixed count/rate chart, missing categories, separate traces, or a builder-specific denominator | Axis title, `value_mode` used by the page, and calculation note | + +The first run is the comparison base where a page reports differences. Confirm +run order before treating a changed difference as a processor regression. + ## Cache Problems -For a reproducible full rebuild, configure the steps and overwrite policy: +To make a repeatable full rebuild, configure the steps and refresh policy: ```yaml pipeline: steps: [prepare, summarize, dashboard] dashboard_mode: live - overwrite: true + refresh: all ``` -Return `overwrite` to `false` after the rebuild. Developers can use targeted -one-off refresh flags while diagnosing a specific cache layer: +Set `refresh` to `[]` after the rebuild. During a diagnostic run, developers +can use a refresh flag for one cache layer: ```bash uv run activitysim-viz --config local_config.yaml --refresh-prepared-cache @@ -45,40 +100,46 @@ uv run activitysim-viz --config local_config.yaml --refresh-summary-cache uv run activitysim-viz --config local_config.yaml --refresh-caches ``` -If only dashboard presentation changed, a refresh usually should not be needed. -If raw inputs or prepare config changed, refresh both caches. +If only dashboard presentation changed, a refresh is usually unnecessary. The +system automatically checks raw-file, skim-file, and relevant configuration +identities. Use a manual refresh only to override a valid cache decision, and +use `pipeline.refresh` for repeatable runs. A prepare refresh +invalidates skimjoin and summary output. A skimjoin refresh keeps +`base_prepared_tables`. A summary refresh keeps final prepared data. ## Missing Page Data -Find the page in [31 - Dashboard Pages](31-dashboard-pages.md) and check: +Find the page in +[31 - Dashboard Page Contract](31-dashboard-pages.md) and check: - required summary IDs - required prepared tables - prepared-data mode -- whether the page is enabled in live/export config +- whether the live or export configuration enables the page -Then find each summary in [24 - Summary Catalog](24-summary-catalog.md) and -check the required input tables/columns. +Then find each summary in [26 - Summary Catalog](26-summary-catalog.md). Check +the required input tables and columns. ### Worked Triage: A Page Says Data Is Unavailable -Suppose Trip Mode opens but shows the standard unavailable card: +If Trip Mode shows the standard unavailable card, follow these steps: 1. Find `trip_mode` in chapter 31. It requires `trip_mode_by_tour_purpose_and_tour_mode`. -2. Find that ID in chapter 24. Note its required prepared table and columns. -3. Open `//manifest.json` and inspect the summary entry. If the - summary is `unavailable`, read its recorded reason before rebuilding - anything. -4. If a required prepared column is missing, inspect the same manifest's - prepared-cache entry and the canonical column settings in `columns`. +2. Find that ID in chapter 26. Note its required prepared table and columns. +3. Open `//manifest.json` and examine the summary entry. If the + summary is `unavailable`, read its recorded reason before a rebuild. +4. If a required prepared column is missing, inspect + `//prepared_tables/manifest.json`, the table schema, and the + canonical column settings in `columns`. 5. If the contract recently changed, rebuild the configured summarize step - with `pipeline.overwrite: true`. -6. If the summary is present and valid, confirm the page's `columns=` request - matches the cached schema and that the selected weighting mode exists. + with `pipeline.refresh: [summarize]`. +6. If the summary is valid, make sure that the page's `columns=` request agrees + with the cached schema. +7. Make sure the selected weighting mode exists. -This sequence moves backward through the declared contracts. It avoids trying -random cache refreshes when the real issue is an input or schema mismatch. +This sequence works backward through the declared contracts and avoids +unnecessary cache refreshes when the problem is an input or schema mismatch. ## Skimjoin Problems @@ -88,40 +149,118 @@ Check the skimjoin reports: - `missing_lookup_report` - `fallback_lookup_report` - `skipped_rule_report` -- `tour_aggregation_summary` - `failure_report` -Common fixes: +Common corrections: - correct skim file globs - correct `network_los_file` - align `activitysim` source columns with prepared tables - add missing dimension values -- change missing matrix/OD policy only after confirming the missing data is - expected +- resolve unexpected missing matrix/OD rows at the source; current annotation + records them identically for every configured missing policy (see + [Missing Policy Execution](23-skimjoin-config-reference.md#missing-policy-execution)) + +Also inspect `config_normalized.yaml` to verify the effective rules and paths. +For CSV skims, confirm whether the file was inventoried as a keyed table or an +OD table and use the generated `__` matrix name. For +OMX/HDF5, qualify duplicate matrix names with `filename::matrix` and verify the +selected zone mapping. + +With `failure_policy: record`, a failure is expected to leave the original +prepared trip/tour tables in place and record a `failure_report`. With `error`, +the same failure stops the workflow. Do not interpret “run continued” as proof +that skim values were applied; read `skimjoin_status` and +`skimjoin_applied_outputs`. + +## Segmentation Problems + +| Symptom | Check | +|---|---| +| Definition or ID rejected | Lowercase path-safe pattern and no leading/trailing punctuation. | +| CSV-backed segment is empty | Quote numeric-looking `segments[*].values`; CSV segment values are stored as strings. | +| Prepared-column segment is empty | Match the prepared column's value and type exactly. | +| Household/person totals look too broad | The anchor may be trip/tour based; relationship expansion retains related parents and children. | +| Segment totals exceed full total | Values overlap with `allow_overlapping: true`, or the counted population differs from the anchor. | +| Mapped external summary is identical in every segment | `summary_table_map` is overlaid unchanged because aggregated rows cannot be re-segmented. | +| Only one segment changed but many tables rebuilt | Read per-unit/per-summary digests and `--explain-cache`; a shared summary/config change can invalidate all units. | + +The run manifest's `segmentation_types` list is the final record of source, +values, stored paths, states, and diagnostics. If a configured segment is not +there, review `on_empty_segment: skip` and whether summarize completed a cache +write. + +## Geography Problems + +Distinguish preparation from presentation: + +1. Confirm `summarize.geography.enabled: true`. +2. Confirm the named aggregation and source zone system in the loaded config. +3. Inspect the role-specific prepared column, such as + `home_geo__district` or `destination_geo__district`. +4. Confirm non-null mapped values and lookup coverage. +5. Confirm the target summary supports that role in chapter 27. +6. Confirm the summary has rows for the geography type. +7. Only then inspect the dashboard selector and + `dashboard.enable_maz_geographies`. + +A valid named mapping does not add geography to every summary. Parking Location +currently uses its base parking zone, and MAZ presentation can be hidden even +when MAZ summary rows exist. ## Export Problems -If live mode works but export does not: +If live mode works but export fails: + +1. Make sure export page selection includes the page. +2. Make sure standard selection lists use `self.select(...)`. +3. Make sure custom widgets use `self.selector(...)`. +4. Make sure `self.section(...)` registers the relevant content. +5. Check browser console errors. +6. Inspect the adjacent `.diagnostics.json` sidecar. +7. Try `?debug_export=1`. + +Export cannot reproduce every Python callback; it can only switch among stored +states and registered selector variants. + +Use the diagnostics sidecar to separate three failure classes: + +| Evidence | Meaning | +|---|---| +| `render_state: skipped` with excluded runs | Data contract or availability problem before serialization. | +| Large `raw_state_count` or `size_analysis` peak | Selector enumeration made the payload large; export fewer values or disable that part. | +| Browser `ExportRuntimeError` | Payload/runtime schema, node, state, or rendering problem; note its error code. | + +If you changed the browser runtime, edit `dashboard/export/js_runtime/`, rebuild +the generated asset, and run runtime build/contract tests. Never patch the +generated asset as the source change. + +## Performance And Hosting Problems -1. Confirm the page is included in export page selection. -2. Confirm ordinary dropdowns use `self.select(...)` and custom widgets use - `self.selector(...)`. -3. Confirm affected content is registered with `self.section(...)`. -4. Check browser console errors. -5. Try `?debug_export=1`. +| Symptom | First action | +|---|---| +| First run is slow | Separate prepare, skimjoin, summarize, and dashboard timings; later valid runs should reuse caches. | +| Skimjoin uses more memory than expected | Confirm a CSV OD file starts with a recognized OD header pair and that rules reference only needed value columns. Integrated OD CSVs retain demanded pairs; keyed CSVs retain every key, and selected OMX/HDF5/H5 datasets load in full. | +| Export is very large or slow to open | Inspect `size_analysis.page_peaks` and `region_peaks`; reduce exported weighting/value/selector states. | +| Live server uses much more memory than export | Check enabled prepared-data pages and concurrent Panel sessions. Prepared runs can be loaded for live-only features. | +| Hosted page loads but controls disconnect | Verify reverse-proxy WebSocket upgrades and `--allow-websocket-origin`. | +| Hosted startup has no runs | Use an explicit config path and persistent compatible caches; fail deployment on missing required data. | +| Permission error during hosting | Use read-only caches for serve-only deployment; grant writes only if startup deliberately builds artifacts. | +| Posit Connect Cloud export differs from live mode | Test the local HTML export first; the hosted static file contains only export-supported pages, sections, and selector states. | -Export cannot reproduce arbitrary Python callbacks. It can only switch among -serialized states and registered selector variants. +For static HTML publishing, see +[17 - Publish An Export With Posit Connect Cloud](17-posit-connect-cloud.md). +For live-server deployment commands and requirements, see +[43 - Weighting And Hosting Extensions](43-weighting-hosting-extensions.md#worked-example-connect-a-hosting-script). -## Still Stuck +## Create a small test case -Create the smallest reproduction: +Reduce the problem to the smallest test case: 1. one run 2. one page or one summary 3. one weighting mode 4. fresh cache root -5. copied log excerpt and manifest diagnostics +5. A copy of the relevant log text and manifest diagnostics. -That usually makes the owning subsystem obvious. +This usually identifies the responsible subsystem. diff --git a/wiki/99-glossary.md b/wiki/99-glossary.md index aa2b97e..e8a0a42 100644 --- a/wiki/99-glossary.md +++ b/wiki/99-glossary.md @@ -3,36 +3,50 @@ | Term | Meaning | |---|---| | ActivitySim output | Raw model output tables such as households, persons, tours, trips, and land use. | +| Analysis unit | One full run or one related segment slice passed to the standard summary builders. | +| Availability state | Stored table or summary status: `available`, `empty`, `unavailable`, or `failed`. Dashboard selections add `missing` and `schema_mismatch` when inspecting a requested input. | +| Cache identity | Normalized input, configuration, upstream-manifest, and implementation information used to decide whether an artifact is reusable. It is recorded in manifests and is not the same as write time. | | Dashboard page | One registered visualizer page with a stable `page_id`. | +| Dashboard page group | Registered navigation container with a stable group ID, ordered child pages, a default child, and default-enabled behavior. | | Dashboard state | Shared visualizer state such as weighting mode, value mode, segmentation, and loaded runs. | | Export | Standalone HTML dashboard output that does not require a Python server. | -| `file_map` | Per-run override for raw ActivitySim output filenames. | +| Extension | Trusted importable code or external data that adds weighting behavior, summaries, prepared fields/tables, pages, or hosting integration through a documented boundary. | +| Failure policy | Config choice that either records a stage/builder failure as diagnostics and continues (`record`) or raises it and stops (`error`). Not every subsystem exposes both choices. | +| `file_map` | Run override for raw ActivitySim output file names. | | `finalweight` | Canonical prepared weight column aggregated by summary builders. | -| Live mode | Python-backed Panel dashboard served locally. | +| Geography aggregation | Named mapping from MAZ or TAZ IDs to a custom spatial system, such as a district or subregion. | +| Live mode | Local Panel dashboard that uses a Python server. | | MAZ | Micro analysis zone. | | OMX | Open Matrix file format commonly used for skims. | | Output Processor | Prepare, skimjoin, segmentation, and summarize workflows. | | Output Visualizer | Live dashboard and HTML export workflows. | +| Page feature | Page-local object that namespaces a related set of selectors and sections under one feature ID. It is composition within a page, not a discoverable page. | +| Page section | Registered stable page region with a section ID, declared selector dependencies, renderer, and export/data behavior. Selector changes mark only dependent sections stale. | | Prepared cache | Per-run cache of canonical prepared tables. | +| Prepared-data mode | Page declaration value `none`, `optional`, or `required` that controls whether live prepared caches are requested and whether the page's feature is expected to need them. | | Prepared table | Normalized table used by summaries and prepared-data pages. | | `prepared_table_map` | Config mapping that supplies canonical prepared tables directly and skips raw prepare. | | Run | One ActivitySim scenario/output set shown in the dashboard. | -| Run key | Cache-directory identifier made by slugifying a run label, such as `Build 2035` to `build-2035`; duplicate normalized labels receive order-dependent `-1`, `-2`, and later suffixes. | -| Segment | Configured slice of prepared data summarized separately. | +| `RunData` | Processor dataclass containing one run's canonical prepared tables, optional skim state, table availability, prepare diagnostics, and skimjoin artifacts. Summary builders receive it. | +| Run key | Cache-directory identifier made from a run label. For example, `Build 2035` becomes `build-2035`. Duplicate normalized labels get order-dependent suffixes such as `-1` and `-2`. | +| `RunTables` | Dashboard multi-run table value that keeps usable `(label, DataFrame)` pairs together with exclusions and source IDs while applying fluent Polars operations. | +| Segment | Configured part of prepared data that the workflow summarizes separately. | +| Segmentation type | Named segment definition containing one source and one or more segment IDs. | | Selector | Registered page-local widget that can refresh sections and participate in export. | -| Skim | Matrix or lookup data used to attach level-of-service values to trips/tours. | +| Skim | Matrix or lookup data that supplies level-of-service values to trips or tours. | | Skimjoin | Optional processor step that joins skim-derived values to prepared trips and tours. | | Summary builder | Function that converts `RunData` and `Config` into one summary `DataFrame`. | | Summary cache | Per-run, per-weighting-mode CSV summary tables consumed by dashboard pages. | | Summary contract | Builder metadata defining output schema and required inputs. | +| `summary_table_map` | Config mapping from registered summary IDs to dashboard-ready CSV/Parquet files. Mapped tables can replace generated IDs but cannot be reweighted or segmented from aggregate rows. | | TAZ | Traffic analysis zone. | -| Weighting mode | Versioned registered transform that presents prepared `finalweight` values to summary builders and prepared-data pages under one cache/dashboard mode ID. | +| Weighting mode | Registered transform with a version. It supplies prepared `finalweight` values under one cache and dashboard mode ID. Summary builders and prepared-data pages use the values. | ## How The Terms Connect -For a run labeled `Build`, raw `final_trips.csv` is normalized into the -prepared `trips` table. A summary builder aggregates its canonical -`finalweight` column and writes a registered summary under the run key's -weighted and unweighted cache directories. A dashboard page declares that -summary ID, reads it through `self.data`, and lets registered selectors refresh -its sections. Export serializes those same declared page states into HTML. +For a run labeled `Build`, prepare converts raw `final_trips.csv` into the +prepared `trips` table. A summary builder aggregates the canonical `finalweight` +column and writes a registered summary to the run key's weighted and unweighted +cache directories. A dashboard page declares that summary ID and reads it +through `self.data`, while registered selectors refresh its sections. Export +turns the same declared page states into HTML. diff --git a/wiki/images/posit-publisher-config.png b/wiki/images/posit-publisher-config.png new file mode 100644 index 0000000..df5e2e9 Binary files /dev/null and b/wiki/images/posit-publisher-config.png differ diff --git a/wiki/images/publishing-workspace.png b/wiki/images/publishing-workspace.png new file mode 100644 index 0000000..a6dc82f Binary files /dev/null and b/wiki/images/publishing-workspace.png differ