From 2c91049d18ce25c9b53c1be902848aa6641e26cb Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 11 Apr 2026 18:21:39 +0000 Subject: [PATCH 1/7] Update CLAUDE.md with comprehensive codebase documentation Add project overview, detailed architecture map (directory structure, key classes, design patterns), test infrastructure guide, environment variables reference, CI/CD summary, and pre-commit hook listing to help AI assistants navigate and contribute to the codebase effectively. https://claude.ai/code/session_017qM7WyFgvqy5UQHEtEcHzY --- CLAUDE.md | 186 +++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 163 insertions(+), 23 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index e8a64b38c..f606206f4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,44 +2,184 @@ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +## Project Overview + +**dandi-cli** is the command-line client for the [DANDI Archive](https://dandiarchive.org), a platform for publishing, sharing, and processing neurophysiology data. It handles uploading, downloading, organizing, and validating neuroscience data files (primarily NWB and BIDS formats). + +- **Language**: Python 3.10+ +- **Build system**: setuptools with versioneer (git-based PEP 440 versioning) +- **Entry point**: `dandi` CLI command (`dandi/cli/command.py:main`) +- **pytest plugin**: Registered as `dandi` entry point (`dandi/pytest_plugin.py`) + ## Build/Test Commands -- Run tests with hatch: `hatch run test:run` -- Run tests with tox: `tox -e py3` or `python -m pytest dandi` if in a venv -- Tests which require an instance of the archive, would use a fixture to start on using docker-compose. -- Set env var `DANDI_TESTS_PULL_DOCKER_COMPOSE=""` (to empty value) to avoid `docker compose pull` to speed up repetitive runs -- Run single test with hatch: `hatch run test:run dandi/tests/test_file.py::test_function -v` -- Run single test with tox: `tox r -e py3 -- dandi/tests/test_file.py::test_function -v` -- Lint and type checking: `tox -e lint,typing` -- Install pre-commit hooks (if not installed as could be indicated by absence of - `.git/hooks/pre-commit`): `pre-commit install` + +- **Run tests with hatch**: `hatch run test:run` +- **Run tests with tox**: `tox -e py3` or `python -m pytest dandi` if in a venv +- **Run single test with hatch**: `hatch run test:run dandi/tests/test_file.py::test_function -v` +- **Run single test with tox**: `tox r -e py3 -- dandi/tests/test_file.py::test_function -v` +- **Lint and type checking**: `tox -e lint,typing` +- **Lint only**: `tox -e lint` (runs codespell + flake8) +- **Type checking only**: `tox -e typing` (runs mypy) +- **Build docs**: `tox -e docs` +- **Install pre-commit hooks**: `pre-commit install` (check for `.git/hooks/pre-commit`) +- **Integration tests**: Tests requiring a local archive instance use docker-compose fixtures +- **Speed up docker tests**: Set `DANDI_TESTS_PULL_DOCKER_COMPOSE=""` to skip `docker compose pull` + +## Codebase Architecture + +### Directory Structure + +``` +dandi/ + cli/ # Click-based CLI commands + command.py # Main entry point, Click group with DYMGroup + base.py # Shared CLI utilities, decorators, custom param types + cmd_*.py # Individual commands (download, upload, organize, etc.) + formatter.py # Output formatters (JSON, YAML, JSONL, PYOUT) + files/ # File type abstractions + bases.py # DandiFile hierarchy (LocalAsset, NWBAsset, etc.) + bids.py # BIDS-specific file types (NWBBIDSAsset, etc.) + zarr.py # Zarr archive handling (ZarrAsset, LocalZarrEntry) + metadata/ # Metadata extraction + core.py # Entry points for metadata extraction + nwb.py # NWB-specific metadata extraction via PyNWB + util.py # get_metadata(), field extraction, caching + validate/ # Validation engine + _types.py # ValidationResult, Severity, Scope, Standard enums + _core.py # validate() generator, validate_bids() + _io.py # JSON Lines I/O for validation results + support/ # Shared utilities + digests.py # Checksum/digest computation (DANDI eTag, Zarr) + pyout.py # Progress display with pyout (LogSafeTabular) + iterators.py # IteratorWithAggregation for progress tracking + threaded_walk.py # Parallel directory traversal + tests/ # Test suite + fixtures.py # Core test fixtures (NWB files, local API, dandisets) + skip.py # Conditional skip helpers + data/ # Test data files + consts.py # Constants: metadata fields, known instances, layout fields + dandiapi.py # API client (RESTFullAPIClient, DandiAPIClient) + dandiarchive.py # URL parsing (ParsedDandiURL, parse_dandi_url()) + dandiset.py # Local dandiset representation (dandiset.yaml) + download.py # Download engine with resume/retry support + upload.py # Upload engine with validation + organize.py # File organization by NWB metadata + delete.py # Asset/dandiset deletion + move.py # Asset move/rename (local + remote) + exceptions.py # Custom exceptions (all end with "Error") + misctypes.py # Shared types: Digest, BasePath + pynwb_utils.py # PyNWB helpers for reading/creating NWB files + utils.py # General utilities +``` + +### Key Design Patterns + +- **CLI delegation**: CLI commands (`cmd_*.py`) are thin wrappers that delegate to core modules (e.g., `cmd_upload.py` calls `upload.upload()`) +- **File type hierarchy**: `DandiFile` abstract base with factory function `dandi_file()` and discovery via `find_dandi_files()` +- **Enum-based configuration**: Operations use enums for modes (e.g., `DownloadExisting`, `FileOperationMode`, `UploadValidation`) +- **Generator-based processing**: Validation, download, and file finding all use generators +- **Context managers**: API clients (`DandiAPIClient`) and URL navigation use context managers +- **Retry logic**: HTTP operations use `tenacity` for exponential backoff retries +- **Lazy imports**: Heavy modules (pynwb, h5py) are imported at point of use, not at module level + +### Key Classes + +- `DandiAPIClient` (`dandiapi.py`): High-level API client with authentication (keyring), pagination, asset management +- `RESTFullAPIClient` (`dandiapi.py`): Base HTTP client with session management and retry logic +- `ParsedDandiURL` (`dandiarchive.py`): Abstract base for URL parsing with subclasses `DandisetURL`, `SingleAssetURL`, `AssetItemURL`, `AssetDirURL` +- `DandiFile` (`files/bases.py`): Abstract base for all file types; subclasses include `NWBAsset`, `ZarrAsset`, `GenericAsset`, `VideoAsset` +- `ValidationResult` (`validate/_types.py`): Pydantic model with origin, severity, scope, message, paths +- `Dandiset` (`dandiset.py`): Local dandiset representation wrapping `dandiset.yaml` +- `DandiInstance` (`consts.py`): Frozen dataclass for known archive instances (dandi, dandi-sandbox, linc, ember-dandi, etc.) ## Committing -- Due to use of `pre-commit` with black and other commands which auto-fix, if changes - were reported to be done, just rerun commit again 2nd time, and if only then if still - does not commit analyze output more + +- Due to use of `pre-commit` with black and other auto-fixers, if changes were reported, just rerun commit a 2nd time. Only then if it still does not commit, analyze output further. ## Test Markers + - When adding AI-generated tests, mark them with `@pytest.mark.ai_generated` - Any new pytest markers must be registered in `pytest_configure` function of `dandi/pytest_plugin.py` +- Existing markers: `integration`, `obolibrary`, `flaky`, `ai_generated` + +## Test Infrastructure + +### Key Fixtures (`dandi/tests/fixtures.py`) + +- `simple1_nwb_metadata()` / `simple1_nwb()`: Session-scoped sample NWB file +- `local_dandi_api`: Docker-based local DANDI Archive instance for integration tests +- `new_dandiset()`: Creates a fresh dandiset on the test instance +- `publish_dandiset()`: Publishes a dandiset version +- `capture_all_logs`: Autouse fixture setting DEBUG level for `dandi` logger + +### Test Organization + +- Tests mirror the module structure: `test_download.py`, `test_upload.py`, etc. +- Integration tests requiring docker use the `local_dandi_api` fixture +- `--dandi-api` pytest flag filters to only integration tests +- `--scheduled` flag enables scheduled-only test configuration +- VCR (vcrpy) is used to record/replay HTTP interactions; disable with `DANDI_TESTS_NO_VCR` + +### pytest Configuration (`tox.ini [pytest]`) + +- Default timeout: 300 seconds per test +- `--tb=short --durations=10` by default +- `filterwarnings` set to `error` with specific ignores for known third-party warnings ## Code Style -- Code is formatted with Black (line length 100) -- Imports sorted with isort (profile="black") -- Type annotations required for new code -- Use PEP 440 for versioning -- Class names: CamelCase; functions/variables: snake_case -- Exception names end with "Error" (e.g., `ValidateError`) -- Docstrings in NumPy style for public APIs -- Prefer specific exceptions over generic ones -- For CLI, use click library patterns -- Imports organized: stdlib, third-party, local (alphabetical within groups) + +- **Formatter**: Black (line length 100) +- **Import sorting**: isort (profile="black", force_sort_within_sections, reverse_relative) +- **Linting**: flake8 (max-line-length=100, ignore E203/W503) +- **Spell checking**: codespell +- **Type checking**: mypy with pydantic plugin, strict settings +- **Type annotations**: Required for new code +- **Naming**: CamelCase for classes, snake_case for functions/variables +- **Exceptions**: Names must end with "Error" (e.g., `UploadError`, `NotFoundError`) +- **Docstrings**: NumPy style for public APIs +- **Dataclass field docs**: Use `#:` comments above the field (Sphinx autodoc format) +- **Imports**: Organized as stdlib, third-party, local (alphabetical within groups) +- **CLI**: Uses click library patterns with `DYMGroup` (did-you-mean suggestions) +- **Excluded from formatting**: `_version.py`, `due.py`, `versioneer.py` + +## Pre-commit Hooks + +The following hooks run on commit (`.pre-commit-config.yaml`): +1. trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files +2. black (code formatting) +3. isort (import sorting) +4. codespell (spell checking) +5. flake8 (linting) + +## Environment Variables + +- `DANDI_DEVEL`: Enables hidden CLI options (e.g., explicit instance selection) +- `DANDI_LOG_LEVEL`: Log level (default INFO, use int like `10` for DEBUG) +- `DANDI_CACHE`: Persistent cache control (`clear` or `ignore`) +- `DANDI_INSTANCEHOST`: Host for local archive instance (default `localhost`) +- `{INSTANCE_NAME}_API_KEY`: API key per instance (e.g., `DANDI_API_KEY`, `DANDI_SANDBOX_API_KEY`) +- `DANDI_TESTS_PERSIST_DOCKER_COMPOSE`: Reuse Docker containers across test runs +- `DANDI_TESTS_PULL_DOCKER_COMPOSE`: Set to empty/`0` to skip pulling Docker images +- `DANDI_TESTS_NO_VCR`: Disable VCR HTTP replay during tests +- `DANDI_PAGINATION_DISABLE_FALLBACK`: Disable fallback to sequential pagination (set in test envs) + +## CI/CD + +- **Tests** (`run-tests.yml`): Matrix of Python 3.10-3.13 across Ubuntu, macOS (M1 + Intel), Windows +- **Lint** (`lint.yml`): codespell + flake8 +- **Typing** (`typing.yml`): mypy +- **Docs** (`docs.yml`): Sphinx build +- **Release** (`release.yml`): Automated via `auto` tool - PR labels (`major`, `minor`, `patch`, `internal`, etc.) drive changelog and version bumps; tagged releases trigger PyPI upload ## Documentation + - Keep docstrings updated when changing function signatures - CLI help text should be clear and include examples where appropriate +- Dataclass fields: document with `#:` comments above the field, not docstrings below ## Issue Tracking with git-bug -This project has GitHub issues synced locally via git-bug. Use these commands + +This project has GitHub issues synced locally via git-bug. Use these commands to get issue context without needing GitHub API access: - `git bug ls status:open` - list open issues - `git bug show ` - show issue details and comments From 3cd422524abba7e15f7884d3677958f66b4d894b Mon Sep 17 00:00:00 2001 From: "GitMate for @yarikoptic" <41385986+yarikoptic-gitmate@users.noreply.github.com> Date: Thu, 6 Aug 2026 13:20:31 -0400 Subject: [PATCH 2/7] Move developer conventions from CLAUDE.md into CONTRIBUTING.md CLAUDE.md was duplicating content already covered (or better suited to) project-level documentation. Following the pattern used in datalad and heudiconv: - Create CONTRIBUTING.md with architecture, code style, test infrastructure, CI/CD, and PR-label reference. - Slim CLAUDE.md to a pointer at CONTRIBUTING.md + DEVELOPMENT.md, keeping only AI-specific notes (pre-commit re-run, git-bug commands). Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_017qM7WyFgvqy5UQHEtEcHzY --- CLAUDE.md | 193 +++++---------------------------------------- CONTRIBUTING.md | 203 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 223 insertions(+), 173 deletions(-) create mode 100644 CONTRIBUTING.md diff --git a/CLAUDE.md b/CLAUDE.md index f606206f4..d839925b5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,190 +2,37 @@ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. -## Project Overview +## MANDATORY: Read before making any code changes -**dandi-cli** is the command-line client for the [DANDI Archive](https://dandiarchive.org), a platform for publishing, sharing, and processing neurophysiology data. It handles uploading, downloading, organizing, and validating neuroscience data files (primarily NWB and BIDS formats). +You MUST read [`CONTRIBUTING.md`](./CONTRIBUTING.md) before making any code changes, commits, or +pull requests. It contains the authoritative project conventions including: -- **Language**: Python 3.10+ -- **Build system**: setuptools with versioneer (git-based PEP 440 versioning) -- **Entry point**: `dandi` CLI command (`dandi/cli/command.py:main`) -- **pytest plugin**: Registered as `dandi` entry point (`dandi/pytest_plugin.py`) +- Build/test commands and CI/CD overview +- Codebase architecture, directory layout, key classes, and design patterns +- Code style rules (formatting, imports, type annotations, docstrings) +- Testing requirements, including the **mandatory `@pytest.mark.ai_generated` marker on any test + written with AI assistance** +- PR labeling and release workflow (intuit/auto) -## Build/Test Commands +Extended documentation (environment setup, environment variables, release procedures, git-bug) +is in [`DEVELOPMENT.md`](./DEVELOPMENT.md). -- **Run tests with hatch**: `hatch run test:run` -- **Run tests with tox**: `tox -e py3` or `python -m pytest dandi` if in a venv -- **Run single test with hatch**: `hatch run test:run dandi/tests/test_file.py::test_function -v` -- **Run single test with tox**: `tox r -e py3 -- dandi/tests/test_file.py::test_function -v` -- **Lint and type checking**: `tox -e lint,typing` -- **Lint only**: `tox -e lint` (runs codespell + flake8) -- **Type checking only**: `tox -e typing` (runs mypy) -- **Build docs**: `tox -e docs` -- **Install pre-commit hooks**: `pre-commit install` (check for `.git/hooks/pre-commit`) -- **Integration tests**: Tests requiring a local archive instance use docker-compose fixtures -- **Speed up docker tests**: Set `DANDI_TESTS_PULL_DOCKER_COMPOSE=""` to skip `docker compose pull` - -## Codebase Architecture - -### Directory Structure - -``` -dandi/ - cli/ # Click-based CLI commands - command.py # Main entry point, Click group with DYMGroup - base.py # Shared CLI utilities, decorators, custom param types - cmd_*.py # Individual commands (download, upload, organize, etc.) - formatter.py # Output formatters (JSON, YAML, JSONL, PYOUT) - files/ # File type abstractions - bases.py # DandiFile hierarchy (LocalAsset, NWBAsset, etc.) - bids.py # BIDS-specific file types (NWBBIDSAsset, etc.) - zarr.py # Zarr archive handling (ZarrAsset, LocalZarrEntry) - metadata/ # Metadata extraction - core.py # Entry points for metadata extraction - nwb.py # NWB-specific metadata extraction via PyNWB - util.py # get_metadata(), field extraction, caching - validate/ # Validation engine - _types.py # ValidationResult, Severity, Scope, Standard enums - _core.py # validate() generator, validate_bids() - _io.py # JSON Lines I/O for validation results - support/ # Shared utilities - digests.py # Checksum/digest computation (DANDI eTag, Zarr) - pyout.py # Progress display with pyout (LogSafeTabular) - iterators.py # IteratorWithAggregation for progress tracking - threaded_walk.py # Parallel directory traversal - tests/ # Test suite - fixtures.py # Core test fixtures (NWB files, local API, dandisets) - skip.py # Conditional skip helpers - data/ # Test data files - consts.py # Constants: metadata fields, known instances, layout fields - dandiapi.py # API client (RESTFullAPIClient, DandiAPIClient) - dandiarchive.py # URL parsing (ParsedDandiURL, parse_dandi_url()) - dandiset.py # Local dandiset representation (dandiset.yaml) - download.py # Download engine with resume/retry support - upload.py # Upload engine with validation - organize.py # File organization by NWB metadata - delete.py # Asset/dandiset deletion - move.py # Asset move/rename (local + remote) - exceptions.py # Custom exceptions (all end with "Error") - misctypes.py # Shared types: Digest, BasePath - pynwb_utils.py # PyNWB helpers for reading/creating NWB files - utils.py # General utilities -``` - -### Key Design Patterns - -- **CLI delegation**: CLI commands (`cmd_*.py`) are thin wrappers that delegate to core modules (e.g., `cmd_upload.py` calls `upload.upload()`) -- **File type hierarchy**: `DandiFile` abstract base with factory function `dandi_file()` and discovery via `find_dandi_files()` -- **Enum-based configuration**: Operations use enums for modes (e.g., `DownloadExisting`, `FileOperationMode`, `UploadValidation`) -- **Generator-based processing**: Validation, download, and file finding all use generators -- **Context managers**: API clients (`DandiAPIClient`) and URL navigation use context managers -- **Retry logic**: HTTP operations use `tenacity` for exponential backoff retries -- **Lazy imports**: Heavy modules (pynwb, h5py) are imported at point of use, not at module level - -### Key Classes - -- `DandiAPIClient` (`dandiapi.py`): High-level API client with authentication (keyring), pagination, asset management -- `RESTFullAPIClient` (`dandiapi.py`): Base HTTP client with session management and retry logic -- `ParsedDandiURL` (`dandiarchive.py`): Abstract base for URL parsing with subclasses `DandisetURL`, `SingleAssetURL`, `AssetItemURL`, `AssetDirURL` -- `DandiFile` (`files/bases.py`): Abstract base for all file types; subclasses include `NWBAsset`, `ZarrAsset`, `GenericAsset`, `VideoAsset` -- `ValidationResult` (`validate/_types.py`): Pydantic model with origin, severity, scope, message, paths -- `Dandiset` (`dandiset.py`): Local dandiset representation wrapping `dandiset.yaml` -- `DandiInstance` (`consts.py`): Frozen dataclass for known archive instances (dandi, dandi-sandbox, linc, ember-dandi, etc.) +Do NOT guess or assume conventions — read the files. ## Committing -- Due to use of `pre-commit` with black and other auto-fixers, if changes were reported, just rerun commit a 2nd time. Only then if it still does not commit, analyze output further. - -## Test Markers - -- When adding AI-generated tests, mark them with `@pytest.mark.ai_generated` -- Any new pytest markers must be registered in `pytest_configure` function of `dandi/pytest_plugin.py` -- Existing markers: `integration`, `obolibrary`, `flaky`, `ai_generated` - -## Test Infrastructure - -### Key Fixtures (`dandi/tests/fixtures.py`) - -- `simple1_nwb_metadata()` / `simple1_nwb()`: Session-scoped sample NWB file -- `local_dandi_api`: Docker-based local DANDI Archive instance for integration tests -- `new_dandiset()`: Creates a fresh dandiset on the test instance -- `publish_dandiset()`: Publishes a dandiset version -- `capture_all_logs`: Autouse fixture setting DEBUG level for `dandi` logger - -### Test Organization - -- Tests mirror the module structure: `test_download.py`, `test_upload.py`, etc. -- Integration tests requiring docker use the `local_dandi_api` fixture -- `--dandi-api` pytest flag filters to only integration tests -- `--scheduled` flag enables scheduled-only test configuration -- VCR (vcrpy) is used to record/replay HTTP interactions; disable with `DANDI_TESTS_NO_VCR` - -### pytest Configuration (`tox.ini [pytest]`) - -- Default timeout: 300 seconds per test -- `--tb=short --durations=10` by default -- `filterwarnings` set to `error` with specific ignores for known third-party warnings - -## Code Style - -- **Formatter**: Black (line length 100) -- **Import sorting**: isort (profile="black", force_sort_within_sections, reverse_relative) -- **Linting**: flake8 (max-line-length=100, ignore E203/W503) -- **Spell checking**: codespell -- **Type checking**: mypy with pydantic plugin, strict settings -- **Type annotations**: Required for new code -- **Naming**: CamelCase for classes, snake_case for functions/variables -- **Exceptions**: Names must end with "Error" (e.g., `UploadError`, `NotFoundError`) -- **Docstrings**: NumPy style for public APIs -- **Dataclass field docs**: Use `#:` comments above the field (Sphinx autodoc format) -- **Imports**: Organized as stdlib, third-party, local (alphabetical within groups) -- **CLI**: Uses click library patterns with `DYMGroup` (did-you-mean suggestions) -- **Excluded from formatting**: `_version.py`, `due.py`, `versioneer.py` - -## Pre-commit Hooks - -The following hooks run on commit (`.pre-commit-config.yaml`): -1. trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files -2. black (code formatting) -3. isort (import sorting) -4. codespell (spell checking) -5. flake8 (linting) - -## Environment Variables - -- `DANDI_DEVEL`: Enables hidden CLI options (e.g., explicit instance selection) -- `DANDI_LOG_LEVEL`: Log level (default INFO, use int like `10` for DEBUG) -- `DANDI_CACHE`: Persistent cache control (`clear` or `ignore`) -- `DANDI_INSTANCEHOST`: Host for local archive instance (default `localhost`) -- `{INSTANCE_NAME}_API_KEY`: API key per instance (e.g., `DANDI_API_KEY`, `DANDI_SANDBOX_API_KEY`) -- `DANDI_TESTS_PERSIST_DOCKER_COMPOSE`: Reuse Docker containers across test runs -- `DANDI_TESTS_PULL_DOCKER_COMPOSE`: Set to empty/`0` to skip pulling Docker images -- `DANDI_TESTS_NO_VCR`: Disable VCR HTTP replay during tests -- `DANDI_PAGINATION_DISABLE_FALLBACK`: Disable fallback to sequential pagination (set in test envs) - -## CI/CD - -- **Tests** (`run-tests.yml`): Matrix of Python 3.10-3.13 across Ubuntu, macOS (M1 + Intel), Windows -- **Lint** (`lint.yml`): codespell + flake8 -- **Typing** (`typing.yml`): mypy -- **Docs** (`docs.yml`): Sphinx build -- **Release** (`release.yml`): Automated via `auto` tool - PR labels (`major`, `minor`, `patch`, `internal`, etc.) drive changelog and version bumps; tagged releases trigger PyPI upload - -## Documentation - -- Keep docstrings updated when changing function signatures -- CLI help text should be clear and include examples where appropriate -- Dataclass fields: document with `#:` comments above the field, not docstrings below +Due to use of `pre-commit` with black and other auto-fixers, if changes were reported, just +rerun commit a 2nd time. Only then if it still does not commit, analyze output further. ## Issue Tracking with git-bug -This project has GitHub issues synced locally via git-bug. Use these commands +This project has GitHub issues synced locally via git-bug. Use these commands to get issue context without needing GitHub API access: -- `git bug ls status:open` - list open issues -- `git bug show ` - show issue details and comments -- `git bug ls "title:keyword"` - search issues by title -- `git bug ls "label:bug"` - filter by label -- `git bug bridge pull` - sync latest issues from GitHub +- `git bug ls status:open` — list open issues +- `git bug show ` — show issue details and comments +- `git bug ls "title:keyword"` — search issues by title +- `git bug ls "label:bug"` — filter by label +- `git bug bridge pull` — sync latest issues from GitHub When working on a bug fix or feature, check `git bug ls` for related issues to understand context and prior discussion. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..0c98ff7c9 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,203 @@ +# Contributing to dandi-cli + +This document covers the conventions and workflows for contributing to +dandi-cli. For detailed environment setup, environment variables, and +release procedures, see [DEVELOPMENT.md](./DEVELOPMENT.md). + +## Build & Test Quick Reference + +```bash +# Run full test suite +hatch run test:run # via hatch +tox -e py3 # via tox +python -m pytest dandi # in a venv + +# Run a single test +hatch run test:run dandi/tests/test_file.py::test_function -v +tox r -e py3 -- dandi/tests/test_file.py::test_function -v + +# Lint + type checking +tox -e lint,typing + +# Lint only (codespell + flake8) +tox -e lint + +# Type checking only (mypy) +tox -e typing + +# Build docs +tox -e docs + +# Install pre-commit hooks (if .git/hooks/pre-commit is absent) +pre-commit install +``` + +### Integration tests + +Tests that need a running DANDI Archive instance use the `local_dandi_api` +docker-compose fixture. Set `DANDI_TESTS_PULL_DOCKER_COMPOSE=""` to skip +`docker compose pull` and speed up repeated runs. + +## Codebase Architecture + +### Directory layout + +``` +dandi/ + cli/ # Click-based CLI commands + command.py # Entry point — Click group with DYMGroup (did-you-mean) + base.py # Shared CLI utilities, decorators, custom param types + cmd_*.py # One file per command (download, upload, organize, …) + formatter.py # Output formatters (JSON, YAML, JSONL, PYOUT) + files/ # File-type abstractions + bases.py # DandiFile hierarchy (LocalAsset, NWBAsset, …) + bids.py # BIDS-specific file types (NWBBIDSAsset, …) + zarr.py # Zarr archive handling (ZarrAsset, LocalZarrEntry) + metadata/ # Metadata extraction + core.py # Entry points for metadata extraction + nwb.py # NWB-specific extraction via PyNWB + util.py # get_metadata(), field extraction, caching + validate/ # Validation engine + _types.py # ValidationResult, Severity, Scope, Standard enums + _core.py # validate() generator, validate_bids() + _io.py # JSON Lines I/O for validation results + support/ # Shared utilities + digests.py # Checksum/digest computation (DANDI eTag, Zarr) + pyout.py # Progress display with pyout (LogSafeTabular) + iterators.py # IteratorWithAggregation for progress tracking + threaded_walk.py # Parallel directory traversal + tests/ # Test suite + fixtures.py # Core test fixtures (NWB files, local API, dandisets) + skip.py # Conditional skip helpers + data/ # Test data files + consts.py # Constants: metadata fields, known instances, layout fields + dandiapi.py # API client (RESTFullAPIClient, DandiAPIClient) + dandiarchive.py # URL parsing (ParsedDandiURL, parse_dandi_url()) + dandiset.py # Local dandiset representation (dandiset.yaml) + download.py # Download engine with resume/retry support + upload.py # Upload engine with validation + organize.py # File organization by NWB metadata + delete.py # Asset/dandiset deletion + move.py # Asset move/rename (local + remote) + exceptions.py # Custom exceptions (all end with "Error") + misctypes.py # Shared types: Digest, BasePath + pynwb_utils.py # PyNWB helpers for reading/creating NWB files + utils.py # General utilities +``` + +### Key design patterns + +- **CLI delegation** — CLI commands (`cmd_*.py`) are thin wrappers that + delegate to core modules (e.g. `cmd_upload.py` → `upload.upload()`). +- **File-type hierarchy** — `DandiFile` abstract base with factory function + `dandi_file()` and discovery via `find_dandi_files()`. +- **Enum-based configuration** — Operations use enums for modes + (`DownloadExisting`, `FileOperationMode`, `UploadValidation`, …). +- **Generator-based processing** — Validation, download, and file finding + all yield results lazily. +- **Context managers** — API clients (`DandiAPIClient`) and URL navigation. +- **Retry logic** — HTTP operations use `tenacity` for exponential backoff. +- **Lazy imports** — Heavy modules (`pynwb`, `h5py`) are imported at point + of use, not at module level. + +### Key classes + +| Class | Module | Role | +|-------|--------|------| +| `DandiAPIClient` | `dandiapi.py` | High-level API client; authentication (keyring), pagination, asset management | +| `RESTFullAPIClient` | `dandiapi.py` | Base HTTP client with session management and retry logic | +| `ParsedDandiURL` | `dandiarchive.py` | Abstract base for URL parsing; subclasses `DandisetURL`, `SingleAssetURL`, `AssetItemURL`, `AssetDirURL` | +| `DandiFile` | `files/bases.py` | Abstract base for all file types; subclasses `NWBAsset`, `ZarrAsset`, `GenericAsset`, `VideoAsset` | +| `ValidationResult` | `validate/_types.py` | Pydantic model: origin, severity, scope, message, paths | +| `Dandiset` | `dandiset.py` | Local dandiset representation wrapping `dandiset.yaml` | +| `DandiInstance` | `consts.py` | Frozen dataclass for known archive instances | + +## Code Style + +- **Formatter**: Black (line length 100) +- **Import sorting**: isort (`profile="black"`, `force_sort_within_sections`, + `reverse_relative`) +- **Linting**: flake8 (`max-line-length=100`, ignore `E203`/`W503`) +- **Spell checking**: codespell +- **Type checking**: mypy with pydantic plugin +- **Type annotations**: Required for new code +- **Naming**: `CamelCase` for classes, `snake_case` for functions/variables +- **Exceptions**: Names must end with `Error` (e.g. `UploadError`, + `NotFoundError`) +- **Docstrings**: NumPy style for public APIs +- **Dataclass field docs**: `#:` comments above the field (Sphinx autodoc + format — see [DEVELOPMENT.md](./DEVELOPMENT.md#dataclass-and-attrs-field-documentation)) +- **Imports**: stdlib → third-party → local (alphabetical within groups) +- **CLI**: Click library with `DYMGroup` (did-you-mean suggestions) +- **Excluded from formatting**: `_version.py`, `due.py`, `versioneer.py` + +### Pre-commit hooks + +The following hooks run on commit (`.pre-commit-config.yaml`): + +1. trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files +2. black (code formatting) +3. isort (import sorting) +4. codespell (spell checking) +5. flake8 (linting) + +Because black and isort auto-fix files, a commit that triggers fixes will +fail the first time. Simply re-run `git commit` — the second attempt should +succeed. Investigate further only if it still fails. + +## Test Infrastructure + +### pytest markers + +| Marker | Purpose | +|--------|--------| +| `@pytest.mark.integration` | Tests requiring a running archive instance | +| `@pytest.mark.obolibrary` | Tests hitting the OBO ontology library | +| `@pytest.mark.flaky` | Known-flaky tests | +| `@pytest.mark.ai_generated` | **Mandatory** on any test written with AI assistance | + +New markers must be registered in `pytest_configure()` in +`dandi/pytest_plugin.py`. + +### Key fixtures (`dandi/tests/fixtures.py`) + +- `simple1_nwb_metadata()` / `simple1_nwb()` — session-scoped sample NWB file +- `local_dandi_api` — Docker-based local DANDI Archive instance +- `new_dandiset()` — creates a fresh dandiset on the test instance +- `publish_dandiset()` — publishes a dandiset version +- `capture_all_logs` — autouse; sets DEBUG level for `dandi` logger + +### Test organization + +- Tests mirror the module structure: `test_download.py`, `test_upload.py`, etc. +- Integration tests use the `local_dandi_api` fixture +- `--dandi-api` flag: run only integration tests +- `--scheduled` flag: enable configuration for scheduled daily runs +- VCR (vcrpy) records/replays HTTP interactions; disable with + `DANDI_TESTS_NO_VCR` + +### pytest configuration (`tox.ini [pytest]`) + +- Default timeout: 300 s per test +- `--tb=short --durations=10` +- `filterwarnings = error` with specific ignores for known third-party warnings + +## CI/CD + +| Workflow | What it checks | +|----------|---------------| +| `run-tests.yml` | Full test matrix — Python 3.10–3.13 × Ubuntu, macOS (M1 + Intel), Windows | +| `lint.yml` | codespell + flake8 | +| `typing.yml` | mypy | +| `docs.yml` | Sphinx build | +| `release.yml` | Automated release via `auto` — see [DEVELOPMENT.md](./DEVELOPMENT.md#releasing-with-github-actions-auto-and-pull-requests) | + +### PR labels (intuit/auto) + +Every PR should carry a semver or category label; `auto` uses them for +changelog sections and version bumps. Recognized labels: + +`major`, `minor`, `patch` (default), `internal`, `documentation`, `tests`, +`dependencies`, `performance` + +A release is published only when the **`release`** label is present. From 78aeed8c5f482023f9655019e1a775c54b69222c Mon Sep 17 00:00:00 2001 From: "GitMate for @yarikoptic" <41385986+yarikoptic-gitmate@users.noreply.github.com> Date: Thu, 6 Aug 2026 15:50:30 -0400 Subject: [PATCH 3/7] Merge CONTRIBUTING.md into DEVELOPMENT.md, drop CONTRIBUTING.md Move architecture, code style, test infrastructure, and CI/CD sections into DEVELOPMENT.md so all developer documentation lives in one file. Update CLAUDE.md to point at DEVELOPMENT.md instead. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_017qM7WyFgvqy5UQHEtEcHzY --- CLAUDE.md | 7 +-- DEVELOPMENT.md | 164 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 166 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d839925b5..465dc5bc9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## MANDATORY: Read before making any code changes -You MUST read [`CONTRIBUTING.md`](./CONTRIBUTING.md) before making any code changes, commits, or +You MUST read [`DEVELOPMENT.md`](./DEVELOPMENT.md) before making any code changes, commits, or pull requests. It contains the authoritative project conventions including: - Build/test commands and CI/CD overview @@ -14,10 +14,7 @@ pull requests. It contains the authoritative project conventions including: written with AI assistance** - PR labeling and release workflow (intuit/auto) -Extended documentation (environment setup, environment variables, release procedures, git-bug) -is in [`DEVELOPMENT.md`](./DEVELOPMENT.md). - -Do NOT guess or assume conventions — read the files. +Do NOT guess or assume conventions — read the file. ## Committing diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 4ccd4b67a..7f9d8a2ba 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -72,12 +72,22 @@ Alternatively, with `tox` (install via `pip install tox`): tox -e py3 ``` +To run a specific test with tox: +``` +tox r -e py3 -- dandi/tests/test_file.py::test_function -v +``` + In order to check proper linting and typing of your changes you can also run `tox` with `lint` and `typing`: ``` tox -e lint,typing ``` +To build documentation: +``` +tox -e docs +``` + ### dandi-archive instance The [dandi-archive](https://github.com/dandi/dandi-archive) repository provides a @@ -92,8 +102,101 @@ instance as `dandi-api-local-docker-tests`. See the note below on the `DANDI_DEVEL` environment variable, which is needed in order to expose the development command line options. +Tests that need a running archive instance use the `local_dandi_api` +docker-compose fixture. Set `DANDI_TESTS_PULL_DOCKER_COMPOSE=""` to skip +`docker compose pull` and speed up repeated runs. + +## Codebase Architecture + +### Directory layout + +``` +dandi/ + cli/ # Click-based CLI commands + command.py # Entry point — Click group with DYMGroup (did-you-mean) + base.py # Shared CLI utilities, decorators, custom param types + cmd_*.py # One file per command (download, upload, organize, …) + formatter.py # Output formatters (JSON, YAML, JSONL, PYOUT) + files/ # File-type abstractions + bases.py # DandiFile hierarchy (LocalAsset, NWBAsset, …) + bids.py # BIDS-specific file types (NWBBIDSAsset, …) + zarr.py # Zarr archive handling (ZarrAsset, LocalZarrEntry) + metadata/ # Metadata extraction + core.py # Entry points for metadata extraction + nwb.py # NWB-specific extraction via PyNWB + util.py # get_metadata(), field extraction, caching + validate/ # Validation engine + _types.py # ValidationResult, Severity, Scope, Standard enums + _core.py # validate() generator, validate_bids() + _io.py # JSON Lines I/O for validation results + support/ # Shared utilities + digests.py # Checksum/digest computation (DANDI eTag, Zarr) + pyout.py # Progress display with pyout (LogSafeTabular) + iterators.py # IteratorWithAggregation for progress tracking + threaded_walk.py # Parallel directory traversal + tests/ # Test suite + fixtures.py # Core test fixtures (NWB files, local API, dandisets) + skip.py # Conditional skip helpers + data/ # Test data files + consts.py # Constants: metadata fields, known instances, layout fields + dandiapi.py # API client (RESTFullAPIClient, DandiAPIClient) + dandiarchive.py # URL parsing (ParsedDandiURL, parse_dandi_url()) + dandiset.py # Local dandiset representation (dandiset.yaml) + download.py # Download engine with resume/retry support + upload.py # Upload engine with validation + organize.py # File organization by NWB metadata + delete.py # Asset/dandiset deletion + move.py # Asset move/rename (local + remote) + exceptions.py # Custom exceptions (all end with "Error") + misctypes.py # Shared types: Digest, BasePath + pynwb_utils.py # PyNWB helpers for reading/creating NWB files + utils.py # General utilities +``` + +### Key design patterns + +- **CLI delegation** — CLI commands (`cmd_*.py`) are thin wrappers that + delegate to core modules (e.g. `cmd_upload.py` → `upload.upload()`). +- **File-type hierarchy** — `DandiFile` abstract base with factory function + `dandi_file()` and discovery via `find_dandi_files()`. +- **Enum-based configuration** — Operations use enums for modes + (`DownloadExisting`, `FileOperationMode`, `UploadValidation`, …). +- **Generator-based processing** — Validation, download, and file finding + all yield results lazily. +- **Context managers** — API clients (`DandiAPIClient`) and URL navigation. +- **Retry logic** — HTTP operations use `tenacity` for exponential backoff. +- **Lazy imports** — Heavy modules (`pynwb`, `h5py`) are imported at point + of use, not at module level. + +### Key classes + +| Class | Module | Role | +|-------|--------|------| +| `DandiAPIClient` | `dandiapi.py` | High-level API client; authentication (keyring), pagination, asset management | +| `RESTFullAPIClient` | `dandiapi.py` | Base HTTP client with session management and retry logic | +| `ParsedDandiURL` | `dandiarchive.py` | Abstract base for URL parsing; subclasses `DandisetURL`, `SingleAssetURL`, `AssetItemURL`, `AssetDirURL` | +| `DandiFile` | `files/bases.py` | Abstract base for all file types; subclasses `NWBAsset`, `ZarrAsset`, `GenericAsset`, `VideoAsset` | +| `ValidationResult` | `validate/_types.py` | Pydantic model: origin, severity, scope, message, paths | +| `Dandiset` | `dandiset.py` | Local dandiset representation wrapping `dandiset.yaml` | +| `DandiInstance` | `consts.py` | Frozen dataclass for known archive instances | + ## Code style conventions +- **Formatter**: Black (line length 100) +- **Import sorting**: isort (`profile="black"`, `force_sort_within_sections`, + `reverse_relative`) +- **Linting**: flake8 (`max-line-length=100`, ignore `E203`/`W503`) +- **Spell checking**: codespell +- **Type checking**: mypy with pydantic plugin +- **Type annotations**: Required for new code +- **Naming**: `CamelCase` for classes, `snake_case` for functions/variables +- **Exceptions**: Names must end with `Error` (e.g. `UploadError`, + `NotFoundError`) +- **Docstrings**: NumPy style for public APIs +- **Imports**: stdlib → third-party → local (alphabetical within groups) +- **CLI**: Click library with `DYMGroup` (did-you-mean suggestions) +- **Excluded from formatting**: `_version.py`, `due.py`, `versioneer.py` + ### Dataclass and attrs field documentation Document dataclass/attrs fields using `#:` comments above the field, not @@ -117,6 +220,67 @@ class Movement: See [dandi.move.Movement on RTD](https://dandi.readthedocs.io/en/latest/modref/generated/dandi.move.html#dandi.move.Movement) for a rendered example. +### Pre-commit hooks + +The following hooks run on commit (`.pre-commit-config.yaml`): + +1. trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files +2. black (code formatting) +3. isort (import sorting) +4. codespell (spell checking) +5. flake8 (linting) + +Because black and isort auto-fix files, a commit that triggers fixes will +fail the first time. Simply re-run `git commit` — the second attempt should +succeed. Investigate further only if it still fails. + +## Test infrastructure + +### pytest markers + +| Marker | Purpose | +|--------|--------| +| `@pytest.mark.integration` | Tests requiring a running archive instance | +| `@pytest.mark.obolibrary` | Tests hitting the OBO ontology library | +| `@pytest.mark.flaky` | Known-flaky tests | +| `@pytest.mark.ai_generated` | **Mandatory** on any test written with AI assistance | + +New markers must be registered in `pytest_configure()` in +`dandi/pytest_plugin.py`. + +### Key fixtures (`dandi/tests/fixtures.py`) + +- `simple1_nwb_metadata()` / `simple1_nwb()` — session-scoped sample NWB file +- `local_dandi_api` — Docker-based local DANDI Archive instance +- `new_dandiset()` — creates a fresh dandiset on the test instance +- `publish_dandiset()` — publishes a dandiset version +- `capture_all_logs` — autouse; sets DEBUG level for `dandi` logger + +### Test organization + +- Tests mirror the module structure: `test_download.py`, `test_upload.py`, etc. +- Integration tests use the `local_dandi_api` fixture +- `--dandi-api` flag: run only integration tests +- `--scheduled` flag: enable configuration for scheduled daily runs +- VCR (vcrpy) records/replays HTTP interactions; disable with + `DANDI_TESTS_NO_VCR` + +### pytest configuration (`tox.ini [pytest]`) + +- Default timeout: 300 s per test +- `--tb=short --durations=10` +- `filterwarnings = error` with specific ignores for known third-party warnings + +## CI/CD + +| Workflow | What it checks | +|----------|---------------| +| `run-tests.yml` | Full test matrix — Python 3.10–3.13 × Ubuntu, macOS (M1 + Intel), Windows | +| `lint.yml` | codespell + flake8 | +| `typing.yml` | mypy | +| `docs.yml` | Sphinx build | +| `release.yml` | Automated release via `auto` (see below) | + ## Environment variables - `DANDI_DEVEL` -- enables otherwise hidden command line options, such as From 836639781be028aade0292ff3d64f5e550584184 Mon Sep 17 00:00:00 2001 From: "GitMate for @yarikoptic" <41385986+yarikoptic-gitmate@users.noreply.github.com> Date: Thu, 6 Aug 2026 15:50:46 -0400 Subject: [PATCH 4/7] Remove CONTRIBUTING.md (merged into DEVELOPMENT.md) Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_017qM7WyFgvqy5UQHEtEcHzY --- CONTRIBUTING.md | 203 ------------------------------------------------ 1 file changed, 203 deletions(-) delete mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index 0c98ff7c9..000000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,203 +0,0 @@ -# Contributing to dandi-cli - -This document covers the conventions and workflows for contributing to -dandi-cli. For detailed environment setup, environment variables, and -release procedures, see [DEVELOPMENT.md](./DEVELOPMENT.md). - -## Build & Test Quick Reference - -```bash -# Run full test suite -hatch run test:run # via hatch -tox -e py3 # via tox -python -m pytest dandi # in a venv - -# Run a single test -hatch run test:run dandi/tests/test_file.py::test_function -v -tox r -e py3 -- dandi/tests/test_file.py::test_function -v - -# Lint + type checking -tox -e lint,typing - -# Lint only (codespell + flake8) -tox -e lint - -# Type checking only (mypy) -tox -e typing - -# Build docs -tox -e docs - -# Install pre-commit hooks (if .git/hooks/pre-commit is absent) -pre-commit install -``` - -### Integration tests - -Tests that need a running DANDI Archive instance use the `local_dandi_api` -docker-compose fixture. Set `DANDI_TESTS_PULL_DOCKER_COMPOSE=""` to skip -`docker compose pull` and speed up repeated runs. - -## Codebase Architecture - -### Directory layout - -``` -dandi/ - cli/ # Click-based CLI commands - command.py # Entry point — Click group with DYMGroup (did-you-mean) - base.py # Shared CLI utilities, decorators, custom param types - cmd_*.py # One file per command (download, upload, organize, …) - formatter.py # Output formatters (JSON, YAML, JSONL, PYOUT) - files/ # File-type abstractions - bases.py # DandiFile hierarchy (LocalAsset, NWBAsset, …) - bids.py # BIDS-specific file types (NWBBIDSAsset, …) - zarr.py # Zarr archive handling (ZarrAsset, LocalZarrEntry) - metadata/ # Metadata extraction - core.py # Entry points for metadata extraction - nwb.py # NWB-specific extraction via PyNWB - util.py # get_metadata(), field extraction, caching - validate/ # Validation engine - _types.py # ValidationResult, Severity, Scope, Standard enums - _core.py # validate() generator, validate_bids() - _io.py # JSON Lines I/O for validation results - support/ # Shared utilities - digests.py # Checksum/digest computation (DANDI eTag, Zarr) - pyout.py # Progress display with pyout (LogSafeTabular) - iterators.py # IteratorWithAggregation for progress tracking - threaded_walk.py # Parallel directory traversal - tests/ # Test suite - fixtures.py # Core test fixtures (NWB files, local API, dandisets) - skip.py # Conditional skip helpers - data/ # Test data files - consts.py # Constants: metadata fields, known instances, layout fields - dandiapi.py # API client (RESTFullAPIClient, DandiAPIClient) - dandiarchive.py # URL parsing (ParsedDandiURL, parse_dandi_url()) - dandiset.py # Local dandiset representation (dandiset.yaml) - download.py # Download engine with resume/retry support - upload.py # Upload engine with validation - organize.py # File organization by NWB metadata - delete.py # Asset/dandiset deletion - move.py # Asset move/rename (local + remote) - exceptions.py # Custom exceptions (all end with "Error") - misctypes.py # Shared types: Digest, BasePath - pynwb_utils.py # PyNWB helpers for reading/creating NWB files - utils.py # General utilities -``` - -### Key design patterns - -- **CLI delegation** — CLI commands (`cmd_*.py`) are thin wrappers that - delegate to core modules (e.g. `cmd_upload.py` → `upload.upload()`). -- **File-type hierarchy** — `DandiFile` abstract base with factory function - `dandi_file()` and discovery via `find_dandi_files()`. -- **Enum-based configuration** — Operations use enums for modes - (`DownloadExisting`, `FileOperationMode`, `UploadValidation`, …). -- **Generator-based processing** — Validation, download, and file finding - all yield results lazily. -- **Context managers** — API clients (`DandiAPIClient`) and URL navigation. -- **Retry logic** — HTTP operations use `tenacity` for exponential backoff. -- **Lazy imports** — Heavy modules (`pynwb`, `h5py`) are imported at point - of use, not at module level. - -### Key classes - -| Class | Module | Role | -|-------|--------|------| -| `DandiAPIClient` | `dandiapi.py` | High-level API client; authentication (keyring), pagination, asset management | -| `RESTFullAPIClient` | `dandiapi.py` | Base HTTP client with session management and retry logic | -| `ParsedDandiURL` | `dandiarchive.py` | Abstract base for URL parsing; subclasses `DandisetURL`, `SingleAssetURL`, `AssetItemURL`, `AssetDirURL` | -| `DandiFile` | `files/bases.py` | Abstract base for all file types; subclasses `NWBAsset`, `ZarrAsset`, `GenericAsset`, `VideoAsset` | -| `ValidationResult` | `validate/_types.py` | Pydantic model: origin, severity, scope, message, paths | -| `Dandiset` | `dandiset.py` | Local dandiset representation wrapping `dandiset.yaml` | -| `DandiInstance` | `consts.py` | Frozen dataclass for known archive instances | - -## Code Style - -- **Formatter**: Black (line length 100) -- **Import sorting**: isort (`profile="black"`, `force_sort_within_sections`, - `reverse_relative`) -- **Linting**: flake8 (`max-line-length=100`, ignore `E203`/`W503`) -- **Spell checking**: codespell -- **Type checking**: mypy with pydantic plugin -- **Type annotations**: Required for new code -- **Naming**: `CamelCase` for classes, `snake_case` for functions/variables -- **Exceptions**: Names must end with `Error` (e.g. `UploadError`, - `NotFoundError`) -- **Docstrings**: NumPy style for public APIs -- **Dataclass field docs**: `#:` comments above the field (Sphinx autodoc - format — see [DEVELOPMENT.md](./DEVELOPMENT.md#dataclass-and-attrs-field-documentation)) -- **Imports**: stdlib → third-party → local (alphabetical within groups) -- **CLI**: Click library with `DYMGroup` (did-you-mean suggestions) -- **Excluded from formatting**: `_version.py`, `due.py`, `versioneer.py` - -### Pre-commit hooks - -The following hooks run on commit (`.pre-commit-config.yaml`): - -1. trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files -2. black (code formatting) -3. isort (import sorting) -4. codespell (spell checking) -5. flake8 (linting) - -Because black and isort auto-fix files, a commit that triggers fixes will -fail the first time. Simply re-run `git commit` — the second attempt should -succeed. Investigate further only if it still fails. - -## Test Infrastructure - -### pytest markers - -| Marker | Purpose | -|--------|--------| -| `@pytest.mark.integration` | Tests requiring a running archive instance | -| `@pytest.mark.obolibrary` | Tests hitting the OBO ontology library | -| `@pytest.mark.flaky` | Known-flaky tests | -| `@pytest.mark.ai_generated` | **Mandatory** on any test written with AI assistance | - -New markers must be registered in `pytest_configure()` in -`dandi/pytest_plugin.py`. - -### Key fixtures (`dandi/tests/fixtures.py`) - -- `simple1_nwb_metadata()` / `simple1_nwb()` — session-scoped sample NWB file -- `local_dandi_api` — Docker-based local DANDI Archive instance -- `new_dandiset()` — creates a fresh dandiset on the test instance -- `publish_dandiset()` — publishes a dandiset version -- `capture_all_logs` — autouse; sets DEBUG level for `dandi` logger - -### Test organization - -- Tests mirror the module structure: `test_download.py`, `test_upload.py`, etc. -- Integration tests use the `local_dandi_api` fixture -- `--dandi-api` flag: run only integration tests -- `--scheduled` flag: enable configuration for scheduled daily runs -- VCR (vcrpy) records/replays HTTP interactions; disable with - `DANDI_TESTS_NO_VCR` - -### pytest configuration (`tox.ini [pytest]`) - -- Default timeout: 300 s per test -- `--tb=short --durations=10` -- `filterwarnings = error` with specific ignores for known third-party warnings - -## CI/CD - -| Workflow | What it checks | -|----------|---------------| -| `run-tests.yml` | Full test matrix — Python 3.10–3.13 × Ubuntu, macOS (M1 + Intel), Windows | -| `lint.yml` | codespell + flake8 | -| `typing.yml` | mypy | -| `docs.yml` | Sphinx build | -| `release.yml` | Automated release via `auto` — see [DEVELOPMENT.md](./DEVELOPMENT.md#releasing-with-github-actions-auto-and-pull-requests) | - -### PR labels (intuit/auto) - -Every PR should carry a semver or category label; `auto` uses them for -changelog sections and version bumps. Recognized labels: - -`major`, `minor`, `patch` (default), `internal`, `documentation`, `tests`, -`dependencies`, `performance` - -A release is published only when the **`release`** label is present. From 7dc00274095cafca577505fc74e356e17b5e11dd Mon Sep 17 00:00:00 2001 From: "GitMate for @yarikoptic" <41385986+yarikoptic-gitmate@users.noreply.github.com> Date: Thu, 6 Aug 2026 16:05:45 -0400 Subject: [PATCH 5/7] Polish DEVELOPMENT.md: pre-commit intro, replace tables with lists - Note that most code style rules are enforced by pre-commit hooks. - Replace pipe tables (key classes, pytest markers, CI/CD) with indented bullet lists for readability in plain text. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_017qM7WyFgvqy5UQHEtEcHzY --- DEVELOPMENT.md | 48 ++++++++++++++++++++++++++---------------------- 1 file changed, 26 insertions(+), 22 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 7f9d8a2ba..06caaeba6 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -170,18 +170,24 @@ dandi/ ### Key classes -| Class | Module | Role | -|-------|--------|------| -| `DandiAPIClient` | `dandiapi.py` | High-level API client; authentication (keyring), pagination, asset management | -| `RESTFullAPIClient` | `dandiapi.py` | Base HTTP client with session management and retry logic | -| `ParsedDandiURL` | `dandiarchive.py` | Abstract base for URL parsing; subclasses `DandisetURL`, `SingleAssetURL`, `AssetItemURL`, `AssetDirURL` | -| `DandiFile` | `files/bases.py` | Abstract base for all file types; subclasses `NWBAsset`, `ZarrAsset`, `GenericAsset`, `VideoAsset` | -| `ValidationResult` | `validate/_types.py` | Pydantic model: origin, severity, scope, message, paths | -| `Dandiset` | `dandiset.py` | Local dandiset representation wrapping `dandiset.yaml` | -| `DandiInstance` | `consts.py` | Frozen dataclass for known archive instances | +- `DandiAPIClient` (`dandiapi.py`) — high-level API client; authentication + (keyring), pagination, asset management +- `RESTFullAPIClient` (`dandiapi.py`) — base HTTP client with session + management and retry logic +- `ParsedDandiURL` (`dandiarchive.py`) — abstract base for URL parsing; + subclasses `DandisetURL`, `SingleAssetURL`, `AssetItemURL`, `AssetDirURL` +- `DandiFile` (`files/bases.py`) — abstract base for all file types; + subclasses `NWBAsset`, `ZarrAsset`, `GenericAsset`, `VideoAsset` +- `ValidationResult` (`validate/_types.py`) — Pydantic model: origin, + severity, scope, message, paths +- `Dandiset` (`dandiset.py`) — local dandiset representation wrapping + `dandiset.yaml` +- `DandiInstance` (`consts.py`) — frozen dataclass for known archive instances ## Code style conventions +Most of these are enforced automatically by `pre-commit` hooks (see below). + - **Formatter**: Black (line length 100) - **Import sorting**: isort (`profile="black"`, `force_sort_within_sections`, `reverse_relative`) @@ -238,12 +244,11 @@ succeed. Investigate further only if it still fails. ### pytest markers -| Marker | Purpose | -|--------|--------| -| `@pytest.mark.integration` | Tests requiring a running archive instance | -| `@pytest.mark.obolibrary` | Tests hitting the OBO ontology library | -| `@pytest.mark.flaky` | Known-flaky tests | -| `@pytest.mark.ai_generated` | **Mandatory** on any test written with AI assistance | +- `@pytest.mark.integration` — tests requiring a running archive instance +- `@pytest.mark.obolibrary` — tests hitting the OBO ontology library +- `@pytest.mark.flaky` — known-flaky tests +- `@pytest.mark.ai_generated` — **mandatory** on any test written with AI + assistance New markers must be registered in `pytest_configure()` in `dandi/pytest_plugin.py`. @@ -273,13 +278,12 @@ New markers must be registered in `pytest_configure()` in ## CI/CD -| Workflow | What it checks | -|----------|---------------| -| `run-tests.yml` | Full test matrix — Python 3.10–3.13 × Ubuntu, macOS (M1 + Intel), Windows | -| `lint.yml` | codespell + flake8 | -| `typing.yml` | mypy | -| `docs.yml` | Sphinx build | -| `release.yml` | Automated release via `auto` (see below) | +- `run-tests.yml` — full test matrix: Python 3.10–3.13 × Ubuntu, + macOS (M1 + Intel), Windows +- `lint.yml` — codespell + flake8 +- `typing.yml` — mypy +- `docs.yml` — Sphinx build +- `release.yml` — automated release via `auto` (see below) ## Environment variables From a2e2e0485a7d3013549f19344264b60d2f95ce7b Mon Sep 17 00:00:00 2001 From: "GitMate for @yarikoptic" <41385986+yarikoptic-gitmate@users.noreply.github.com> Date: Fri, 7 Aug 2026 05:29:27 -0400 Subject: [PATCH 6/7] Reference .lad/ framework from CLAUDE.md Point Claude at the LAD (LLM-Assisted Development) prompt workflows so it knows about the phased development framework when asked to use it. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_017qM7WyFgvqy5UQHEtEcHzY --- CLAUDE.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 465dc5bc9..9ad499d03 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,6 +16,16 @@ pull requests. It contains the authoritative project conventions including: Do NOT guess or assume conventions — read the file. +## LLM-Assisted Development (LAD) Framework + +The [`.lad/`](./.lad/) directory contains the +[LAD framework](https://github.com/chrisfoulon/LAD) — structured prompt +workflows for feature development using Claude Code or GitHub Copilot Agent +Mode. When asked to "use LAD" or to follow a phased development workflow, +start from [`.lad/claude_prompts/00_feature_kickoff.md`](./.lad/claude_prompts/00_feature_kickoff.md). +See [`.lad/README.md`](./.lad/README.md) for the full overview and +[`.lad/CLAUDE.md`](./.lad/CLAUDE.md) for project-specific LAD context. + ## Committing Due to use of `pre-commit` with black and other auto-fixers, if changes were reported, just From b33cd10b58001d89898c9729ed7aef4cb4e8ef55 Mon Sep 17 00:00:00 2001 From: "GitMate for @yarikoptic" <41385986+yarikoptic-gitmate@users.noreply.github.com> Date: Fri, 7 Aug 2026 11:22:55 -0400 Subject: [PATCH 7/7] Replace 'tox r' with 'tox -e' for consistency MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 'tox r' is a tox 4 alias for 'tox run' — valid but less obvious. Use the same 'tox -e' form as the rest of the file. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_017qM7WyFgvqy5UQHEtEcHzY --- DEVELOPMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 06caaeba6..60953a662 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -74,7 +74,7 @@ tox -e py3 To run a specific test with tox: ``` -tox r -e py3 -- dandi/tests/test_file.py::test_function -v +tox -e py3 -- dandi/tests/test_file.py::test_function -v ``` In order to check proper linting and typing of your changes