diff --git a/CLAUDE.md b/CLAUDE.md index e8a64b38c..9ad499d03 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,50 +2,44 @@ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. -## Build/Test Commands -- Run tests with hatch: `hatch run test:run` -- Run tests with tox: `tox -e py3` or `python -m pytest dandi` if in a venv -- Tests which require an instance of the archive, would use a fixture to start on using docker-compose. -- Set env var `DANDI_TESTS_PULL_DOCKER_COMPOSE=""` (to empty value) to avoid `docker compose pull` to speed up repetitive runs -- Run single test with hatch: `hatch run test:run dandi/tests/test_file.py::test_function -v` -- Run single test with tox: `tox r -e py3 -- dandi/tests/test_file.py::test_function -v` -- Lint and type checking: `tox -e lint,typing` -- Install pre-commit hooks (if not installed as could be indicated by absence of - `.git/hooks/pre-commit`): `pre-commit install` +## MANDATORY: Read before making any code changes + +You MUST read [`DEVELOPMENT.md`](./DEVELOPMENT.md) before making any code changes, commits, or +pull requests. It contains the authoritative project conventions including: + +- Build/test commands and CI/CD overview +- Codebase architecture, directory layout, key classes, and design patterns +- Code style rules (formatting, imports, type annotations, docstrings) +- Testing requirements, including the **mandatory `@pytest.mark.ai_generated` marker on any test + written with AI assistance** +- PR labeling and release workflow (intuit/auto) + +Do NOT guess or assume conventions — read the file. + +## LLM-Assisted Development (LAD) Framework + +The [`.lad/`](./.lad/) directory contains the +[LAD framework](https://github.com/chrisfoulon/LAD) — structured prompt +workflows for feature development using Claude Code or GitHub Copilot Agent +Mode. When asked to "use LAD" or to follow a phased development workflow, +start from [`.lad/claude_prompts/00_feature_kickoff.md`](./.lad/claude_prompts/00_feature_kickoff.md). +See [`.lad/README.md`](./.lad/README.md) for the full overview and +[`.lad/CLAUDE.md`](./.lad/CLAUDE.md) for project-specific LAD context. ## Committing -- Due to use of `pre-commit` with black and other commands which auto-fix, if changes - were reported to be done, just rerun commit again 2nd time, and if only then if still - does not commit analyze output more - -## Test Markers -- When adding AI-generated tests, mark them with `@pytest.mark.ai_generated` -- Any new pytest markers must be registered in `pytest_configure` function of `dandi/pytest_plugin.py` - -## Code Style -- Code is formatted with Black (line length 100) -- Imports sorted with isort (profile="black") -- Type annotations required for new code -- Use PEP 440 for versioning -- Class names: CamelCase; functions/variables: snake_case -- Exception names end with "Error" (e.g., `ValidateError`) -- Docstrings in NumPy style for public APIs -- Prefer specific exceptions over generic ones -- For CLI, use click library patterns -- Imports organized: stdlib, third-party, local (alphabetical within groups) - -## Documentation -- Keep docstrings updated when changing function signatures -- CLI help text should be clear and include examples where appropriate + +Due to use of `pre-commit` with black and other auto-fixers, if changes were reported, just +rerun commit a 2nd time. Only then if it still does not commit, analyze output further. ## Issue Tracking with git-bug + This project has GitHub issues synced locally via git-bug. Use these commands to get issue context without needing GitHub API access: -- `git bug ls status:open` - list open issues -- `git bug show ` - show issue details and comments -- `git bug ls "title:keyword"` - search issues by title -- `git bug ls "label:bug"` - filter by label -- `git bug bridge pull` - sync latest issues from GitHub +- `git bug ls status:open` — list open issues +- `git bug show ` — show issue details and comments +- `git bug ls "title:keyword"` — search issues by title +- `git bug ls "label:bug"` — filter by label +- `git bug bridge pull` — sync latest issues from GitHub When working on a bug fix or feature, check `git bug ls` for related issues to understand context and prior discussion. diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 4ccd4b67a..60953a662 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -72,12 +72,22 @@ Alternatively, with `tox` (install via `pip install tox`): tox -e py3 ``` +To run a specific test with tox: +``` +tox -e py3 -- dandi/tests/test_file.py::test_function -v +``` + In order to check proper linting and typing of your changes you can also run `tox` with `lint` and `typing`: ``` tox -e lint,typing ``` +To build documentation: +``` +tox -e docs +``` + ### dandi-archive instance The [dandi-archive](https://github.com/dandi/dandi-archive) repository provides a @@ -92,8 +102,107 @@ instance as `dandi-api-local-docker-tests`. See the note below on the `DANDI_DEVEL` environment variable, which is needed in order to expose the development command line options. +Tests that need a running archive instance use the `local_dandi_api` +docker-compose fixture. Set `DANDI_TESTS_PULL_DOCKER_COMPOSE=""` to skip +`docker compose pull` and speed up repeated runs. + +## Codebase Architecture + +### Directory layout + +``` +dandi/ + cli/ # Click-based CLI commands + command.py # Entry point — Click group with DYMGroup (did-you-mean) + base.py # Shared CLI utilities, decorators, custom param types + cmd_*.py # One file per command (download, upload, organize, …) + formatter.py # Output formatters (JSON, YAML, JSONL, PYOUT) + files/ # File-type abstractions + bases.py # DandiFile hierarchy (LocalAsset, NWBAsset, …) + bids.py # BIDS-specific file types (NWBBIDSAsset, …) + zarr.py # Zarr archive handling (ZarrAsset, LocalZarrEntry) + metadata/ # Metadata extraction + core.py # Entry points for metadata extraction + nwb.py # NWB-specific extraction via PyNWB + util.py # get_metadata(), field extraction, caching + validate/ # Validation engine + _types.py # ValidationResult, Severity, Scope, Standard enums + _core.py # validate() generator, validate_bids() + _io.py # JSON Lines I/O for validation results + support/ # Shared utilities + digests.py # Checksum/digest computation (DANDI eTag, Zarr) + pyout.py # Progress display with pyout (LogSafeTabular) + iterators.py # IteratorWithAggregation for progress tracking + threaded_walk.py # Parallel directory traversal + tests/ # Test suite + fixtures.py # Core test fixtures (NWB files, local API, dandisets) + skip.py # Conditional skip helpers + data/ # Test data files + consts.py # Constants: metadata fields, known instances, layout fields + dandiapi.py # API client (RESTFullAPIClient, DandiAPIClient) + dandiarchive.py # URL parsing (ParsedDandiURL, parse_dandi_url()) + dandiset.py # Local dandiset representation (dandiset.yaml) + download.py # Download engine with resume/retry support + upload.py # Upload engine with validation + organize.py # File organization by NWB metadata + delete.py # Asset/dandiset deletion + move.py # Asset move/rename (local + remote) + exceptions.py # Custom exceptions (all end with "Error") + misctypes.py # Shared types: Digest, BasePath + pynwb_utils.py # PyNWB helpers for reading/creating NWB files + utils.py # General utilities +``` + +### Key design patterns + +- **CLI delegation** — CLI commands (`cmd_*.py`) are thin wrappers that + delegate to core modules (e.g. `cmd_upload.py` → `upload.upload()`). +- **File-type hierarchy** — `DandiFile` abstract base with factory function + `dandi_file()` and discovery via `find_dandi_files()`. +- **Enum-based configuration** — Operations use enums for modes + (`DownloadExisting`, `FileOperationMode`, `UploadValidation`, …). +- **Generator-based processing** — Validation, download, and file finding + all yield results lazily. +- **Context managers** — API clients (`DandiAPIClient`) and URL navigation. +- **Retry logic** — HTTP operations use `tenacity` for exponential backoff. +- **Lazy imports** — Heavy modules (`pynwb`, `h5py`) are imported at point + of use, not at module level. + +### Key classes + +- `DandiAPIClient` (`dandiapi.py`) — high-level API client; authentication + (keyring), pagination, asset management +- `RESTFullAPIClient` (`dandiapi.py`) — base HTTP client with session + management and retry logic +- `ParsedDandiURL` (`dandiarchive.py`) — abstract base for URL parsing; + subclasses `DandisetURL`, `SingleAssetURL`, `AssetItemURL`, `AssetDirURL` +- `DandiFile` (`files/bases.py`) — abstract base for all file types; + subclasses `NWBAsset`, `ZarrAsset`, `GenericAsset`, `VideoAsset` +- `ValidationResult` (`validate/_types.py`) — Pydantic model: origin, + severity, scope, message, paths +- `Dandiset` (`dandiset.py`) — local dandiset representation wrapping + `dandiset.yaml` +- `DandiInstance` (`consts.py`) — frozen dataclass for known archive instances + ## Code style conventions +Most of these are enforced automatically by `pre-commit` hooks (see below). + +- **Formatter**: Black (line length 100) +- **Import sorting**: isort (`profile="black"`, `force_sort_within_sections`, + `reverse_relative`) +- **Linting**: flake8 (`max-line-length=100`, ignore `E203`/`W503`) +- **Spell checking**: codespell +- **Type checking**: mypy with pydantic plugin +- **Type annotations**: Required for new code +- **Naming**: `CamelCase` for classes, `snake_case` for functions/variables +- **Exceptions**: Names must end with `Error` (e.g. `UploadError`, + `NotFoundError`) +- **Docstrings**: NumPy style for public APIs +- **Imports**: stdlib → third-party → local (alphabetical within groups) +- **CLI**: Click library with `DYMGroup` (did-you-mean suggestions) +- **Excluded from formatting**: `_version.py`, `due.py`, `versioneer.py` + ### Dataclass and attrs field documentation Document dataclass/attrs fields using `#:` comments above the field, not @@ -117,6 +226,65 @@ class Movement: See [dandi.move.Movement on RTD](https://dandi.readthedocs.io/en/latest/modref/generated/dandi.move.html#dandi.move.Movement) for a rendered example. +### Pre-commit hooks + +The following hooks run on commit (`.pre-commit-config.yaml`): + +1. trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files +2. black (code formatting) +3. isort (import sorting) +4. codespell (spell checking) +5. flake8 (linting) + +Because black and isort auto-fix files, a commit that triggers fixes will +fail the first time. Simply re-run `git commit` — the second attempt should +succeed. Investigate further only if it still fails. + +## Test infrastructure + +### pytest markers + +- `@pytest.mark.integration` — tests requiring a running archive instance +- `@pytest.mark.obolibrary` — tests hitting the OBO ontology library +- `@pytest.mark.flaky` — known-flaky tests +- `@pytest.mark.ai_generated` — **mandatory** on any test written with AI + assistance + +New markers must be registered in `pytest_configure()` in +`dandi/pytest_plugin.py`. + +### Key fixtures (`dandi/tests/fixtures.py`) + +- `simple1_nwb_metadata()` / `simple1_nwb()` — session-scoped sample NWB file +- `local_dandi_api` — Docker-based local DANDI Archive instance +- `new_dandiset()` — creates a fresh dandiset on the test instance +- `publish_dandiset()` — publishes a dandiset version +- `capture_all_logs` — autouse; sets DEBUG level for `dandi` logger + +### Test organization + +- Tests mirror the module structure: `test_download.py`, `test_upload.py`, etc. +- Integration tests use the `local_dandi_api` fixture +- `--dandi-api` flag: run only integration tests +- `--scheduled` flag: enable configuration for scheduled daily runs +- VCR (vcrpy) records/replays HTTP interactions; disable with + `DANDI_TESTS_NO_VCR` + +### pytest configuration (`tox.ini [pytest]`) + +- Default timeout: 300 s per test +- `--tb=short --durations=10` +- `filterwarnings = error` with specific ignores for known third-party warnings + +## CI/CD + +- `run-tests.yml` — full test matrix: Python 3.10–3.13 × Ubuntu, + macOS (M1 + Intel), Windows +- `lint.yml` — codespell + flake8 +- `typing.yml` — mypy +- `docs.yml` — Sphinx build +- `release.yml` — automated release via `auto` (see below) + ## Environment variables - `DANDI_DEVEL` -- enables otherwise hidden command line options, such as