docs: expand DEVELOPMENT.md with architecture and testing guide

CLAUDE.md

@@ -2,50 +2,44 @@
 
 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 
-## Build/Test Commands
-- Run tests with hatch: `hatch run test:run`
-- Run tests with tox: `tox -e py3` or `python -m pytest dandi` if in a venv
-- Tests which require an instance of the archive, would use a fixture to start on using docker-compose.
-- Set env var `DANDI_TESTS_PULL_DOCKER_COMPOSE=""` (to empty value) to avoid `docker compose pull` to speed up repetitive runs
-- Run single test with hatch: `hatch run test:run dandi/tests/test_file.py::test_function -v`
-- Run single test with tox: `tox r -e py3 -- dandi/tests/test_file.py::test_function -v`
-- Lint and type checking: `tox -e lint,typing`
-- Install pre-commit hooks (if not installed as could be indicated by absence of
-  `.git/hooks/pre-commit`): `pre-commit install`
+## MANDATORY: Read before making any code changes
+
+You MUST read [`DEVELOPMENT.md`](./DEVELOPMENT.md) before making any code changes, commits, or
+pull requests.  It contains the authoritative project conventions including:
+
+- Build/test commands and CI/CD overview
+- Codebase architecture, directory layout, key classes, and design patterns
+- Code style rules (formatting, imports, type annotations, docstrings)
+- Testing requirements, including the **mandatory `@pytest.mark.ai_generated` marker on any test
+  written with AI assistance**
+- PR labeling and release workflow (intuit/auto)
+
+Do NOT guess or assume conventions — read the file.
+
+## LLM-Assisted Development (LAD) Framework
+
+The [`.lad/`](./.lad/) directory contains the
+[LAD framework](https://github.com/chrisfoulon/LAD) — structured prompt
+workflows for feature development using Claude Code or GitHub Copilot Agent
+Mode.  When asked to "use LAD" or to follow a phased development workflow,
+start from [`.lad/claude_prompts/00_feature_kickoff.md`](./.lad/claude_prompts/00_feature_kickoff.md).
+See [`.lad/README.md`](./.lad/README.md) for the full overview and
+[`.lad/CLAUDE.md`](./.lad/CLAUDE.md) for project-specific LAD context.
 
 ## Committing
-- Due to use of `pre-commit` with black and other commands which auto-fix, if changes
-  were reported to be done, just rerun commit again 2nd time, and if only then if still
-  does not commit analyze output more
-
-## Test Markers
-- When adding AI-generated tests, mark them with `@pytest.mark.ai_generated`
-- Any new pytest markers must be registered in `pytest_configure` function of `dandi/pytest_plugin.py`
-
-## Code Style
-- Code is formatted with Black (line length 100)
-- Imports sorted with isort (profile="black")
-- Type annotations required for new code
-- Use PEP 440 for versioning
-- Class names: CamelCase; functions/variables: snake_case
-- Exception names end with "Error" (e.g., `ValidateError`)
-- Docstrings in NumPy style for public APIs
-- Prefer specific exceptions over generic ones
-- For CLI, use click library patterns
-- Imports organized: stdlib, third-party, local (alphabetical within groups)
-
-## Documentation
-- Keep docstrings updated when changing function signatures
-- CLI help text should be clear and include examples where appropriate
+
+Due to use of `pre-commit` with black and other auto-fixers, if changes were reported, just
+rerun commit a 2nd time.  Only then if it still does not commit, analyze output further.
 
 ## Issue Tracking with git-bug
+
 This project has GitHub issues synced locally via git-bug.  Use these commands
 to get issue context without needing GitHub API access:
-- `git bug ls status:open` - list open issues
-- `git bug show <id-prefix>` - show issue details and comments
-- `git bug ls "title:keyword"` - search issues by title
-- `git bug ls "label:bug"` - filter by label
-- `git bug bridge pull` - sync latest issues from GitHub
+- `git bug ls status:open` — list open issues
+- `git bug show <id-prefix>` — show issue details and comments
+- `git bug ls "title:keyword"` — search issues by title
+- `git bug ls "label:bug"` — filter by label
+- `git bug bridge pull` — sync latest issues from GitHub
 
 When working on a bug fix or feature, check `git bug ls` for related issues
 to understand context and prior discussion.

DEVELOPMENT.md

@@ -72,12 +72,22 @@ Alternatively, with `tox` (install via `pip install tox`):
 tox -e py3
 ```
 
+To run a specific test with tox:
+```
+tox -e py3 -- dandi/tests/test_file.py::test_function -v
+```
+
 In order to check proper linting and typing of your changes
 you can also run `tox` with `lint` and `typing`:
 ```
 tox -e lint,typing
 ```
 
+To build documentation:
+```
+tox -e docs
+```
+
 ### dandi-archive instance
 
 The [dandi-archive](https://github.com/dandi/dandi-archive) repository provides a
@@ -92,8 +102,107 @@ instance as `dandi-api-local-docker-tests`.  See the note below on the
 `DANDI_DEVEL` environment variable, which is needed in order to expose the
 development command line options.
 
+Tests that need a running archive instance use the `local_dandi_api`
+docker-compose fixture.  Set `DANDI_TESTS_PULL_DOCKER_COMPOSE=""` to skip
+`docker compose pull` and speed up repeated runs.
+
+## Codebase Architecture
+
+### Directory layout
+
+```
+dandi/
+  cli/              # Click-based CLI commands
+    command.py      # Entry point — Click group with DYMGroup (did-you-mean)
+    base.py         # Shared CLI utilities, decorators, custom param types
+    cmd_*.py        # One file per command (download, upload, organize, …)
+    formatter.py    # Output formatters (JSON, YAML, JSONL, PYOUT)
+  files/            # File-type abstractions
+    bases.py        # DandiFile hierarchy (LocalAsset, NWBAsset, …)
+    bids.py         # BIDS-specific file types (NWBBIDSAsset, …)
+    zarr.py         # Zarr archive handling (ZarrAsset, LocalZarrEntry)
+  metadata/         # Metadata extraction
+    core.py         # Entry points for metadata extraction
+    nwb.py          # NWB-specific extraction via PyNWB
+    util.py         # get_metadata(), field extraction, caching
+  validate/         # Validation engine
+    _types.py       # ValidationResult, Severity, Scope, Standard enums
+    _core.py        # validate() generator, validate_bids()
+    _io.py          # JSON Lines I/O for validation results
+  support/          # Shared utilities
+    digests.py      # Checksum/digest computation (DANDI eTag, Zarr)
+    pyout.py        # Progress display with pyout (LogSafeTabular)
+    iterators.py    # IteratorWithAggregation for progress tracking
+    threaded_walk.py # Parallel directory traversal
+  tests/            # Test suite
+    fixtures.py     # Core test fixtures (NWB files, local API, dandisets)
+    skip.py         # Conditional skip helpers
+    data/           # Test data files
+  consts.py         # Constants: metadata fields, known instances, layout fields
+  dandiapi.py       # API client (RESTFullAPIClient, DandiAPIClient)
+  dandiarchive.py   # URL parsing (ParsedDandiURL, parse_dandi_url())
+  dandiset.py       # Local dandiset representation (dandiset.yaml)
+  download.py       # Download engine with resume/retry support
+  upload.py         # Upload engine with validation
+  organize.py       # File organization by NWB metadata
+  delete.py         # Asset/dandiset deletion
+  move.py           # Asset move/rename (local + remote)
+  exceptions.py     # Custom exceptions (all end with "Error")
+  misctypes.py      # Shared types: Digest, BasePath
+  pynwb_utils.py    # PyNWB helpers for reading/creating NWB files
+  utils.py          # General utilities
+```
+
+### Key design patterns
+
+- **CLI delegation** — CLI commands (`cmd_*.py`) are thin wrappers that
+  delegate to core modules (e.g. `cmd_upload.py` → `upload.upload()`).
+- **File-type hierarchy** — `DandiFile` abstract base with factory function
+  `dandi_file()` and discovery via `find_dandi_files()`.
+- **Enum-based configuration** — Operations use enums for modes
+  (`DownloadExisting`, `FileOperationMode`, `UploadValidation`, …).
+- **Generator-based processing** — Validation, download, and file finding
+  all yield results lazily.
+- **Context managers** — API clients (`DandiAPIClient`) and URL navigation.
+- **Retry logic** — HTTP operations use `tenacity` for exponential backoff.
+- **Lazy imports** — Heavy modules (`pynwb`, `h5py`) are imported at point
+  of use, not at module level.
+
+### Key classes
+
+- `DandiAPIClient` (`dandiapi.py`) — high-level API client; authentication
+  (keyring), pagination, asset management
+- `RESTFullAPIClient` (`dandiapi.py`) — base HTTP client with session
+  management and retry logic
+- `ParsedDandiURL` (`dandiarchive.py`) — abstract base for URL parsing;
+  subclasses `DandisetURL`, `SingleAssetURL`, `AssetItemURL`, `AssetDirURL`
+- `DandiFile` (`files/bases.py`) — abstract base for all file types;
+  subclasses `NWBAsset`, `ZarrAsset`, `GenericAsset`, `VideoAsset`
+- `ValidationResult` (`validate/_types.py`) — Pydantic model: origin,
+  severity, scope, message, paths
+- `Dandiset` (`dandiset.py`) — local dandiset representation wrapping
+  `dandiset.yaml`
+- `DandiInstance` (`consts.py`) — frozen dataclass for known archive instances
+
 ## Code style conventions
 
+Most of these are enforced automatically by `pre-commit` hooks (see below).
+
+- **Formatter**: Black (line length 100)
+- **Import sorting**: isort (`profile="black"`, `force_sort_within_sections`,
+  `reverse_relative`)
+- **Linting**: flake8 (`max-line-length=100`, ignore `E203`/`W503`)
+- **Spell checking**: codespell
+- **Type checking**: mypy with pydantic plugin
+- **Type annotations**: Required for new code
+- **Naming**: `CamelCase` for classes, `snake_case` for functions/variables
+- **Exceptions**: Names must end with `Error` (e.g. `UploadError`,
+  `NotFoundError`)
+- **Docstrings**: NumPy style for public APIs
+- **Imports**: stdlib → third-party → local (alphabetical within groups)
+- **CLI**: Click library with `DYMGroup` (did-you-mean suggestions)
+- **Excluded from formatting**: `_version.py`, `due.py`, `versioneer.py`
+
 ### Dataclass and attrs field documentation
 
 Document dataclass/attrs fields using `#:` comments above the field, not
@@ -117,6 +226,65 @@ class Movement:
 See [dandi.move.Movement on RTD](https://dandi.readthedocs.io/en/latest/modref/generated/dandi.move.html#dandi.move.Movement)
 for a rendered example.
 
+### Pre-commit hooks
+
+The following hooks run on commit (`.pre-commit-config.yaml`):
+
+1. trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files
+2. black (code formatting)
+3. isort (import sorting)
+4. codespell (spell checking)
+5. flake8 (linting)
+
+Because black and isort auto-fix files, a commit that triggers fixes will
+fail the first time.  Simply re-run `git commit` — the second attempt should
+succeed.  Investigate further only if it still fails.
+
+## Test infrastructure
+
+### pytest markers
+
+- `@pytest.mark.integration` — tests requiring a running archive instance
+- `@pytest.mark.obolibrary` — tests hitting the OBO ontology library
+- `@pytest.mark.flaky` — known-flaky tests
+- `@pytest.mark.ai_generated` — **mandatory** on any test written with AI
+  assistance
+
+New markers must be registered in `pytest_configure()` in
+`dandi/pytest_plugin.py`.
+
+### Key fixtures (`dandi/tests/fixtures.py`)
+
+- `simple1_nwb_metadata()` / `simple1_nwb()` — session-scoped sample NWB file
+- `local_dandi_api` — Docker-based local DANDI Archive instance
+- `new_dandiset()` — creates a fresh dandiset on the test instance
+- `publish_dandiset()` — publishes a dandiset version
+- `capture_all_logs` — autouse; sets DEBUG level for `dandi` logger
+
+### Test organization
+
+- Tests mirror the module structure: `test_download.py`, `test_upload.py`, etc.
+- Integration tests use the `local_dandi_api` fixture
+- `--dandi-api` flag: run only integration tests
+- `--scheduled` flag: enable configuration for scheduled daily runs
+- VCR (vcrpy) records/replays HTTP interactions; disable with
+  `DANDI_TESTS_NO_VCR`
+
+### pytest configuration (`tox.ini [pytest]`)
+
+- Default timeout: 300 s per test
+- `--tb=short --durations=10`
+- `filterwarnings = error` with specific ignores for known third-party warnings
+
+## CI/CD
+
+- `run-tests.yml` — full test matrix: Python 3.10–3.13 × Ubuntu,
+  macOS (M1 + Intel), Windows
+- `lint.yml` — codespell + flake8
+- `typing.yml` — mypy
+- `docs.yml` — Sphinx build
+- `release.yml` — automated release via `auto` (see below)
+
 ## Environment variables
 
 - `DANDI_DEVEL` -- enables otherwise hidden command line options, such as