Skip to content

chore: fold darnit-example's test fixtures into darnit-testchecks and remove it (#487 part 3) - #570

Merged
Marc-cn merged 4 commits into
darnitdevorg:mainfrom
mlieberman85:487-fold-darnit-example
Oct 9, 2026
Merged

Marc-cn merged 4 commits into
darnitdevorg:mainfrom
mlieberman85:487-fold-darnit-example

Conversation

@mlieberman85

Copy link
Copy Markdown
Contributor

Summary

Part 3 of #487, the last one. darnit-example is removed, and the custom step types core tests used from it now live in darnit-testchecks. That leaves one plugin template (darnit-hello) and one test-only plugin (darnit-testchecks).

Why remove darnit-example:

  • It was never published (packaging/README.md lists it as internal).
  • Its framework TOML sat outside src/ and was resolved via Path(__file__), against feature 021, so a wheel would not include it.
  • Its seven remediations used a legacy table format the executor never runs.
  • Its MCP tools and remediation/actions.py were only exercised by its own tests.

What moved:

  • New test framework testchecks-steps. It has its own TOML inside src/darnit_testchecks/ and its own implementation. It holds the three custom step types, renamed testchecks_readme_description, testchecks_readme_quality and testchecks_ci_config so a test plugin doesn't take generic names. They register through register_handlers() with their settings and no ceiling, so their results are evidence only. Two controls use them: TCS-DOC-01 and TCS-CI-01.
  • Why a second framework: the CLI end-to-end and harness tests audit testchecks. A control that only reaches manual review would change their pass counts and exit codes, so testchecks keeps its 12 controls unchanged.
  • Core tests that needed a plugin with custom step types now use testchecks-steps: strict loading, expression references, registry metadata, handler-registration protocol, composition, and factory. The example's handler tests moved to tests/darnit_testchecks/ (git mv), and there are new implementation tests.
  • Feature 021 for testchecks: testchecks.toml moved into src/darnit_testchecks/ and resolves via importlib.resources. The built wheel contains both TOMLs. Its two legacy remediation tables, which never ran, are gone.

Deleted:

  • packages/darnit-example, and the tests that only covered its dead code: remediation actions, MCP tool registration, example-only implementation tests;
  • scripts/create-example-test-repo.py, which only built repositories for the example framework;
  • docs/architecture/example-plugin.md.

The docs now point plugin authors at darnit-hello, and the CHANGELOG notes the removal. It isn't marked BREAKING, since the package was internal. uv.lock only drops darnit-example. framework-design.md had no references to it, so there's no spec change.

With parts 1 (#566) and 2 (#569), every item in #487 is done.

Closes #487

Type of Change

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)

Framework Changes Checklist

  • Updated framework spec (docs/architecture/framework-design.md) if behavior changed (no reference to change)
  • Ran uv run python scripts/validate_sync.py --verbose and it passes

Testing

  • Tests pass locally (uv run pytest tests/ -v): 5138 passed, 26 skipped (main: 5150)
  • Added tests for new functionality (if applicable)
  • Linting passes (uv run ruff check .)

The net change of −12 tests:

  • +13 for the testchecks implementation tests, and for testchecks-steps entries in the tests that run over every plugin;
  • −24 for the example's own tests (17 of them only covered dead code) and its entries in those same tests;
  • −1 because a test that runs once per docs file lost example-plugin.md.

Also run:

  • integration tests with GitHub Actions environment variables set: 334 passed;
  • darnit list shows testchecks and testchecks-steps and nothing named example;
  • darnit audit . -f testchecks exits 0;
  • darnit plan and darnit validate work on both testchecks TOMLs;
  • uv build --package darnit-testchecks: the wheel contains both TOMLs and the entry points.

AI assistance

  • No AI assistance was used
  • AI assistance was used

Claude (Claude Code, claude-opus-5-5) made this change: fixture move, removals, tests and docs. This description was also drafted with Claude. Commits carry an Assisted-by: Claude:claude-opus-5-5 trailer.

Additional Notes

Follow-ups found across #487 but outside its scope:

  • Attestation: the predicate still writes adapters_used: ["builtin"]. Consumers can see that field, so changing it is a spec change.
  • Unread fields: EffectiveControl.remediation_handler and remediation_config.
  • Dead hooks: the core/plugin.py comment lists get_check_handlers, get_context_handlers and get_remediation_handlers, which nothing calls.
  • server/factory.py has its own copy of the handler-registration helper; it could delegate.
  • Feature 021: darnit-hello resolves its TOML with Path(__file__) instead of importlib.resources.
  • Generated data: docs/threatmodel/raw-findings.json still names deleted files. The rendered threat-model docs have removal notes.

🤖 Generated with Claude Code

…via importlib.resources

The framework TOML sat beside src/, so a wheel did not contain it, and
get_framework_path() climbed out of the package with Path(__file__)
(feature 021). It now lives in src/darnit_testchecks/ and resolves with
importlib.resources. The two legacy remediation tables (handler =
"create_readme" / "create_gitignore") named handlers that do not exist and
the executor never ran them; they are removed.

Refs darnitdevorg#487

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
…nit-testchecks (darnitdevorg#487)

darnit-testchecks gains a second framework, testchecks-steps, and the
implementation that registers its three step types in register_handlers()
(testchecks_readme_description, testchecks_readme_quality,
testchecks_ci_config, moved from darnit-example's handlers.py and renamed
so they cannot collide with a real plugin's names). The trivial testchecks
framework is unchanged, so the CLI and harness tests that audit it keep
their results. Both TOMLs ship inside src/darnit_testchecks/ (feature 021).

The core tests that used darnit-example as the plugin with custom step
types (strict loading, expr references, registry metadata, the fresh
registry load, composition, protocol naming) now use testchecks-steps.
The handler tests move to tests/darnit_testchecks/. testchecks is now an
implementation package, so the module-path policy test uses yaml as the
installed package that is not one.

Refs darnitdevorg#487

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
darnit-hello is the plugin template and darnit-testchecks the test plugin.
darnit-example was neither a working template (its TOML sat outside src/
and was resolved with Path(__file__), so a wheel would not contain it, and
its seven remediations used a legacy table format the executor never ran)
nor published. Its step-type fixtures moved to darnit-testchecks in the
previous commit.

Removed with it: tests/darnit_example (the remediation-action and MCP-tool
tests exercised only the package's dead code) and
scripts/create-example-test-repo.py, which only built repositories for the
example-hygiene MCP server.

Refs darnitdevorg#487

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
…arnitdevorg#487)

The implementation and handler-authoring guides, packaging-plugins, and the
darnit-hello README now name darnit-hello as the template and
darnit-reproducibility / darnit-gittuf as plugins with custom step types.
darnit-testchecks describes itself as the test-only plugin. The
example-plugin architecture page described only darnit-example and is
removed. Threat-model files note the removed files without regenerating.

Refs darnitdevorg#487

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>

@Marc-cn Marc-cn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on a clean RHEL 10 box at d902ed3.

  • Full suite: 5138 passed, 26 skipped (matches).
  • ruff and validate_sync pass.
  • darnit list: testchecks (12 controls) and testchecks-steps (2), nothing named example; same 8 frameworks / 6 implementations as main, with testchecks-steps in place of example-hygiene.
  • darnit audit . -f testchecks exits 0; testchecks-steps gives 2/2 WARN (manual), as intended with no ceiling.
  • Built wheel contains both TOMLs and the frameworks/implementations entry points.
  • Remaining darnit-example references are specs/ history or annotated threat-model notes.

Not run here: darnit plan/validate on the two TOMLs, and the integration suite with Actions env vars.

@Marc-cn
Marc-cn merged commit 190e85c into darnitdevorg:main Oct 9, 2026
8 checks passed
@mlieberman85
mlieberman85 deleted the 487-fold-darnit-example branch October 10, 2026 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove dead adapter system, legacy storage backend, unused protocol methods, and other unreachable code (~3,500 lines)

2 participants