Skip to content

darnit plan and darnit validate crash on openssf-baseline: controls mix level=None and int #568

Description

@mlieberman85

darnit plan and darnit validate crash on the shipped openssf-baseline framework (reproduced on main 06e905e).

$ darnit plan
  File ".../darnit/cli.py", line 425, in cmd_plan
    for level in sorted(by_level.keys()):
TypeError: '<' not supported between instances of 'int' and 'NoneType'

$ darnit validate packages/darnit-baseline/src/darnit_baseline/openssf-baseline.toml
    logger.info(f"  By level: {', '.join(f'L{k}={v}' for k, v in sorted(by_level.items()))}")
TypeError: '<' not supported between instances of 'int' and 'NoneType'

Cause: both commands group controls by ControlConfig.level. openssf-baseline declares levels in tags.level (tags = { level = 1, ... }), so 65 of its 66 controls have level = None. One control, STAGE1-REF-SECURITY-01, sets the top-level level, so the keys mix int and None and sorted() raises. darnit plan -f community-spec works because none of its controls set level.

Same root cause, no crash: FrameworkConfig.get_controls_by_level(n) filters on ControlConfig.level, so it returns nothing for the 65 baseline controls that use tags.level. It has no production caller today. The audit path reads the level some other way and is not affected; a fix should check that.

Suggested direction:

  1. Give a control's level one source. Either ControlConfig.level falls back to tags.level, or the shipped TOMLs move to the top-level field. The spec (framework-design.md) should say which one is canonical.
  2. Make plan and validate sort with None last instead of crashing.
  3. Add a test that runs plan and validate against every shipped framework TOML.

Found while removing dead code for #487 (#566).

Drafted with Claude Code; reviewed and posted by me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions