Skip to content

Implement severity-based drift classification and tiered documentation#7

Merged
justinemach merged 2 commits into
mainfrom
devin/1771888783-severity-tiers
Feb 24, 2026
Merged

Implement severity-based drift classification and tiered documentation#7
justinemach merged 2 commits into
mainfrom
devin/1771888783-severity-tiers

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Feb 23, 2026

Copy link
Copy Markdown
Contributor

Add severity-based drift classification and tiered documentation

Summary

Refines the Drift Guard system to classify all detected API drift by severity and adjust behavior accordingly:

  • drift_detect.py: Rewritten with Severity enum (GREEN/YELLOW/RED), DriftEvent dataclass, and enhanced DriftReport. Schema comparison now tracks field types, required flags, and enum values to classify changes (e.g., new optional field = GREEN, type change = YELLOW, removed endpoint = RED).
  • guard.py: New exit codes (10=GREEN, 11=YELLOW, 2=RED). cmd_check regenerates Tier 1 docs always, appends a "Proposed Narrative Updates" review section for YELLOW/RED, and only sends Slack for YELLOW/RED. New _build_runtime_signals() groups repeated audit mismatches for human review.
  • slack_notify.py: Accepts optional severity param; adds "Action required" block with severity-specific framing (RED = merge blocked, YELLOW = PR review required).
  • pre-commit hook: GREEN/YELLOW → auto-regenerate baseline + docs and stage them (commit proceeds). RED → block commit with instructions.
  • CI workflow: GREEN → pass (auto-update baseline). YELLOW → fail, require PR review. RED → fail + escalate.

Review & Testing Checklist for Human

  • Severity classification correctness: The detect_drift() function in drift_detect.py was fully rewritten. Verify the classification rules match your expectations — especially edge cases like allOf schemas, $ref resolution, and fields with defaults being classified as optional vs required. No unit tests were added for the new classification logic.
  • Exit code migration: Old behavior used exit code 2 for any drift. Now 10=GREEN, 11=YELLOW, 2=RED only. Verify no external scripts or tooling depend on exit code 2 meaning "any drift detected."
  • cmd_check updates baseline even for RED drift (guard.py ~line 220): After detecting RED/breaking drift, the baseline is overwritten. This means running check twice in a row yields "no drift" on the second run. Confirm this is intended — it means the RED warning is a one-shot signal.
  • Pre-commit auto-staging: The hook now runs git add artifacts/openapi_baseline.json docs/api_reference.md automatically for GREEN/YELLOW changes. Verify you're comfortable with the hook modifying staged files silently.
  • _find_endpoints_using_schema() uses string matching (drift_detect.py ~line 189): It converts the operation dict to a string and checks if the schema name appears. This could produce false positives (e.g., schema "User" matching a description mentioning "User"). Test with schemas that have common names.

Test Plan

  1. GREEN drift: Add an optional field to a model, stage the change, run git commit. Verify the hook auto-updates baseline + docs and allows the commit.
  2. YELLOW drift: Change a field type (e.g., strint), stage, commit. Verify the hook auto-updates but the CI fails with "PR review required."
  3. RED drift: Remove an endpoint or required field, stage, commit. Verify the hook blocks the commit with a clear error message.
  4. Slack: Set SLACK_WEBHOOK_URL and trigger YELLOW/RED drift. Verify the Slack message includes severity badge and action-required text. Verify GREEN drift does NOT send Slack.
  5. Runtime signals: Run bash scripts/demo_bad_client.sh multiple times to generate repeated mismatches, then python -m drift_guard.guard audit. Verify the "Runtime Drift Signals" section appears with counts.

Notes

  • Backward compatibility: Legacy added_endpoints, removed_endpoints, schema_changes lists are still populated for existing consumers.
  • No test coverage was added for the new severity classification logic — recommend adding unit tests for detect_drift() edge cases.
  • Link to Devin run: https://app.devin.ai/sessions/23d1e596e5f845ec8c47886363e9b2b8
  • Requested by: @justinemach

Open with Devin

- Add Severity enum (GREEN/YELLOW/RED) and DriftEvent dataclass to drift_detect.py
- Classify all drift changes with severity, human-readable explanation, impacted endpoints
- New exit codes: 0=no drift, 10=GREEN, 11=YELLOW, 2=RED, 3=runtime mismatches
- Pre-commit hook: auto-fix GREEN/YELLOW drift, only block RED (breaking)
- CI pipeline: pass GREEN, fail YELLOW (require review), fail+escalate RED
- Slack notifications: severity badges, action-required text, only for YELLOW/RED
- Tier 1 docs (api_reference.md) regenerated deterministically
- Tier 2/3 review section appended for YELLOW/RED changes
- Runtime audit: surface repeated mismatches as Runtime Drift Signals
- Backward compatible with existing baseline format

Co-Authored-By: unknown <>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

devin-ai-integration[bot]

This comment was marked as resolved.

@justinemach justinemach merged commit c510c0d into main Feb 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant