Skip to content

Integrate audit findings into docs/api_reference.md#8

Merged
justinemach merged 1 commit into
mainfrom
devin/1771890749-audit-in-api-reference
Feb 23, 2026
Merged

Integrate audit findings into docs/api_reference.md#8
justinemach merged 1 commit into
mainfrom
devin/1771890749-audit-in-api-reference

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Feb 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a "Runtime Mismatches" section to docs/api_reference.md so developers see audit findings alongside the API reference instead of in a separate artifacts/audit_report.md file.

When python -m drift_guard.guard audit runs:

  1. It still writes artifacts/audit_report.md (unchanged, for CI artifact uploads)
  2. New: It also injects a ## Runtime Mismatches section into docs/api_reference.md, inserted just before the auto-generated footer line
  3. The section includes: mismatch count, a table (Method, Path, Issue Type, Details, Client IP), and an audit timestamp
  4. Running audit again replaces the previous section (no stale accumulation)
  5. Running baseline or check fully regenerates api_reference.md from the OpenAPI spec, which naturally clears the audit section

No changes to openapi_to_md.py — the clearing behavior is implicit because openapi_to_markdown() rewrites the entire file.

Review & Testing Checklist for Human

  • Footer marker coupling: _AUTOGEN_FOOTER_PREFIX = "_This file was auto-generated by" is hardcoded in guard.py but the actual footer is generated in openapi_to_md.py (line 188). If the footer wording ever changes in one place but not the other, the audit section silently falls through to appending at the end of the file instead of inserting before the footer. Verify these stay in sync — consider whether they should share a constant.
  • Section replacement with str.index(): The logic to strip an existing audit section finds "## Runtime Mismatches" by string search, then looks for the footer after it. If api_reference.md contains that exact heading in a different context (unlikely but possible), it would incorrectly strip content. Review the stripping logic in _append_audit_section_to_api_ref (lines 220–231 of the new code) for edge cases.
  • Dead code path: _append_audit_section_to_api_ref handles total == 0, but cmd_audit exits with sys.exit(0) before calling it when there are no mismatches. The zero-mismatch branch is unreachable.
  • Pipe escaping is inconsistent: details field escapes | characters for the markdown table, but method, path, issue_type, and client_ip do not. Probably fine in practice but worth noting.

Test Plan

  1. Start server: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
  2. Clear audit log: curl -X DELETE http://localhost:8000/v1/audit
  3. Make bad calls: curl -X POST http://localhost:8000/v1/users -H "Content-Type: application/json" -d '{"name": "Test", "email": "t@t.com", "age": 30}'
  4. Run audit: python -m drift_guard.guard audit
  5. Verify docs/api_reference.md has a ## Runtime Mismatches section with the table before the footer
  6. Run audit again: verify the section is replaced (not duplicated)
  7. Run baseline: python -m drift_guard.guard baseline
  8. Verify the audit section is gone from docs/api_reference.md

Notes


Open with Devin

- cmd_audit now appends a Runtime Mismatches section to docs/api_reference.md
  with summary count, table (Method, Path, Issue Type, Details, Client IP),
  and audit timestamp
- Section is inserted before the auto-generated footer line
- Running audit again replaces the previous section (no stale accumulation)
- Running baseline fully regenerates api_reference.md, clearing the audit section
- artifacts/audit_report.md kept as-is for CI artifact uploads

Co-Authored-By: unknown <>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@justinemach justinemach merged commit 20fa678 into main Feb 23, 2026
1 check passed

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

View 5 additional findings in Devin Review.

Open in Devin Review

Comment thread drift_guard/guard.py
Comment on lines +336 to +339
_append_audit_section_to_api_ref(entries, total, now)
console.print(
f" Appended Runtime Mismatches section to [cyan]{API_REF_PATH}[/cyan]"
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Misleading success message printed when api_reference.md does not exist

When API_REF_PATH does not exist, _append_audit_section_to_api_ref silently returns at line 215-216 without writing anything. However, cmd_audit unconditionally prints "Appended Runtime Mismatches section to docs/api_reference.md" at lines 337-339, misleading the user into thinking the section was successfully written.

Root Cause

The function _append_audit_section_to_api_ref has an early return when the file doesn't exist (drift_guard/guard.py:215-216), but the caller at drift_guard/guard.py:336-339 does not check whether the operation actually succeeded:

# Line 215-216: silently returns
if not API_REF_PATH.exists():
    return
# Lines 336-339: always prints success
_append_audit_section_to_api_ref(entries, total, now)
console.print(
    f"  Appended Runtime Mismatches section to [cyan]{API_REF_PATH}[/cyan]"
)

Impact: If a user runs audit before running baseline (which creates api_reference.md), they see a success message but the audit section is never written. This could cause confusion during CI or manual debugging.

Suggested change
_append_audit_section_to_api_ref(entries, total, now)
console.print(
f" Appended Runtime Mismatches section to [cyan]{API_REF_PATH}[/cyan]"
)
# Append Runtime Mismatches section to docs/api_reference.md
if API_REF_PATH.exists():
_append_audit_section_to_api_ref(entries, total, now)
console.print(
f" Appended Runtime Mismatches section to [cyan]{API_REF_PATH}[/cyan]"
)
else:
console.print(
f" [dim]Skipped updating {API_REF_PATH} (file not found)[/dim]"
)
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant