[krishanth7]

Summary

This pull request implements Phase 5.5: Metadata Normalization & Identity for DERA (Database Engineering & Reverse Analysis). It establishes DERA's canonical metadata contracts and deterministic object identity layer in packages/core/src/dera/metadata/, transforming source metadata from PostgreSQL, Supabase, SQLite, MySQL, and MariaDB into strongly typed, immutable canonical models with cryptographic stable object identifiers (ObjectId).

Change Type

• Feature (feat)
• Bug fix (fix)
• Documentation (docs)
• Refactor (refactor)
• Performance (perf)
• Test improvement (test)
• CI/CD or Build (ci, build)
• Security enhancement (security)
• Breaking change (!)

Related Issues

Closes #41
Refers to Phase 5.5 Engineering Implementation Directive

Implementation Details

• Established canonical metadata contracts version 1.0.0 in packages/core/src/dera/metadata/contracts.py.
• Defined 17 canonical data type categories in enums.py alongside native type declarations and attributes.
• Implemented cryptographic stable object identity generation in identity.py using length-prefixed SHA-256 digests (dera:<kind>:<24_hex_digits>) to eliminate delimiter collision attacks.
• Created engine-specific metadata adapters in adapters/ for PostgreSQL, SQLite, MySQL, and MariaDB.
• Implemented reference integrity validation and duplicate detection in validation.py.
• Implemented deterministic multi-level sorting in ordering.py and canonical UTF-8 JSON serialization in serialization.py.
• Authored 14 architecture specifications in docs/metadata/ and completed the Phase 5.5 baseline and acceptance audits.

Testing

• Relevant tests passed (python scripts/development/dev.py all or test)
• Tests not applicable — explanation below
• Tests not run / blocked — explanation below

Test evidence & notes:

• Executed 39 metadata unit, identity, contract, cross-engine, security, integration, and golden file tests in tests/metadata/:

  Ran 39 tests in 0.009s
  OK
  

• Full repository test suite passed with 534 tests:

  Ran 534 tests in 66.064s
  OK (skipped=2)
  

• Documentation linting (npm run docs:lint) passed with 0 issues.
• Documentation link verification (python scripts/ci/validate_docs.py) passed.

Documentation Impact

• Documentation in docs/ updated or verified where required
• Documentation validation passed (npm run docs:check)
• No documentation updates required for this change

Security Impact & Deterministic Engineering

• Strict Non-AI Policy: Verified zero generative AI, LLM, or external inference APIs are introduced.
• Database Safety: Verified all database connector operations enforce read-only introspection without data modification.
• Credential Protection: Confirmed that no production credentials, passwords, or .env secrets are committed.
• Least Privilege: Validated that permissions and access boundaries are minimized.
• No Security Impact: Change does not affect database connections, security boundaries, or credentials.

Security Assessment Note:

• Enforces strict input validation, length limits on identifiers (1,024 chars), definition size caps (1MB), and max entity counts (100k).
• Stable object IDs exclude database credentials and connection secrets.
• Deserialization and hashing are completely deterministic with zero arbitrary code execution.

Breaking Changes

• No breaking changes.
• Breaking change introduced (detailed explanation and migration path below):

Contributor Checklist

• My code adheres to the project's coding standards (PEP 8, TypeScript strict mode).
• My branch is rebased on the latest main branch.
• The PR title follows Conventional Commits (e.g., feat(connectors): add SQLite metadata support).
• No unrelated changes or accidental files are included.

Summary by CodeRabbit

• New Features

  • Added a metadata normalization capability that converts database inspection reports into a consistent format across PostgreSQL, SQLite, MySQL, and MariaDB.
  • Added stable object identities, deterministic ordering and serialization, metadata completeness indicators, and validation for references and size limits.
  • Added database metadata report support and lookup helpers for schemas, tables, and columns.

• Documentation

  • Added guides covering metadata formats, engine compatibility, identity, security, and usage.

• Tests

  • Added cross-engine and integration coverage. The audit reports 39 metadata tests and 534 monorepo tests passing, with two skipped.

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

This pull request adds a canonical metadata subsystem. It defines connector reports and frozen metadata contracts, normalizes reports from four database engines, validates and serializes canonical databases, and adds tests and documentation.

Changes: Canonical metadata pipeline

Layer / File(s)	Summary
Report and canonical metadata contracts packages/connectors/src/dera_connectors/metadata.py, packages/core/src/dera/metadata/models/*, packages/core/src/dera/metadata/{enums,errors,provenance,contracts}.py, packages/core/src/dera/metadata/__init__.py, packages/core/src/dera_metadata/__init__.py, tests/metadata/contracts/*	Adds the connector report model and frozen canonical models for database objects, constraints, provenance, and metadata types. Exports these contracts through the package APIs and tests model immutability and database lookup.
Engine adapters and identity helpers packages/core/src/dera/metadata/adapters/*, packages/core/src/dera/metadata/{capabilities,identifiers,identity}.py, tests/metadata/{cross_engine/test_type_normalization.py,identity/*,unit/test_capabilities.py,unit/test_identifiers.py}	Adds engine-specific type, generation-mode, and index parsers for PostgreSQL, SQLite, MySQL, and MariaDB. Adds capability assessment, identifier parsing, stable object IDs, and synthetic constraint names, with focused tests.
Normalization, validation, and deterministic output packages/core/src/dera/metadata/{normalization,ordering,validation,serialization}.py, tests/metadata/{cross_engine/test_cross_engine_normalization.py,integration/*,unit/test_ordering.py,unit/test_serialization.py,golden/*,security/*}	Normalizes connector reports into canonical databases, orders metadata collections, validates references and size limits, and provides canonical JSON serialization and hashing. Tests exercise cross-engine normalization, integration, ordering, serialization, identities, and validation boundaries.
Subsystem guides and acceptance records docs/metadata/*.md, docs/audit/PHASE_5_5_METADATA_*.md, docs/audit/README.md	Adds metadata architecture, contract, engine compatibility, identity, security, and testing documentation. Adds Phase 5.5 baseline and acceptance audits and lists both in the audit index.

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant ConnectorReport
  participant MetadataNormalizer
  participant EngineAdapter
  participant CanonicalDatabase
  participant MetadataValidator
  participant CanonicalSerializer
  ConnectorReport->>MetadataNormalizer: provide metadata report
  MetadataNormalizer->>EngineAdapter: classify engine-specific metadata
  EngineAdapter-->>MetadataNormalizer: return canonical classifications
  MetadataNormalizer->>CanonicalDatabase: assemble and order canonical objects
  MetadataNormalizer->>MetadataValidator: validate references and limits
  MetadataValidator-->>MetadataNormalizer: return validation result
  MetadataNormalizer-->>ConnectorReport: return canonical database
  ConnectorReport->>CanonicalSerializer: serialize database or compute hash
  CanonicalSerializer-->>ConnectorReport: return JSON or SHA-256 digest

Loading

Possibly related PRs

• krishanth7/dera#46: Adds MySQL and MariaDB connector metadata consumed by this PR’s canonical adapters.
• krishanth7/dera#44: Adds PostgreSQL and Supabase connector metadata consumed by this PR’s normalization pipeline.
• krishanth7/dera#45: Adds SQLite connector metadata normalized by this PR’s SQLite adapter.

Merge Risk: Moderate · up to 11b3e

The new metadata normalizer can fail on valid schemas or produce unstable fingerprints. This affects case-distinct names, unnamed CHECK constraints, MySQL boolean and generated-column handling, and repeatability of the canonical hash. Resolve these before downstream phases rely on the output.

Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}