Skip to content

77 data driven design tooling#78

Merged
lxsaah merged 46 commits intomainfrom
77-data-driven-design-tooling
Mar 11, 2026
Merged

77 data driven design tooling#78
lxsaah merged 46 commits intomainfrom
77-data-driven-design-tooling

Conversation

@lxsaah
Copy link
Copy Markdown
Contributor

@lxsaah lxsaah commented Mar 10, 2026

Overview

This PR introduces data-driven design tooling for AimDB — a conversational architecture agent that lets developers describe systems in natural language and have the agent resolve data flows into concrete AimDB primitives, code, and diagrams. It also adds a WebSocket connector, a WASM adapter for browser-based usage, and a code generation pipeline.


New Crates

Crate Purpose
aimdb-codegen Reads .aimdb/state.toml (architecture agent decisions) and emits Mermaid diagrams and compilable Rust source code. Includes validation.
aimdb-ws-protocol Shared WebSocket protocol types (messages, frames) used by both server and client sides.
aimdb-websocket-connector Full WebSocket connector with server, client, session management, authentication, and client manager.
aimdb-wasm-adapter WebAssembly runtime adapter for AimDB — includes JS/WASM bindings, WebSocket bridge, buffer implementation, schema registry, and React hooks (useAimDb).

Architecture Agent (MCP Server)

  • New architecture agent module in aimdb-mcp with session management, conflict resolution, and structured state tracking.
  • MCP tools for record proposals, task/binary proposals, and session management — enabling LLM-driven architecture design through conversation.
  • Architecture resources endpoint exposing state to MCP clients.
  • CONVENTIONS.md asset for agent behavior guidance.
  • Enhanced prompts module.

Code Generation CLI

  • New generate command in aimdb-cli that produces Mermaid diagrams and Rust schema source from the architecture state file.

Data Contracts & Migration

  • Refactored schema migration system: replaced Migratable trait with MigrationStep and MigrationChain for type-safe, bidirectional migrations.
  • Added Streamable trait for streaming data contracts.
  • Updated temperature contract to new migration pattern.
  • Removed legacy TypeScript export tests and generation script.

Other Changes

  • Updated README with new content.
  • Makefile updates for publishing sequence and formatting.
  • Embassy submodule updated.
  • Minor persistence error-handling improvement.
  • Design documents: 023 (Architecture Agent), 024 (Codegen Common Crate), 025 (WASM Adapter).

- Implemented `generate_rust` function to convert `ArchitectureState` into Rust source code.
- Added support for generating value structs and key enums for records.
- Introduced `configure_schema` function to register records with `AimDbBuilder`.
- Created utility functions for handling imports, field definitions, and key variants.
- Added tests to validate generated output against expected structures.

feat: define architecture state types and TOML parser

- Created `ArchitectureState`, `RecordDef`, and related structs to represent the state.
- Implemented TOML parsing and serialization for architecture state.
- Added support for metadata, buffer types, fields, and connectors.
- Included tests to ensure correct parsing and serialization of TOML data.

feat: implement architecture state validation

- Added `validate` function to check for structural and semantic errors in `ArchitectureState`.
- Defined `ValidationError` and `Severity` types for error reporting.
- Implemented checks for record names, field types, connector definitions, and more.
- Included tests to verify validation logic and error handling.
…proposals

- Implemented new tools for the architecture agent, including:
  - `get_architecture`: Retrieves the current architecture state.
  - `propose_add_record`: Proposes adding a new record with detailed schema.
  - `propose_modify_buffer`: Proposes changing the buffer type of an existing record.
  - `propose_add_connector`: Proposes adding a connector to an existing record.
  - `propose_modify_fields`: Proposes replacing fields of an existing record.
  - `propose_modify_key_variants`: Proposes updating key variants of an existing record.
  - `resolve_proposal`: Resolves a pending proposal with options to confirm, reject, or revise.
  - `remove_record`: Proposes the removal of an existing record.
  - `rename_record`: Proposes renaming an existing record.
  - `validate_against_instance`: Validates state against a live AimDB instance.
  - `get_buffer_metrics`: Retrieves live buffer metrics for records.
  - `save_memory`: Persists ideation context to memory.
  - `reset_session`: Resets the architecture session.

- Added a new module `architecture.rs` to encapsulate the architecture agent tools.
- Updated `mod.rs` to include the new architecture tools and re-export them for use.
- Enhanced session management by initializing the session store and handling state transitions.
- Added validation for schema_version to ensure it is >= 1.
- Introduced warnings for records with settable fields lacking a timestamp field.
- Implemented validation for observable signal fields to ensure they exist and are numeric.
- Updated the generate command to support a common crate mode, producing a complete crate structure.
- Enhanced TOML schema to include project metadata and record-level serialization options.
- Added tests for new validation rules and common crate generation.
- Documented the design for extending aimdb-codegen to generate common crates.
- Introduced `tasks` and `binaries` fields in `ArchitectureState` struct.
- Added `TaskDef`, `TaskIo`, `TaskType`, `BinaryDef`, and `ExternalConnectorDef` structs for task and binary definitions.
- Implemented validation for tasks and binaries in `validate.rs`, ensuring proper references and existence checks.
- Enhanced the CLI with a new `--hub` option to generate a hub binary crate scaffold, including `Cargo.toml`, `main.rs`, and `tasks.rs`.
- Updated default paths to be relative to `AIMDB_WORKSPACE` for better project structure management.
…n handling

- Add ClientManager for tracking connected WebSocket clients and their subscriptions.
- Implement WebSocketConnector for managing outbound publishing and routing.
- Create protocol definitions for server-client communication.
- Develop server functionality using Axum for WebSocket upgrades and health checks.
- Establish session management for individual WebSocket connections, including message handling and subscription management.
- Introduce snapshot functionality for late-join clients.
- Implement topic matching for subscription patterns.
- Add tests for client registration, subscription, and broadcasting.
- Introduced a new design document for the AimDB WASM adapter, detailing its architecture, motivation, and implementation plan.
- Outlined the benefits of running the full dataflow engine in the browser, including native contract enforcement and offline capabilities.
- Provided a comprehensive overview of the crate layout, dependency graph, and feature flags.
- Included detailed sections on trait implementations, buffer design, TypeScript bindings, and WebSocket sync strategies.
- Established a clear migration path for integrating the WASM adapter with existing AimDB UI components.
- Added `WsClientConnectorBuilder` for building WebSocket client connections.
- Implemented `WsClientConnectorImpl` to manage WebSocket connections, including inbound routing, outbound publishing, reconnection logic, and keepalive pings.
- Introduced shared wire protocol types in `aimdb-ws-protocol` for communication between clients and servers.
- Defined `ServerMessage` and `ClientMessage` enums for structured message handling.
- Implemented topic matching logic to support MQTT-style wildcards.
- Added tests for message serialization and topic matching functionality.
…grationStep` and `MigrationChain` for type-safe, bidirectional migrations

- Updated `lib.rs` to expose new migration types and modify documentation accordingly.
- Enhanced `migratable.rs` with a new architecture for migration steps and chains, including detailed examples and improved error handling.
- Removed obsolete TypeScript export test and associated scripts, streamlining the codebase.
@lxsaah lxsaah requested a review from Copilot March 10, 2026 20:49
@lxsaah lxsaah self-assigned this Mar 10, 2026
@lxsaah lxsaah linked an issue Mar 10, 2026 that may be closed by this pull request
5 tasks
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds data-driven design tooling across the workspace: an MCP “architecture agent” resource surface, a codegen pipeline + CLI command to emit Mermaid/Rust from .aimdb/state.toml, plus a new WebSocket protocol/connector and a WASM adapter for browser usage.

Changes:

  • Added MCP architecture resources + session/conflict tracking infrastructure in aimdb-mcp.
  • Introduced aimdb-codegen and wired aimdb-cli generate to validate and emit diagrams/Rust/crate scaffolds.
  • Added WebSocket protocol + connector crates and a browser-focused aimdb-wasm-adapter (runtime, buffers, bindings, React hooks).

Reviewed changes

Copilot reviewed 64 out of 68 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
tools/aimdb-mcp/src/resources/mod.rs Routes resource listing/reading across instance + new architecture resources
tools/aimdb-mcp/src/resources/architecture.rs Adds aimdb://architecture/* resource descriptors and readers
tools/aimdb-mcp/src/lib.rs Exposes new architecture module
tools/aimdb-mcp/src/architecture/session.rs Implements session state machine + tests
tools/aimdb-mcp/src/architecture/mod.rs Adds state I/O helpers, proposal model, change application
tools/aimdb-mcp/src/architecture/conflicts.rs Adds instance-vs-state conflict detection + tests
tools/aimdb-mcp/assets/CONVENTIONS.md Adds canonical diagram conventions content to embed/expose
tools/aimdb-mcp/Cargo.toml Adds aimdb-codegen + fs2 deps for architecture tooling
tools/aimdb-cli/src/main.rs Adds generate subcommand wiring
tools/aimdb-cli/src/commands/mod.rs Registers new generate command module
tools/aimdb-cli/src/commands/generate.rs Implements aimdb generate (validate + emit outputs)
tools/aimdb-cli/Cargo.toml Adds aimdb-codegen dependency for generation
scripts/gen-ts-bindings.sh Removes legacy TS export script
aimdb-ws-protocol/src/lib.rs Introduces shared WS wire types + topic matching + tests
aimdb-ws-protocol/Cargo.toml Adds new protocol crate manifest
aimdb-websocket-connector/src/session.rs Implements per-connection WS session (send/recv/query/list_topics)
aimdb-websocket-connector/src/server.rs Adds Axum WS server + health endpoint
aimdb-websocket-connector/src/protocol.rs Re-exports shared WS protocol crate for compatibility
aimdb-websocket-connector/src/lib.rs Adds crate public surface and feature-gated modules
aimdb-websocket-connector/src/connector.rs Implements connector trait + outbound publisher tasks
aimdb-websocket-connector/src/client_manager.rs Adds client registry + subscription fan-out + tests
aimdb-websocket-connector/src/client/mod.rs Adds WS client connector module entrypoint
aimdb-websocket-connector/src/client/builder.rs Adds builder implementing ConnectorBuilder for ws-client scheme
aimdb-websocket-connector/src/builder.rs Adds server connector builder + topic discovery support
aimdb-websocket-connector/src/auth.rs Adds auth hooks + permissions model
aimdb-websocket-connector/Cargo.toml Adds connector crate manifest + features
aimdb-wasm-adapter/src/time.rs Adds WASM TimeOps implementation using performance.now + timeout sleep
aimdb-wasm-adapter/src/schema_registry.rs Adds runtime schema dispatch registry for Streamable types
aimdb-wasm-adapter/src/runtime.rs Adds WasmAdapter runtime/spawn implementation
aimdb-wasm-adapter/src/react/useAimDb.tsx Adds React provider + hooks for WASM AimDB usage
aimdb-wasm-adapter/src/logger.rs Adds console-backed logger for WASM
aimdb-wasm-adapter/src/lib.rs Adds crate root exports + registrar ext macro usage
aimdb-wasm-adapter/src/buffer.rs Adds single-threaded WASM buffer implementations + cancel token/handle
aimdb-wasm-adapter/README.md Adds adapter documentation + usage guidance
aimdb-wasm-adapter/LICENSE Adds Apache-2.0 license file for the crate
aimdb-wasm-adapter/Cargo.toml Adds WASM adapter manifest + wasm-runtime feature gating
aimdb-persistence/src/ext.rs Improves persistence loop error handling for BufferLagged
aimdb-data-contracts/tests/export_ts.rs Removes legacy TS export test
aimdb-data-contracts/src/streamable.rs Adds Streamable + visitor registry for cross-boundary dispatch
aimdb-data-contracts/src/lib.rs Exposes Streamable APIs + migratable trait refactor exports/docs
aimdb-data-contracts/src/contracts/temperature.rs Migrates temperature contract to typed migration chain + adds tests
aimdb-data-contracts/.gitignore Removes bindings ignore entry for deleted TS export pipeline
aimdb-core/src/builder.rs Adds outbound topic TypeId discovery API for connectors
aimdb-codegen/src/mermaid.rs Adds Mermaid generator + tests
aimdb-codegen/src/lib.rs Adds codegen public API + re-exports
aimdb-codegen/Cargo.toml Adds codegen crate manifest
_external/embassy Updates embassy submodule revision
README.md Updates top-level project documentation/positioning
Makefile Adds build/test/docs/publish steps for new crates + WASM targets
Cargo.toml Adds new workspace members
.vscode/mcp.json Adds AIMDB_WORKSPACE env for MCP server
.devcontainer/Dockerfile Adds wasm32 target to dev container toolchain

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tools/aimdb-mcp/src/resources/architecture.rs
Comment thread tools/aimdb-mcp/src/resources/architecture.rs Outdated
Comment thread tools/aimdb-mcp/src/resources/architecture.rs Outdated
Comment thread tools/aimdb-mcp/src/resources/architecture.rs Outdated
Comment thread tools/aimdb-mcp/src/resources/architecture.rs
Comment thread aimdb-wasm-adapter/src/time.rs
Comment thread tools/aimdb-cli/src/commands/generate.rs
Comment thread tools/aimdb-cli/src/commands/generate.rs Outdated
Comment thread aimdb-codegen/src/mermaid.rs Outdated
Comment thread tools/aimdb-mcp/src/resources/architecture.rs Outdated
@lxsaah lxsaah merged commit 323f381 into main Mar 11, 2026
12 checks passed
@lxsaah lxsaah deleted the 77-data-driven-design-tooling branch March 11, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Data-Driven Design Tooling

2 participants