Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 46 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,22 +21,32 @@ policy enforcement, bounded recovery, and production-friendly delivery interface

---

QueryForge is a local-first AI data analytics platform built around one principle:
QueryForge is a local-first, domain-first AI data analytics platform built around
one principle:
**generated SQL should be governed like application code, not trusted like prose**.

It combines natural-language-to-SQL with semantic contracts, AST-level security,
read-only execution, multi-candidate selection, repair budgets, and complete run
artifacts. The result is a reference implementation for building AI data systems
that are useful, inspectable, and difficult to misuse.
Users create or select a data domain first—such as retail, finance, product, or
the bundled Anime Streaming sample—then onboard that domain's data, review its
semantic contract, and ask questions inside the same governance boundary.
QueryForge combines that workflow with natural-language-to-SQL, AST-level
security, read-only execution, multi-candidate selection, repair budgets, and
complete run artifacts.

> QueryForge currently targets SQLite and controlled environments. It is a
> portfolio-grade reference architecture, not a multi-tenant analytics service.

## Product Tour

<div align="center">
<img src="docs/assets/queryforge-studio-domains.png" width="100%" alt="QueryForge Data Domain Center for creating and selecting isolated business contexts">
<sub>Data Domain Center — create or select a governed context before adding data or semantics.</sub>
</div>

<br>

<div align="center">
<img src="docs/assets/queryforge-studio-overview.png" width="100%" alt="QueryForge Studio overview with semantic health, business metrics, and engagement trends">
<sub>Workspace overview — live semantic health, governed metrics, and anime-platform activity.</sub>
<sub>Domain overview — the Anime Streaming dataset is shown as one selected sample, not the platform identity.</sub>
</div>

<br>
Expand All @@ -57,6 +67,7 @@ delivery loop:
| --- | --- |
| Trust the generated SQL | Parse with SQLGlot and enforce a named policy before execution |
| Keep business meaning consistent | Define metrics, dimensions, grain, and join paths in YAML |
| Prevent context from leaking | Scope sources, semantic contracts, policies, and run history to a selected data domain |
| Recover from imperfect output | Reflect, repair, and retry within explicit budgets |
| Handle harder questions | Use bounded schema discovery and parallel SQL candidates |
| Trace what happened | Persist run state, policy decisions, quality evidence, and artifacts |
Expand Down Expand Up @@ -127,9 +138,10 @@ queryforge \

### Explore QueryForge Studio

The repository now includes a complete visual workspace for onboarding data,
reviewing the required semantic layer, asking governed questions, inspecting SQL
and Trust Trace evidence, and auditing run history.
The repository includes a complete visual workspace for creating and switching
data domains, onboarding domain-owned data, reviewing the required semantic
layer, asking governed questions, inspecting SQL and Trust Trace evidence, and
auditing domain-scoped run history.

```bash
# Terminal 1: QueryForge API
Expand All @@ -141,9 +153,10 @@ make web-install
make web-dev
```

Open <http://localhost:3000>. If the Python API is offline, the interface
automatically uses a deterministic demo response so every screen remains
explorable. See the [Studio guide](docs/studio.md).
Open <http://localhost:3000>. Start in **Data Domains**, select the bundled Anime
Streaming example or create a clean domain, then add data inside that domain. If
the Python API is offline, the sample-domain analysis remains explorable with a
deterministic demo response. See the [Studio guide](docs/studio.md).

## How It Works

Expand Down Expand Up @@ -217,6 +230,23 @@ QueryForge requires a validated model by default; it auto-discovers a model besi
the database and rejects schema-only analysis unless the caller explicitly selects
the diagnostic escape hatch.

In Studio, semantic construction is domain-first and gated:

```text
Create/select domain
→ upload domain-owned sources
→ profile physical schema
→ confirm entity identity and grain
→ define dimensions, measures, metrics, and time
→ review relationships, cardinality, and Join Paths
→ classify sensitivity, ownership, policy, and quality
→ validate 100% of blocking checks
→ publish data + semantics atomically
```

Technical names are treated as evidence, not business truth. A new domain starts
empty and never inherits the Anime sample's entities or metrics.

Build or incrementally refresh one:

```bash
Expand All @@ -235,9 +265,10 @@ The repository also includes a Monday-morning
metrics, relationships, Join Paths, and data-quality contracts against a reviewed
baseline.

### Anime platform showcase
### Bundled sample domain: Anime Streaming

The bundled dataset is purpose-built for QueryForge and fully synthetic: **370,762
Anime Streaming is one ready-to-run example data domain, not a product-wide
schema. The dataset is purpose-built for QueryForge and fully synthetic: **370,762
rows**, **15 tables**, **30 declared relationships**, **7 governed Join Paths**, and
**11 business metrics** across content, engagement, subscriptions, advertising,
community, and merchandise.
Expand Down Expand Up @@ -340,7 +371,7 @@ python -m queryforge.interfaces.mcp.server --transport stdio

| Surface | Entry point | Best for |
| --- | --- | --- |
| Studio | `make web-dev` | Visual onboarding, semantic authoring, and governed analysis |
| Studio | `make web-dev` | Data-domain management, visual onboarding, semantic authoring, and governed analysis |
| CLI | `queryforge --question "..."` | Local exploration and engineering workflows |
| REST | `POST /ask` and `POST /plan` | Application integration |
| SSE | `POST /ask/stream` | Progress-aware clients |
Expand Down
51 changes: 40 additions & 11 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,20 +21,28 @@

---

QueryForge 是一个本地优先的 AI 数据分析平台,它围绕一个核心原则设计:
QueryForge 是一个本地优先、以数据域为第一入口的 AI 数据分析平台,它围绕一个核心原则设计:
**模型生成的 SQL 应该像应用代码一样被治理,而不是像自然语言一样被直接信任。**

项目将 NL2SQL、语义契约、AST 级安全策略、只读执行、多候选选择、有界修复和完整
运行产物串成一个闭环,适合作为可信 AI 数据系统的参考实现。
用户先创建或选择数据域(如零售、金融、产品分析,或仓库内置的 Anime Streaming
示例域),再在该域内上传数据、评审语义契约和发起分析。项目把这套域级工作流与
NL2SQL、AST 级安全策略、只读执行、多候选选择、有界修复和完整运行产物串成闭环。

> QueryForge 当前专注 SQLite 和受控环境,是面向作品展示与架构验证的参考项目,
> 不是可直接公网部署的多租户分析服务。

## 产品界面

<div align="center">
<img src="docs/assets/queryforge-studio-domains.png" width="100%" alt="QueryForge 数据域中心:创建和选择相互隔离的业务上下文">
<sub>数据域中心——先创建或选择治理边界,再接入该域的数据与语义。</sub>
</div>

<br>

<div align="center">
<img src="docs/assets/queryforge-studio-overview.png" width="100%" alt="QueryForge Studio 总览:语义契约健康度、业务指标与互动趋势">
<sub>工作台总览——实时展示语义健康度、受治理指标和动漫平台业务动态。</sub>
<sub>数据域总览——Anime Streaming 只是当前选中的示例域,不再是平台身份。</sub>
</div>

<br>
Expand All @@ -54,6 +62,7 @@ QueryForge 是一个本地优先的 AI 数据分析平台,它围绕一个核
| --- | --- |
| 如何信任生成的 SQL | 使用 SQLGlot 解析 AST,执行前应用具名安全策略 |
| 如何保证业务口径一致 | 用 YAML 定义指标、维度、粒度和 Join Path |
| 如何防止上下文串域 | 数据源、语义契约、策略和运行历史全部绑定当前数据域 |
| 模型输出不完美怎么办 | 在明确预算内反思、修复和重试 |
| 复杂问题如何处理 | 启用有界 Schema 探索和并发 SQL 候选 |
| 如何追踪运行过程 | 保存状态、策略决策、质量证据和交付产物 |
Expand Down Expand Up @@ -119,8 +128,9 @@ queryforge \

### 体验 QueryForge Studio

仓库现在包含一个完整的可视化工作台:可以接入数据、评审强制语义层、提出受治理的
自然语言问题、检查 SQL 与 Trust Trace 证据,并审计历史运行。
仓库现在包含一个完整的可视化工作台:可以创建和切换数据域、在域内接入数据、评审
强制语义层、提出受治理的自然语言问题、检查 SQL 与 Trust Trace 证据,并审计域内
历史运行。

```bash
# 终端 1:QueryForge API
Expand All @@ -132,8 +142,9 @@ make web-install
make web-dev
```

打开 <http://localhost:3000>。Python API 离线时,界面会自动使用确定性的演示结果,
所有页面仍然可以完整体验。详见 [Studio 使用指南](docs/studio.md)。
打开 <http://localhost:3000>,从「Data Domains」进入:可选择内置 Anime Streaming
示例域,也可创建一个完全空白的新域,然后只在当前域内上传数据。Python API 离线时,
示例域分析会使用确定性的演示结果。详见 [Studio 使用指南](docs/studio.md)。

## 系统架构

Expand Down Expand Up @@ -203,6 +214,23 @@ metrics:
物理质量规则。QueryForge 现在默认强制使用经过校验的语义模型,并自动发现数据库旁边
的模型;除非显式启用诊断逃生口,否则不允许只依赖裸 Schema 查询。

Studio 中的语义层构建是「域优先 + 强门禁」流程:

```text
创建/选择数据域
→ 上传该域数据
→ Profiling 物理结构
→ 确认实体身份和行粒度
→ 定义维度、度量、指标和时间口径
→ 评审关系、基数和 Join Path
→ 标注敏感级别、Owner、策略和质量规则
→ 通过全部阻断性检查
→ 数据 + 语义原子发布
```

技术字段名只作为推断证据,不能替代业务定义。新建域从空白契约开始,绝不会继承
Anime 示例域的实体或指标。

创建或增量更新语义层:

```bash
Expand All @@ -219,9 +247,10 @@ python scripts/build_semantic_model.py \
[语义漂移检查](.github/workflows/semantic-weekly.yml),会基于审核后的基线检查
Schema、指标、关系、Join Path 和数据质量契约。

### 动漫平台数据集
### 内置示例数据域:Anime Streaming

内置样例是专为 QueryForge 构造的全合成数据:**370,762 行**、**15 张表**、
Anime Streaming 是一个开箱即用的示例数据域,不是全平台 Schema。其数据是专为
QueryForge 构造的全合成数据:**370,762 行**、**15 张表**、
**30 条声明关系**、**7 条受治理 Join Path**、**11 个业务指标**,覆盖内容、
观看、订阅、广告、社区和动漫周边。

Expand Down Expand Up @@ -321,7 +350,7 @@ python -m queryforge.interfaces.mcp.server --transport stdio

| 方式 | 入口 | 适合场景 |
| --- | --- | --- |
| Studio | `make web-dev` | 可视化数据接入、语义编写与受治理分析 |
| Studio | `make web-dev` | 数据域管理、可视化接入、语义编写与受治理分析 |
| CLI | `queryforge --question "..."` | 本地探索和工程工作流 |
| REST | `POST /ask`、`POST /plan` | 应用集成 |
| SSE | `POST /ask/stream` | 需要进度事件的客户端 |
Expand Down
Binary file modified docs/assets/queryforge-studio-analysis.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/queryforge-studio-domains.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/assets/queryforge-studio-overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/assets/queryforge-studio-semantic.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
54 changes: 38 additions & 16 deletions docs/studio.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# QueryForge Studio

QueryForge Studio is the browser interface for the governed analytics runtime.
It is designed to make the semantic layer visible and editable instead of
hiding it behind an opaque chat box.
It is designed around a domain-first rule: business context must be selected
before data, semantics, or analysis can exist. The semantic layer stays visible
and editable instead of hiding behind an opaque chat box.

## Start the interface

Expand All @@ -26,30 +27,51 @@ The Studio sends requests through its same-origin `/api/queryforge/*` proxy. Set
`QUERYFORGE_API_URL` in `web/.env.local` if the API is not available at
`http://127.0.0.1:8000`.

## Domain-first workflow

1. Open **Data Domains**.
2. Select the bundled Anime Streaming sample or create a clean business domain.
3. Add SQLite, CSV, or Parquet files inside the active domain.
4. Profile physical schema and candidate keys.
5. Review identity, grain, dimensions, measures, time fields, and ownership.
6. Define metrics, relationships, cardinality, and safe Join Paths.
7. Classify sensitivity, apply SQL policy, and pass quality checks.
8. Publish data and semantics atomically.
9. Ask questions and inspect Trust Trace evidence within that same domain.

The top-bar domain switcher changes the context for Overview, Data Sources,
Semantic Studio, Ask & Analyze, and Run History. Sources and runs carry a
`domain_id`; uploaded objects use
`domains/{domainId}/sources/{sourceId}/...`. A new domain never reuses semantic
definitions from the sample.

## Mandatory semantic onboarding

The Data Sources workflow profiles an uploaded SQLite, CSV, or Parquet asset,
generates a semantic draft, and then stops at a human review gate. Publication
remains disabled until:
The Data Sources workflow profiles an uploaded asset, generates a conservative
semantic draft, and then stops at a human review gate. Physical column names are
evidence—not accepted business truth. Publication remains disabled until:

1. entities, dimensions, metrics, relationships, and join paths have been
1. business entity identity and row grain are explicit;
2. primary keys, dimensions, measures, time semantics, and metric units are
reviewed;
2. contract validation succeeds; and
3. the reviewer explicitly confirms the semantic contract.
3. relationship cardinality and every multi-hop Join Path are approved;
4. owner, sensitivity, policy, and data-quality contracts are complete;
5. blocking validation succeeds; and
6. the reviewer explicitly confirms the contract.

The server enforces the same rule. Calling the upload endpoint without
`reviewed=true` returns an error, so the requirement cannot be bypassed through
the UI.
The server enforces both boundaries. Calling the upload endpoint without a valid
`domain_id` or without `reviewed=true` returns an error, so neither domain
selection nor semantic review can be bypassed through the UI.

## Runtime persistence

Hosted deployments use D1 for source metadata and run history and R2 for uploaded
files. The Python runtime still owns SQL planning, policy enforcement, read-only
execution, and analytical artifacts.
Hosted deployments use D1 for domains, source metadata, and run history and R2
for uploaded files. The Python runtime still owns SQL planning, policy
enforcement, read-only execution, and analytical artifacts.

The bundled anime dataset under `sample_data/anime_streaming/` is read-only
showcase data for the Studio. Upload tests and hosted storage never modify or
delete it.
showcase data attached to the Anime Streaming sample domain. It is not the
platform schema. Upload tests and hosted storage never modify or delete it.

## Validate the frontend

Expand Down
34 changes: 19 additions & 15 deletions web/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# QueryForge Studio

The visual workspace for QueryForge. It brings data onboarding, mandatory
semantic review, governed natural-language analysis, SQL inspection, trust
evidence, and run history into one responsive interface.
The domain-first visual workspace for QueryForge. Users create or select a data
domain, then onboard that domain's sources, build its mandatory semantic
contract, run governed analysis, inspect SQL and trust evidence, and audit
domain-scoped history in one responsive interface.

## Run locally

Expand All @@ -13,8 +14,9 @@ npm ci
npm run dev
```

Open <http://localhost:3000>. The Studio starts in demo mode when the Python API
is unavailable, so the full product tour remains interactive.
Open <http://localhost:3000>. The bundled Anime Streaming domain remains
interactive in demo mode when the Python API is unavailable. Newly created
domains start empty and never inherit sample entities or metrics.

For live answers, start the QueryForge API from the repository root in a second
terminal:
Expand All @@ -28,18 +30,19 @@ If the API uses another address, copy `.env.example` to `.env.local` and change

## Product areas

- **Overview** — platform metrics, engagement trends, semantic contract health,
and recent activity.
- **Data Sources** — anime dataset inventory plus SQLite, CSV, and Parquet
onboarding.
- **Semantic Studio** — entities, metrics, relationships, join paths, and
contract status in one graph-and-editor workspace.
- **Data Domains** — create, select, and manage isolated business contexts.
- **Overview** — active-domain readiness, governed metrics, contract health, and
recent activity.
- **Data Sources** — domain-scoped SQLite, CSV, and Parquet onboarding.
- **Semantic Studio** — a required contract builder for identity, grain,
dimensions, measures, metrics, relationships, Join Paths, policy, and quality.
- **Ask & Analyze** — natural language to governed SQL with progress, results,
export, and a complete Trust Trace.
- **Run History** — filterable, persistent history for audit and replay.
- **Run History** — filterable, domain-scoped history for audit and replay.

Data uploads are intentionally atomic: the upload endpoint refuses publication
until a semantic contract has been reviewed and validated.
until a data domain is selected and its semantic contract has been reviewed and
validated.

## Quality checks

Expand All @@ -54,8 +57,9 @@ tests.

The hosted build uses Cloudflare-compatible bindings:

- `DB` (D1) stores source metadata and Studio run history.
- `UPLOADS` (R2) stores uploaded source files.
- `DB` (D1) stores data domains, source metadata, and domain-scoped run history.
- `UPLOADS` (R2) stores files under
`domains/{domainId}/sources/{sourceId}/...`.

Database migrations live in `drizzle/`. The bundled anime source data remains in
the repository-level `sample_data/` directory and is not copied or modified by
Expand Down
Loading
Loading