Skip to content

TensorGymnastic/lean

BranchesTags

Repository files navigation

lean

An MCP server for ingesting Lean Six Sigma PDFs into a searchable multimodal knowledge base. Uses marker-pdf for high-quality extraction (with optional remote GPU acceleration), Vision-Language Model enrichment for charts and figures, section-aware chunking, GPU-accelerated embeddings, provenance-tracked storage, and hybrid BM25 + vector search via pgvector.

Features

  • Marker-pdf extraction (primary) — datalab-to/marker (surya OCR + texify) with proper table/equation/heading formatting and figure extraction. Runs locally on CPU or on a remote GPU server via HTTP (44× faster — 14s vs 626s for a 29-page PDF). Falls back to Unlimited-OCR (remote transformers) then markitdown (pure Python) when unavailable.
  • VLM chart/image enrichment — Vision-Language Model (MiniMax M3 default, Ollama Qwen3.5 fallback) describes every chart, diagram, and figure at ingest time. Descriptions are structured (title, chart_type, axis_labels, key_data_points, description) and embedded alongside text, making visual content searchable. ~3.5s/image, ~$0.004/image via MiniMax M3 API.
  • Provenance metadata — every chunk tracks embedding_model, embedding_dim; every image chunk tracks image_hash, provenance_model (which VLM described it). Enables model-version auditing and future dedup.
  • Chunk-type filter — search returns text and image chunks; filter by chunk_type="image" to surface only charts/figures.
  • Section-aware chunking — mistune AST parser splits markdown by headings, then a recursive tiktoken-based splitter bounds chunks to a target token window
  • GPU-accelerated embeddings — LiquidAI/LFM2.5-Embedding-350M (1024-dim) served via Ollama on the GPU server, with automatic local CPU fallback
  • Hybrid search — BM25 full-text (PostgreSQL tsvector) fused with pgvector cosine similarity via Reciprocal Rank Fusion (RRF, k=60)
  • Cross-encoder reranking — fetch wide candidate set, rerank with ms-marco-MiniLM-L-6-v2, return top-k
  • MCP server — 8 tools, 4 resources, 3 prompts exposed over stdio or HTTP, plus a REST mirror (7 MCP-mirroring endpoints plus /health; reingest is intentionally CLI-only)
  • Retrieval evaluation — lean eval command computing hit_rate@k, MRR@k, NDCG@k, Recall@k (see docs/evaluation.md for caveats)
  • Optional LLM sidecar — Contextual Retrieval, HyDE, multi-query generation — all opt-in, pipeline works without LLM
  • Security hardening — corpus-root path confinement, API key validation (min 16 chars, change-me rejected), HuggingFace model revisions pinned to SHA hashes, non-root Docker user, multi-stage build
  • Optional ML deps — torch/transformers/sentence-transformers only when local CPU embeddings are needed (uv sync --extra local-models); marker-pdf for high-quality extraction (uv sync --extra marker)

Tech Stack

Layer Technology
MCP fastmcp v3.4.4
Extraction (primary) datalab-to/marker (surya OCR + texify) — local CPU or remote GPU via HTTP
Extraction (fallback) baidu/Unlimited-OCR via transformers (remote GPU), then markitdown
VLM MiniMax M3 via API (default), or Ollama Qwen3.5 (local fallback)
Embeddings LiquidAI/LFM2.5-Embedding-350M (1024-dim) via Ollama (remote GPU)
Search pgvector cosine + PostgreSQL tsvector BM25 + RRF fusion
Storage Supabase (Postgres 15 + pgvector) via Docker
Framework Python 3.12+, uv-managed, strict mypy + ruff

Prerequisites

  • Python 3.12+ with uv
  • Docker (for local Supabase)
  • Remote GPU server (NVIDIA, 12GB+ VRAM) running:
    • Marker server: pure-Python HTTP wrapper around marker's PdfConverter on port 8000 (44× faster than CPU)
    • Ollama: lfm2.5-embed-32k model (LFM2.5-Embedding-350M, 32K context) on port 11434
    • (optional) Unlimited-OCR server: baidu/Unlimited-OCR via transformers on port 8001 (secondary extraction fallback)

All GPU services are optional — lean falls back to local marker-pdf (CPU), local CPU embeddings, and markitdown when remote servers are not configured. VLM enrichment is also optional (disable via vlm.enabled: false).

Quick Start

git clone <repo-url> && cd lean

# 1. Install dependencies
uv sync --all-groups
uv sync --extra marker          # marker-pdf for high-quality extraction
# (optional) Local CPU embeddings + reranker (~2GB torch):
uv sync --extra local-models

# 2. Install git hooks
make hooks-install

# 3. Configure secrets
cp .env.example .env  # set SUPABASE_DB_URL, LEAN_MCP_API_KEY (min 16 chars, not 'change-me')
# (optional) VLM_API_KEY for MiniMax, MARKER_REMOTE_URL for GPU marker server

# 4. Start local database
docker compose up -d supabase-db
make db-init

# 5. Ingest PDFs
make ingest-all

# 6. Search (text + image chunks)
make search QUERY="What is DMAIC?"
make search QUERY="Pareto chart of defects"   # surfaces VLM-described images

For the remote GPU server (marker + Ollama + optional OCR setup), see docs/ocr-server-deployment.md.

Commands

CLI (lean)

All commands support --json for structured output. Use -v / --verbose for debug logging.

Command Description
lean ingest <path> Ingest a PDF into the corpus (uses marker → OCR → markitdown fallback chain, optional VLM enrichment)
lean search "<query>" Semantic + hybrid search (--k, --doc-id, --section, --author, --year-min, --year-max, --min-score, --chunk-type text|image)
lean list-documents List all documents in the corpus
lean get-chunk <chunk_id> Retrieve a single chunk by UUID
lean get-markdown <doc_id> Get extracted markdown for a document
lean delete <doc_id> Delete a document and all its chunks
lean reingest <doc_id> Re-extract a document with current settings
lean reingest-all Batch reingest all documents (--force to re-extract OCR'd docs)
lean eval Run retrieval evaluation (--sample-size, --k, --dataset <path> for curated queries)
lean health Check marker server, OCR server, database, and Ollama connectivity
lean mcp-serve Start the MCP server (--transport stdio|http, --port)
lean api-serve Start the FastAPI REST API server (--reload for dev)
lean db-init Apply all SQL migrations to the database

Makefile

Target Description
make verify format-check + lint + typecheck + unit tests
make verify-all all tests (incl. integration/e2e)
make format / make format-check / make lint / make typecheck individual checks
make db-init apply all SQL migrations
make db-reset supabase db reset (destroys data)
make ingest-all / make ingest-one FILE=… ingest PDFs
make search QUERY="…" search from the command line
make mcp-serve / make mcp-serve-http start MCP server (stdio / HTTP)
make api-serve start FastAPI REST mirror (port 8766)
make health / make smoke check marker server, OCR server, database, Ollama (smoke is an alias for health)
make build / make up / make down Docker lifecycle

Usage

MCP Client (Claude Desktop, opencode, etc.)

{
  "mcpServers": {
    "lean": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/lean", "lean", "mcp-serve"]
    }
  }
}

8 tools: ingest_pdf, search, get_chunk, list_documents, get_document_markdown, delete_document, reingest, corpus_stats

4 resources: lean://documents, lean://documents/{id}/markdown, lean://documents/{id}/chunks, lean://stats

3 prompts: lean_qa, lean_glossary, lean_compare_concepts

REST API (near-full mirror)

make api-serve   # http://localhost:8766

curl -H "Authorization: Bearer $LEAN_MCP_API_KEY" \
     "http://localhost:8766/search?query=What+is+DMAIC%3F&k=5"

Endpoints (7/8 MCP tools — reingest is CLI-only; see docs/limitations.md):

Endpoint MCP equivalent
GET /health (no auth, always 200)
GET /search search (supports chunk_type filter)
GET /documents list_documents
GET /documents/{id}/markdown get_document_markdown
DELETE /documents/{id} delete_document
GET /chunks/{chunk_id} get_chunk
GET /stats corpus_stats
POST /ingest ingest_pdf

Domain errors map to HTTP codes: ValueError → 400, PermissionError → 403, FileNotFoundError → 404.

Documentation

  • docs/ — reference docs (configuration, operations, architecture, evaluation, limitations)
  • docs/configuration.md — every Settings field, defaults, validators, gotchas
  • docs/operations.md — Docker ports, healthcheck semantics, post-ingest reindex, reingest semantics
  • docs/architecture.md — pipeline, directory layout, transport tier pattern
  • docs/limitations.md — known caveats (page fields on remote path, eval pseudo-queries, SHA-256 dedup orphans, REST near-full mirror)
  • docs/decisions/ — Architecture Decision Records
  • AGENTS.md — operational rules for agents

License

MIT for project code. See model licenses for third-party weights (datalab-to/marker, baidu/Unlimited-OCR, LiquidAI/LFM2.5-Embedding-350M, MiniMax M3).

For known caveats and runtime behavior, see docs/limitations.md and AGENTS.md.

About

Lean Six Sigma PDF corpus MCP server (local snapshot of quanti/lean at HEAD 75cc41d)

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages