RAG Financial Multimodal — Enterprise v2.0

Production-grade multimodal RAG for financial document intelligence. Chart understanding · hybrid retrieval · numeric guardrails · multi-tenancy · full observability.

About the history: this codebase was rebuilt from scratch in 2026. The commits from February 2024 at the start of the history are an earlier, unrelated upload: a notebook, two pickled data files and an image. Nothing in the rebuild used them, so they were removed from the tree in October 2026. They remain in the history (commits 781d919–1d08a81).

---

System Architecture

Ingestion (top) and query (bottom) pipelines. Every component is pluggable — switch provider by changing one config value.

---

Why This System

Financial documents are mixed-media: narrative text, tables, charts, footnotes, cross-references. Standard RAG pipelines fail on charts and hallucinate numbers.

Problem	Solution
Charts contain the most important data but RAG ignores them	GPT-4o / Gemini / Qwen2-VL vision extraction — every chart yields exact axis values
Exact figures like $23.35B or TSLA miss semantic search	Hybrid RRF: dense embeddings + BM25 keyword fused with Reciprocal Rank Fusion
LLMs fabricate financial numbers	Numeric grounding guardrail — every stated number verified against source context
PII in analyst queries leaks to APIs	Presidio + CUSIP/ISIN/account number redaction before any external call
One broken vendor = full outage	Fallback chains — primary → secondary → local for every model-facing layer

---

Retrieval Quality

Measured offline on the 16 real filings behind the golden dataset (119 questions), using this repo's BM25, local bge-small embeddings and RRF fusion with default settings and no reranker. Evidence hit@5 means a top-5 chunk contains the exact supporting text for the answer.

Strategy (4,000-char chunks, default)	Evidence hit@5	Doc hit@5
BM25 only	0.71	0.87
Dense only	0.61	0.99
Hybrid RRF	0.71	0.93
Hybrid RRF + knowledge graph (optional)	0.72	0.90

With 1,000-character chunks, hybrid reaches 0.53 versus 0.42 (BM25) and 0.46 (dense), so fusion helps there. At the default chunk size it only ties BM25. About 3 in 10 questions get no answer-bearing chunk in the top 5. Full results, method and caveats are in BENCHMARK_RESULTS.md. Earlier versions of this README claimed 89% Recall@5. That figure could not be reproduced and has been withdrawn; the original charts are kept, with a provenance note, in BENCHMARK_RESULTS.md section 4.

---

Quickstart

git clone https://github.com/Mattral/filing-qa
cd filing-qa
cp .env.example .env          # set OPENAI_API_KEY or GOOGLE_API_KEY
docker compose up -d

curl -X POST http://localhost:8000/api/v1/ingest \
  -F "file=@tesla_10k.pdf" -F "tenant_id=demo"

curl -X POST http://localhost:8000/api/v1/query \
  -H "Content-Type: application/json" \
  -d '{"query": "What was gross margin in Q3 2023?", "tenant_id": "demo"}'

# Same query, answer streamed token by token (server-sent events)
curl -N -X POST http://localhost:8000/api/v1/query/stream \
  -H "Content-Type: application/json" \
  -d '{"query": "What was gross margin in Q3 2023?", "tenant_id": "demo"}'

Streamed tokens are provisional: guardrails run on the full answer and are reported in the final done event (REST API docs).

Or use the CLI:

pip install -e ".[all]"
filing-qa ingest tesla_10k.pdf --tenant demo
filing-qa query "What was Q3 revenue?" --tenant demo --show-sources

---

Multi-Provider Support

Every model-facing layer (text generation, vision, embeddings, vector store) is independently pluggable. Switch via a single .env line — zero code changes.

Fully open-source / zero-API-cost configuration

LLM_CONFIG__PROVIDER=local_vllm
LLM_CONFIG__MODEL=meta-llama/Llama-3.1-8B-Instruct
LOCAL_VLLM_GENERATOR_BASE_URL=http://localhost:8090/v1

VISION_CONFIG__PROVIDER=local_vllm
VISION_CONFIG__MODEL=Qwen/Qwen2-VL-7B-Instruct

VECTOR_STORE_CONFIG__EMBEDDING_PROVIDER=local
VECTOR_STORE_CONFIG__EMBEDDING_MODEL=BAAI/bge-small-en-v1.5

vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8090 --host 0.0.0.0
vllm serve Qwen/Qwen2-VL-7B-Instruct --port 8080 --host 0.0.0.0

---

Evaluation Quality

• Golden dataset: 119 QA pairs over 16 public filings from 10 companies. The mix is 56 factual, 24 calculation, 17 comparative, 13 table and 9 chart questions. Every figure is traced to a quoted passage in its source document (209/209 verified against the primary documents) and every derived number has an arithmetic check. Re-checking found errors in 9 of the original 22 samples, and those have been corrected.

• Scoring: RAGAS and an LLM-as-judge numeric scorer (needs an OpenAI key), plus two deterministic checks that need no LLM: whether the answer states the expected values, and whether it cites the expected source document.

• Answer quality: run on all 119 questions with gemini-3.8-flash over top-5 hybrid context, with deterministic scoring:

  Metric	All 119	When retrieval found the supporting text (63)
  Asked-for figure stated	0.68	0.84
  Cites an expected document	0.75	0.95
  Abstained ("not available")	0.35	0.16
  Fabricated numbers found	0	0

  Six answers stated numbers absent from their context, and all six were correct derivations. Most misses are honest abstentions caused by retrieval. Calculation questions are weakest: the model abstained on 15 of 24. Details are in BENCHMARK_RESULTS.md.

pytest tests/unit/test_golden_dataset.py      # dataset integrity, offline
python -m evals.retrieval_benchmark           # retrieval, offline
python -m evals.answer_eval --rpm 5           # retrieval + generation, needs an LLM key

---

Latency and Cost

Not yet measured in a reproducible run. Earlier latency, load-test and cost-per-query charts had no committed run outputs, so they are no longer cited. They are kept, with a provenance note, in BENCHMARK_RESULTS.md section 4. The only measured latency so far is generation alone, from the answer eval: p50 3.0 s, p95 5.0 s with gemini-3.8-flash. tests/load/locustfile.py and scripts/benchmark.py can measure them against a running deployment. The SLO targets (p99 < 8 s, < 1% errors) are configured in src/filing_qa/config.py and the alerting rules.

---

SLO Alerting

Design diagram, not live data: it illustrates how the configured alert rules behave and is not taken from a running deployment. Multi-window multi-burn-rate alerting from the Google SRE Workbook, with four alert tiers (14.4×, 6×, 3×, 1× burn rate) routing to PagerDuty/OpsGenie.

---

Architecture Layers

Layer	Component	Technology
Parsing	PDF text + tables	Unstructured.io / Docling / Marker
Vision	Chart + graph extraction	GPT-4o / Gemini 2.5 Flash / Qwen2-VL / Local vLLM (fallback chain)
Layout	Semantic grouping	Table-caption pairing, multi-page merge, HTML wrapping
Embedding	Dense vectors + cache	OpenAI / Voyage / Cohere / local BAAI/bge, Redis cached
Indexing	Vector store	DeepLake / pgvector / Qdrant
Retrieval	Hybrid (dense + BM25)	Reciprocal Rank Fusion, k=60
Reranking	Cross-encoder	ms-marco-MiniLM / Cohere Rerank v3
Generation	Cost-routed, multi-provider	OpenAI / Gemini / Anthropic / Local vLLM
Guardrails	Numeric grounding + PII	Presidio + custom regex + restricted-interpreter PoT calculator
API	REST + OpenAPI	FastAPI + uvicorn
Observability	Traces + metrics	OpenTelemetry + Prometheus + Grafana
Security	Auth + audit trail	API key + SHA-256 tamper-evident audit log
Multi-tenancy	Isolated namespaces	Per-tenant vector partitions + quotas + rate limits
Deployment	Container + K8s	Docker + Helm + HPA + NetworkPolicy

---

Key Features

Multimodal ingestion

• Vision LLM fallback chain: primary → secondary → local, never silently fails
• Layout-aware chunker: tables stay with their captions; multi-page tables merged
• ColPali visual retrieval (component only, not yet called by the query pipeline): late-interaction MaxSim scoring on page images (no OCR)
• Delta detection: skip unchanged documents on re-ingest; version history for rollback

Retrieval

• Hybrid RRF (k=60): dense + BM25, equal-weight Reciprocal Rank Fusion. Measured: beats both dense-only and BM25 at 1,000-char chunks (evidence hit@5 0.53 vs 0.46 and 0.42), and ties BM25 at the default 4,000-char chunks (0.71). See Retrieval Quality
• Cross-encoder reranking: ms-marco-MiniLM-L-6-v2 or Cohere Rerank v3
• Query analyzer: intent classification → cost routing, entity extraction, query rewriting
• Semantic cache: a query whose embedding is close enough to a cached one (cosine ≥ 0.92) is answered from cache, skipping retrieval and generation. The saving has not been measured
• Knowledge graph (optional, ENABLE_KNOWLEDGE_GRAPH=true): LLM-extracted entities/relations (COMPANY, METRIC, DATE; REPORTED_METRIC, FOR_PERIOD, …) and graph-augmented retrieval that walks query entities to related metrics/periods and RRF-fuses the linked chunks with hybrid results. The graph is in memory only (tutorial)

Quality assurance

• Numeric grounding guardrail: every number in the answer is verified against context
• Program-of-Thought calculator: the LLM writes the calculation; a restricted arithmetic interpreter (no exec(), no attribute access) evaluates it
• RAGAS evaluation: faithfulness, answer relevancy, context precision + LLM-as-judge
• Test suite: unit, integration, property-based (Hypothesis), API, chaos engineering. CI runs it on every PR, with lint, mypy and a 70% coverage floor, and blocks on failure

Production operations

• OpenTelemetry: distributed traces with ingest/retrieve/generate spans
• 17 Prometheus metrics: latency, cost, hallucination score, citation coverage, cache hit rate
• Multi-window SLO alerting: Google SRE workbook burn-rate pattern, PagerDuty/OpsGenie
• Kubernetes: HPA 2–10 replicas, PodDisruptionBudget, NetworkPolicy, IRSA
• Terraform: full EKS + RDS(pgvector) + ElastiCache + S3 + KMS infrastructure-as-code

---

Try It Live

Interactive notebook: notebooks/quickstart.ipynb walks through the full pipeline end-to-end — ingestion, hybrid retrieval, vision/chart queries, Program-of-Thought calculations, guardrails, and cost tracking — using Tesla's real Q3 2023 Investor Update as the example document.

Hosted demo: Mattral/RAG-Financial-Multimodal on Hugging Face Spaces (status RUNNING when checked on 2026-10-14). Bring your own OpenAI or Gemini key. The Space defaults to gemini-2.5-flash, which returns 404 for newly created Google API keys; pick a 3.x model or use OpenAI. On the free CPU tier the Space sleeps when idle, so the first load can take a minute.

Its source is in spaces/rag-financial/. To run it locally:

cd spaces/rag-financial
pip install -r requirements.txt
python app.py

Bring your own OpenAI or Google Gemini API key (Gemini has a free tier). The Space mirrors src/filing_qa/'s hybrid retrieval + guardrail logic in a self-contained form with full pipeline transparency — see exactly how each answer was retrieved and generated.

---

Repository Structure

src/filing_qa/
├── api/          FastAPI app, routers (ingest/query/documents/tenants/feedback)
├── agentic/      LangGraph multi-step reasoning with self-correction loop
├── cli.py        Typer CLI (ingest, query, evaluate, serve, health)
├── components/
│   ├── base.py            ABCs for all pluggable components
│   ├── parser/             Unstructured, Docling, Marker adapters
│   ├── vision/             GPT-4o, Gemini, Qwen2-VL, LocalVLLM + fallback chain
│   ├── embedder/           OpenAI, Voyage, Cohere, local (BAAI/bge)
│   ├── vector_store/       DeepLake, pgvector, Qdrant
│   ├── retriever/          HybridRetriever (dense + BM25 + RRF)
│   ├── reranker/           CrossEncoder, Cohere, NoOp
│   ├── generator/          OpenAI, Gemini, Anthropic, LocalVLLM
│   ├── evaluator/          RAGAS + LLM-as-judge numeric scorer
│   ├── guardrails/         Numeric grounding, PII redaction, injection detection
│   ├── knowledge_graph.py  Real LLM entity/relation extraction + graph traversal
│   ├── colpali_retriever.py  Real MaxSim late-interaction visual retrieval
│   ├── pot_executor.py     Program-of-Thought sandboxed calculator
│   ├── layout_parser.py    Table-caption pairing, semantic chunking
│   ├── query_analyzer.py   Intent classification, entity extraction
│   ├── version_manager.py  Delta detection, point-in-time retrieval
│   └── connectors/         S3, Azure Blob, GCS
├── config.py     Pydantic v2 BaseSettings, 12 nested sub-configs
├── pipeline/     RAGPipeline orchestrator (dependency injection)
├── sdk/          Python SDK (async + sync wrappers)
└── utils/        Telemetry, cost tracker, audit log, semantic cache, drift detector

terraform/        EKS + RDS(pgvector) + ElastiCache + S3 + IAM + KMS (IaC)
k8s/               Base manifests + Kustomize overlays (dev/prod)
helm/              Production Helm chart
spaces/            Standalone HuggingFace Space demos
└── rag-financial/ Gradio app — OpenAI + Gemini, hybrid retrieval, guardrails

---

Quick Reference

make setup          # Install deps + copy .env
make dev            # Start full stack with observability
make test           # Run the test suite with coverage
make eval           # Run RAGAS evaluation against golden dataset
make lint           # Ruff lint
make typecheck      # mypy
make docs           # Serve MkDocs site
make query Q="What was Q3 revenue?"

---

Limitations

• Evaluation is partial. Dataset integrity, offline retrieval and retrieval-plus-generation with one model are measured. The full ingest pipeline (unstructured parsing, reranker, guardrails, Program-of-Thought), RAGAS scores, load latency and cost have not been measured on a reproducible run yet (see BENCHMARK_RESULTS.md).
• Retrieval misses often. About 30% of golden questions get no answer-bearing chunk in the top 5 at default settings, and multi-row tables are the weakest case when chunks are small.
• Small, single-author golden set. 119 questions on fiscal 2023, Tesla-heavy (40 of 119), written by one person from the filings themselves. Their wording favors keyword search.
• Chart questions are text-checkable. The 9 chart questions use charts whose values also appear in tables, so they do not isolate the vision path.
• Gemini judge not wired. The LLM-as-judge and RAGAS scorers call OpenAI. With only a Gemini key, just the deterministic metrics are available.
• Calculations are often declined. Under the default "never estimate" system prompt the model frequently refuses to compute derived figures (ratios, sums) unless Program-of-Thought handles them.
• Page citations are unreliable when chunks lack page metadata. The model invents them.
• Thinking-token budget. On Gemini thinking models, reasoning tokens count against the output limit. LLM_CONFIG__THINKING_BUDGET_TOKENS (default 1024) now reserves them separately; with it unset, answers can truncate.
• Retired model names. Google no longer serves gemini-2.5-* to new API keys. Defaults that still name them need a current model such as gemini-3.8-flash.

---

Documentation

Document	Description
Architecture Overview	System design and component interactions
Configuration Reference	All environment variables
Quickstart	Up and running in 10 minutes
10-K Analysis Tutorial	End-to-end walkthrough
Anomaly Detection	Multi-quarter statistical analysis
Performance & Cost Tuning	Latency and cost optimisation
Troubleshooting	Common issues + on-call runbook
Security	Auth, PII, audit, compliance
ADR Index	Architecture decisions
BENCHMARK_RESULTS.md	Quality, latency, cost numbers
CONTRIBUTING.md	Contribution guide

---

License

MIT — see LICENSE. Built with care by @Mattral.