A local research prototype for finding Python functions from natural-language descriptions and generating answers grounded in retrieved code. It combines a retrieval-fine-tuned GraphCodeBERT, Chroma vector search, local Qwen3:14b reranking/generation through Ollama, FastAPI, and Streamlit.
The project includes a small, explicitly synthetic offline demo and a separately configured real-model path. The demo works without model weights, API credentials, or Ollama. A fresh checkout does not contain the real corpus, model weights, trained checkpoint, vector database, or raw experiment logs. Obtain or regenerate those artifacts before using real mode.
flowchart LR
Q[User request] --> E[GraphCodeBERT query encoder]
E --> I[Chroma: Top 5 functions]
I --> R[Qwen: permutation reranking]
R --> C[Up to 3 references + token budget]
C --> G[Qwen: structured answer]
G --> A[FastAPI / Streamlit]
G -. separate evaluation .-> T[Docker behavior checks]
The offline pipeline adapts CodeSearchNet records, removes AST docstrings, canonicalizes and deduplicates functions, preserves provenance, assigns repository/duplicate groups, trains the encoder, and builds fingerprinted indexes. Generated code is never executed by the request handler.
Prerequisites: Python 3.12 and uv. Run from the repository root; make is required for these convenience targets.
make setup
make test
make demo-data
make demo-index
make demo-eval
make demo-smokeThen run these in separate terminals:
make demo-apimake demo-uiOpen the UI or interactive API documentation.
Demo boundary: lexical hash embeddings, identity reranking, a codepoint token counter, and a reference-copy generator are explicit test doubles. They verify wiring and error handling; they do not establish learned retrieval or code-generation quality.
uv.lock is the authoritative project dependency lock. requirements.txt is its generated core-only projection (make export-lock), not an independent version policy. make setup selects Python 3.12.6 because AST normalization is versioned; it does not alter unrelated virtual environments.
A CodeSearchNet Python experiment used 10,000 training pairs, 500 independent training-validation pairs, 50 development queries, 200 test queries, and a 20,000-function retrieval corpus. Ten epochs / 3,130 optimizer updates were recorded on Apple MPS. Epoch 10 was selected by validation MRR@5 in a 500-candidate validation pool; this is distinct from the following 20,000-function test benchmark.
| System | Recall@5 / Hit@5 | MRR@5 |
|---|---|---|
| Pretrained code-only baseline | 0.100000 | 0.068500 |
| Baseline + Qwen reranking | 0.100000 | 0.097500 |
| Graph-aware fine-tuned path | 0.850000 | 0.703167 |
| Fine-tuned path + Qwen reranking | 0.850000 | 0.788750 |
The fine-tuned reranking difference was +0.085583 MRR@5; a 10,000-draw paired query bootstrap (seed 42) gave a 95% percentile interval of [0.048165, 0.123583]. Query outcomes: 41 improved, 12 regressed, 147 unchanged in reciprocal rank.
These are results recomputed from recorded per-query logs, not a claim that the benchmark was freshly rerun during documentation cleanup. The source package includes an ID-only ranking export and evidence manifest; recompute the published test table and interval without models:
.venv312/bin/python scripts/recompute_published_metrics.py
``` Labels identify one docstring-paired function per query; they do not exhaustively judge all relevant functions. Baseline/fine-tuned differences include graph inputs, pooling, training, and approximate-search behavior: this is not a controlled weight-only ablation. Timed benchmark retrieval reused verified query vectors; query-encoding precomputation is outside that timing boundary. See [evaluation and evidence](docs/EVALUATION.md).
## Real-model setup
Read [reproducibility instructions](docs/REPRODUCIBILITY.md) before downloading or training. These operations are explicit, potentially large, and are not part of the quick start.
```sh
make setup-models
make download-codesearchnet
make prepare-codesearchnet
make download-models CONFIG=configs/codesearchnet_mps.yaml
make download-qwen-tokenizer CONFIG=configs/codesearchnet_mps.yamlUse configs/codesearchnet_real.yaml for CPU, or configs/codesearchnet_mps.yaml for supported Apple MPS. For another device, create an explicitly named configuration; do not silently change a recorded experiment. Install and start Ollama, and obtain qwen3:14b separately. The recorded strict tokenizer adapter targets Ollama 0.24.0 and a bound Qwen digest; a newer service is not automatically compatible.
Formal training, indexing, and serving are separate steps:
make formal-training-preflight CONFIG=configs/codesearchnet_mps.yaml
make train-graphcodebert CONFIG=configs/codesearchnet_mps.yaml
make index CONFIG=configs/codesearchnet_mps.yaml BACKEND=graphcodebert_base
make index CONFIG=configs/codesearchnet_mps.yaml BACKEND=graphcodebert_finetuned
make prepare-ollama-tokenizer CONFIG=configs/codesearchnet_mps.yaml
make real-api CONFIG=configs/codesearchnet_mps.yaml
# Another terminal:
make real-uiThe checkpoint and matching manifest are both required for fine-tuned inference. The code rejects mismatched model/parser/tokenizer artifacts and incompatible indexes. Neither the release package nor a base-model download supplies the trained checkpoint. Existing local artifacts remain local; they are not a hosted model release.
| Endpoint | Purpose |
|---|---|
GET /health |
Mode and per-component readiness |
POST /search |
Query, explicit backend, top_k from 1 to 5 |
POST /rerank |
Validate and reorder the same search candidates |
POST /chat |
Search → rerank → context → structured generation |
curl -X POST http://127.0.0.1:8000/search \
-H 'Content-Type: application/json' \
-d '{"query":"Add two numbers","backend":"demo_hash","top_k":5}'Use graphcodebert_finetuned with a configured real service. Streamed chat uses metadata, token, done, and terminal error SSE events. A token event is a text chunk, not necessarily one model token. The current UI receives the stream but displays the parsed answer after completion.
Invalid requests return 422; unavailable required services return 503. Reranking failures get a bounded retry and then a logged identity-order fallback. Uninitialized real components are unknown; search-ready without validated generation is degraded; an initialized search route plus valid generation binding is ready. These statuses describe service state, not answer correctness.
Docker execution is optional and separate from generation. It uses an already available image, no network, non-root execution, read-only mounts/filesystem, and CPU/memory/PID/time limits. The built-in harness supports simple JSON-serializable function cases; it is not a general audio or arbitrary-program evaluator.
Two previously saved real outputs (add, safe_divide) passed their four predefined behavior cases in a later Docker follow-up. A separate audio-conversion output failed three of four post-hoc development cases, first encountering an undefined pydub name. A clearly labeled manual repair passed the same four audio cases. This is useful failure evidence, not a general 100% generation-success claim. See the evaluation boundary.
Known limits include weak labels, possible pretrained/source overlap, MinHash misses, approximate-search discrepancies, no semantic rejection threshold for unrelated requests, and limited functional coverage. Nonempty Top K, legal citations, valid JSON, and service completion do not establish code correctness. The project is a single-user local prototype; authentication, multi-user isolation, deployment SLOs, and broad generation reliability have not been established.
- System architecture
- Data sources and adapters
- Evaluation definitions and measured boundaries
- Reproduction and artifact requirements
- Current implementation contract
- Publication checklist
- Source cleanup and verification record
The current scope is GraphCodeBERT + local Qwen. The unused Gemini runtime integration has been removed. Old local experiment manifests may still mention the former optional provider: those historical identities are not rewritten or claimed as measured comparisons.
Graph parsing derives from pinned Microsoft CodeBERT/GraphCodeBERT sources; attribution and MIT notice are retained under src/training/vendor/. The native token counter retains upstream notices under src/rag/vendor_ollama/.
Model weights, dataset contents, source-code snippets, and third-party libraries retain their own terms. A dataset download is not blanket permission to redistribute every repository's code. Unknown snippet licenses remain unknown; they are not relicensed by this project. Large/private runtime artifacts are excluded from the source package. No project-wide open-source license has yet been selected for the original project code; public visibility alone is not a license grant. See third-party notices and publication guidance before sharing an archive.