Function-Aware Fill-in-the-Middle as Mid-Training
for Coding Agent Foundation Models

A coding agent's inner loop — act → observe → continue — is structurally the same shape as a function call site: a caller binds arguments, a callee returns a value computed elsewhere, and downstream code consumes that value. That conditioning structure already exists, at internet scale, in ordinary code.

This project exploits it. We mask functions — chosen by program-dependency-graph analysis and a complexity–inferability double criterion, not at random — and mid-train on recovering them, rationale first. Then we hand the checkpoint to an existing agentic post-training pipeline, unmodified. The gain survives post-training, transfers across post-training pipelines (R2E-Gym, SWE-Smith, SWE-Lego) and base families (Qwen2.5-Coder, Qwen3), and pays back most of the general-capability erosion that agentic post-training quietly charges.

🔥 News

• [2026-07] Full release: corpus (400K samples / 2.6B tokens), six checkpoints, data-construction pipeline, training configs, and evaluation guides.
• [2026-07] Paper released on arXiv (a copy also ships in this repo as paper.pdf).

📦 Released artifacts

Models — every arm of the paper's pipeline, on the Hub:

Model	Base	Recipe	SWE-Bench-Verified	SWE-Bench-Lite
FIM-Mid-7B	Qwen2.5-Coder-7B-Instruct	FIM mid-training only	—	—
FIM-Mid-8B	Qwen3-8B	FIM mid-training only	—	—
FIM-Mid-14B	Qwen2.5-Coder-14B-Instruct	FIM mid-training only	—	—
FIM-7B	FIM-Mid-7B	+ R2E-Gym post-training	17.80 (+2.8)	15.00 (+3.7)
FIM-8B	FIM-Mid-8B	+ SWE-Lego post-training	35.00 (+3.2)	32.70 (+5.4)
FIM-14B	FIM-Mid-14B	+ R2E-Gym post-training	29.20 (+3.0)	22.00 (+4.0)

The FIM-Mid-* checkpoints are released for reproducibility and further post-training — the paper deliberately never scores them directly (a FIM-only model has degraded instruction-following and cannot be compared fairly against instruction-tuned baselines).

Dataset — TIGER-Lab/FIM-Midtraining-400K:

Config	Samples	Masked target	Tokens
all_merged (main-results corpus)	400,000	80% single / 15% pair / 5% triple	~2.63B
single_function	320,000	one function body	~2.0B
function_pair	60,000	two coupled functions, jointly	~0.4B
function_triple	20,000	three coupled functions, jointly	~0.2B

Built from 968 permissively-licensed Python repositories, zero overlap with SWE-Bench source repos, commit-date cutoff before the earliest SWE-Bench base commit, every sample carrying a filtered Gemini-3-Flash rationale and full provenance metadata.

📊 Results

In-domain (Table 1). Means over three evaluation seeds; baseline is base + post-training reproduced under our harness, ours adds FIM mid-training in front of the identical pipeline.

Setting	SWE-Bench-Verified	SWE-Bench-Lite	Average
Qwen2.5-Coder-7B-Instruct
+ R2E-Gym (reproduced)	15.00	11.33	13.17
+ R2E-Gym (officially reported)	19.00	11.00	15.00
+ FIM-Midtrain + R2E-Gym → FIM-7B	17.80	15.00	16.40
Δ (ours vs. reproduced)	+2.80	+3.67	+3.24
+ SWE-Smith (reproduced)	12.30	14.20	13.25
+ FIM-Midtrain + SWE-Smith	17.60	14.70	16.15
Δ (ours vs. reproduced)	+5.30	+0.50	+2.90
Qwen2.5-Coder-14B-Instruct
+ R2E-Gym (reproduced)	26.20	18.00	22.10
+ FIM-Midtrain + R2E-Gym → FIM-14B	29.20	22.00	25.60
Δ (ours vs. reproduced)	+3.00	+4.00	+3.50
Qwen3-8B
+ SWE-Lego (reproduced)	31.80	27.30	29.55
+ FIM-Midtrain + SWE-Lego → FIM-8B	35.00	32.70	33.85
Δ (ours vs. reproduced)	+3.20	+5.40	+4.30

Capability preservation (Table 2, 14B + R2E-Gym). Agentic post-training buys SWE-Bench points and quietly charges for them elsewhere — 4.81 points on average across six benchmarks. Mid-training pays most of that bill back in the same run:

Setting	LiveCode	OJBench	FSB-EN	Terminal	τ-bench	BFCL	Avg
Instruct model (ceiling)	37.20	5.20	53.80	0.00	5.70	23.20	20.85
+ R2E-Gym	24.10	2.80	47.72	2.41	3.40	15.80	16.04
+ FIM Mid-Train + R2E-Gym	35.20	4.74	48.25	3.66	7.30	18.20	19.56
Δ (vs. R2E-Gym only)	+11.10	+1.94	+0.53	+1.25	+3.90	+2.40	+3.52

The load-bearing cells are τ-bench (+3.9) and BFCL (+2.4): neither contains Python code-editing data, and the corpus contains no tool-use trajectories. There is no data overlap that could explain those gains — only a structural prior installed at mid-training that outlives post-training.

🔍 How targets are selected

Not every function is worth masking. For each file we parse the AST into a program dependency graph (call edges + same-class sibling edges), score every function on complexity Ĥ (LoC, cyclomatic complexity, nesting) and inferability Î (call-site specificity, in-file callees, signature, docstring, class context), and keep targets where FIM = Ĥ·Î/(Ĥ+Î)·ρ(Δ) clears a threshold — substantial and recoverable, with a one-sided penalty on targets that stay hard even given full context. Groups of 2–3 coupled functions are selected over eight dependency topologies with inferability recomputed under joint masking. Each target then gets a rationale generated from the masked context only, judged against the ground truth, and formatted rationale-before-body — the think-then-act structure of an agent step.

🚀 Quickstart

Just use a model (FIM-8B is the strongest released checkpoint):

vllm serve TIGER-Lab/FIM-8B --served-model-name FIM-8B --max-model-len 163840

Scoring it on SWE-Bench, with the scaffold each checkpoint was post-trained for: evaluation/swebench/released_checkpoints.md.

Reproduce the paper, stage by stage:

# 0. (optional) Rebuild the corpus from scratch — or skip: it is on the Hub
cd data_construction && pip install -r requirements.txt
export GEMINI_API_KEY='...'
./scripts/run_all.sh single          # start with 2 repos, a few cents of API spend

# 1. Mid-train (8x H100, LLaMA-Factory)          -> midtraining/
huggingface-cli download TIGER-Lab/FIM-Midtraining-400K all_merged_400k.jsonl \
  --repo-type dataset --local-dir <LLaMA-Factory>/data/
llamafactory-cli train midtraining/configs/fim_midtrain.yaml

# 2. Post-train twice: from the stock model, and from the mid-trained one
#    R2E-Gym | SWE-Smith | SWE-Lego, upstream recipes unmodified   -> posttraining/

# 3. Evaluate both arms through the identical harness              -> evaluation/

Every stage is run twice — once from the stock instruct model (baseline), once from the mid-trained checkpoint (ours). The difference between those two runs is the entire claim. Both arms must go through an identical post-training and evaluation harness; comparing against a published baseline instead of your own reproduction is how a harness difference gets reported as a method gain.

🗂️ Repository layout

data_construction/   968 GitHub repos -> 400K FIM samples (~2.6B tokens)
        |            PDG parsing, Ĥ/Î scoring, masking, CoT generation + filtering
midtraining/         base model + FIM corpus -> FIM-Mid-{7B,8B,14B}
        |            one reference config + the three as-run configs
posttraining/        + R2E-Gym | SWE-Smith | SWE-Lego  -> FIM-{7B,8B,14B}
        |            upstream recipes, unmodified; reference + as-run configs
evaluation/          SWE-Bench Verified/Lite (scripts) + six capability
                     benchmarks (pinned repro guides)

Stage	What's there
data_construction/	Complete and tested — builds the corpus end to end from a repo list
midtraining/	LLaMA-Factory configs; identical recipe for all three base models
posttraining/	One directory per pipeline (R2E-Gym, SWE-Smith, SWE-Lego)
evaluation/	SWE-Bench scripts + exact repro guides for the six Table-2 benchmarks

💻 Compute

Everything ran on a single node of 8x H100 80GB. Mid-training is 1 epoch at 32K context. SWE-Bench evaluation is by far the slowest step: hundreds of Docker containers per pass, and the paper averages three seeds per arm per split. The full paper is ~5,760 GPU-hours.

📜 Corpus licensing

The 968 source repositories are each under their own license (>80% MIT/Apache/BSD; all at least research-permissive). The per-repository inventory ships in data_construction/data/code_repo_list_968.csv and joins to every released sample via its repository_url metadata. Check it against your intended use before training on anything derived from this corpus. Code in this repository is Apache-2.0.

📖 Citation

@article{wang2026fim,
  title={Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models},
  author={Wang, Yubo and Liang, Jiarong and Zhang, Yuxuan and Liu, Xuye and Wei, Cong and Zhang, Yuyu and Nie, Ping and Chen, Wenhu},
  journal={arXiv preprint arXiv:2607.12463},
  year={2026}
}

🙏 Acknowledgements

The post-training pipelines and agent scaffolds are used as released by their authors — R2E-Gym, SWE-Smith, SWE-Lego, OpenHands, and SWE-bench for scoring. Training uses LLaMA-Factory and torchtune; base models are Qwen2.5-Coder and Qwen3; rationales come from Gemini-3-Flash.