Extract-0: A Specialized Language Model for Document Information Extraction

A 7-billion parameter model that outperforms GPT-4.1, o3, and GPT-4.1-2025 on document extraction — trained for $196 on a single H100.

This repository is the public record of the project: the academic paper, the reference implementation, and a sprint-by-sprint account of how Extract-0 was built over 20 sprints across 4 modules.

---

Headline Result

Extract-0 achieves a mean reward of 0.573 on 1,000 held-out document extraction tasks, surpassing far larger general-purpose models:

Model	Mean Reward	Parameters
Extract-0 (ours)	0.573	7B
o3	0.464	undisclosed (large)
GPT-4.1-2025	0.459	undisclosed (large)
GPT-4.1	0.457	undisclosed (large)

Three technical contributions make this possible:

1. Memory-preserving synthetic data generation — 280,128 training examples generated with accumulated cross-chunk context (M_i = M_{i-1} ∪ E(c_i)).
2. Parameter-efficient fine-tuning — LoRA adapts only 0.53% of weights (40.4M / 7.66B).
3. Semantic-similarity reward for RL — recognizes equivalent extractions despite surface-form variation, optimized via GRPO.

---

How It Was Built — 20 Sprints, 4 Modules

The project ran as four modules of five sprints each. Each module's detailed log lives in docs/.

Module 1 — Foundations & Synthetic Data (Sprints 1-5)

Problem definition, 53-paper literature review, first end-to-end pipeline, and the memory-preserving synthetic data architecture producing 281,128 examples (1,000 held out as the benchmark).

Module 2 — Supervised Fine-Tuning (Sprints 6-10)

Multi-source document collection, token-optimized augmentation, and LoRA SFT (r=16, α=32) adapting 0.53% of weights. Result: 0.507 reward, 79.9% JSON validity.

Module 3 — Reinforcement Learning (Sprints 11-15)

A type-aware semantic-similarity reward function and GRPO with adaptive KL control. Result: 0.573 reward, 89.0% JSON validity (+147% over base).

Module 4 — Evaluation & Dissemination (Sprints 16-20)

Unbiased benchmark construction, comparison against frontier models, stage ablation, and publication of the paper + code.

See reports.md for the full sprint timeline.

---

Training Stage Progression

Configuration	Mean Reward	JSON Validity	Relative Improvement
Base (DeepSeek-R1-Distill-Qwen-7B)	0.232	42.7%	baseline
Base + SFT	0.507	79.9%	+118.5%
Base + SFT + GRPO (Extract-0)	0.573	89.0%	+147.0%

---

Repository Structure

2025-2A-T20-G96-PUBLICO/
├── README.md                       # this file
├── paper.tex                       # the Extract-0 academic paper (source)
├── paper.pdf                       # compiled paper
├── arxiv.sty                       # paper style
├── PUBLIC_REPORT.md                # theoretical framework report (Module 1)
├── reports.md                      # 20-sprint timeline
├── requirements.txt                # dependencies
├── docs/
│   ├── module1_foundations_and_synthetic_data.md
│   ├── module2_supervised_finetuning.md
│   ├── module3_reinforcement_learning.md
│   └── module4_evaluation_and_dissemination.md
├── src/
│   ├── data_pipeline/
│   │   └── generate_data.py        # memory-preserving synthetic data generation
│   ├── training/
│   │   ├── supervised_finetuning.py # SFT with LoRA
│   │   └── reinforcement_learning.py # GRPO + semantic reward
│   └── evaluation/
│       └── evaluate_model.py        # benchmark evaluation + baseline comparison
└── logs/
    └── related_papers.md            # annotated bibliography

---

Quick Start

Requirements

• Python 3.8+, PyTorch 2.0+
• CUDA 11.8+ GPU (24GB+ VRAM recommended; training used a single H100 80GB)

git clone https://github.com/Inteli-College/2025-2A-T20-G96-PUBLICO.git
cd 2025-2A-T20-G96-PUBLICO
pip install -r requirements.txt

1. Generate synthetic data

python src/data_pipeline/generate_data.py --out data

2. Supervised fine-tuning

python src/training/supervised_finetuning.py --output-dir models/sft_checkpoint

Defaults: DeepSeek-R1-Distill-Qwen-7B, LoRA r=16/α=32, batch 16, lr 1e-4, 5 epochs.

3. Reinforcement learning (GRPO)

python src/training/reinforcement_learning.py --sft-model models/sft_checkpoint

Defaults: lr 5e-5, 8 generations/prompt, max 532 new tokens, adaptive KL.

4. Evaluation

python src/evaluation/evaluate_model.py --compare

---

Usage Example

import json, torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "models/grpo_checkpoint/best_model")
tokenizer = AutoTokenizer.from_pretrained("models/grpo_checkpoint/best_model")

schema = {"type": "object", "properties": {
    "author": {"type": "string"},
    "date": {"type": "string", "format": "date"},
    "findings": {"type": "array", "items": {"type": "string"}}}}
document = "Your document text here..."

prompt = f"""### System:
You are an expert data extraction system. Extract structured information from
documents according to the provided schema. Return only valid JSON.

### User:
Schema:
{json.dumps(schema)}

Document:
{document}

### Assistant:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=532, temperature=0.7)
extracted = json.loads(tokenizer.decode(out[0][len(inputs["input_ids"][0]):],
                                         skip_special_tokens=True))

---

Cost Breakdown ($196 total, single H100)

Stage	Cost
Synthetic data generation	$42
Supervised fine-tuning	$98
Reinforcement learning	$56

---

Author

Henrique Godoy · Inteli — Institute of Technology and Leadership · São Paulo, Brazil henriquegodoy.com

Licensed under the MIT License.