A tool for generating RAG (Retrieval-Augmented Generation) evaluation datasets from local files or Confluence pages, using any OpenAI-compatible LLM.
This system uses a distributed architecture with Kafka for task queue management, separate scraper and worker processes, and supports both batch and streaming processing.
- Scraper: Discovers and submits Confluence pages to a Kafka topic
- Worker: Processes pages from Kafka, generates QA pairs, and stores results in SQLite
- Kafka/Redpanda: Message queue for task distribution
- SQLite: Storage for processed documents and generated QA pairs
- Pluggable Connectors: Ingest from local files or a Confluence space via
--connector - Any OpenAI-Compatible LLM: OpenRouter by default, or point
--generator-base-urlat any endpoint - Distributed Processing: Separate scraper and worker processes for scalability
- Batch Processing: Support for processing documents in batches for cross-document questions
- Date Filtering: Filter Confluence pages by last modification date (e.g., only pages updated in the last 6 months)
- Intelligent Question Generation: Creates questions of varying difficulty levels
- Multiple Languages: Support for German and English content
- HTML Processing: Robust HTML parsing for clean text extraction
- Fail-Fast Configuration: Validates env vars and CLI flags at startup with a clear, aggregated error
# Install core package (using uv)
uv sync
# Install with standalone script dependencies
uv sync --extra scripts
# Start Redpanda (Kafka-compatible broker); the compose file lives in infra/
docker compose -f infra/docker-compose.yaml up -dSlurp is configured from a .env settings file. Copy .env.example to .env
in your working directory and fill in the values — it is loaded automatically on
startup, so you don't need to export anything. Every setting can still be supplied
as an environment variable or a CLI flag; precedence is
CLI flag > env var > .env file > default.
cp .env.example .env
# edit .env, then just run:
python -m slurp workerThe .env file is the recommended way to configure Slurp: it keeps your API keys
and connector settings in one place, out of your shell history, and version-control
ignored. Legacy names (CONFLUENCE_*, KAFKA_*, SQLITE_*, OPENROUTER_API_KEY)
still work as aliases for the new SLURP_ vars.
Key settings (see .env.example for the full list):
SLURP_LLM_API_KEY="" # required when the generator is enabled
SLURP_CONNECTOR="local" # local | confluence
SLURP_CONFLUENCE_BASE_URL="https://your-domain.atlassian.net"
SLURP_CONFLUENCE_USERNAME="you@example.com"
SLURP_CONFLUENCE_API_KEY=""
SLURP_CONFLUENCE_SPACE="" # required when SLURP_CONNECTOR=confluence
SLURP_KAFKA_BOOTSTRAP_SERVERS="localhost:19092"
SLURP_KAFKA_TOPIC="tasks"
SLURP_SQLITE_DATABASE="./data.db"See .env.example for the full list including generator, local connector, and
observability options.
The QA generator talks to any OpenAI-compatible endpoint, selected by
--generator-base-url and an API key. The key is read from SLURP_LLM_API_KEY
(legacy LLM_API_KEY and OPENROUTER_API_KEY still work as aliases).
# Default: OpenRouter
export SLURP_LLM_API_KEY="your-openrouter-key"
# Any other OpenAI-compatible endpoint
export SLURP_LLM_API_KEY="$(your-token-command)" # or a static key
python -m slurp worker \
--generator-base-url https://your-llm-endpoint.example/v1 \
--generator-model your-modelSlurp ingests content through pluggable connectors, selected with
--connector. The default is local.
| Connector | Source | Requires |
|---|---|---|
local |
Files on disk (.md/.html/.txt) |
nothing (no Confluence creds) |
confluence |
A Confluence space | SLURP_CONFLUENCE_* credentials |
Both connectors still flow through Kafka and the LLM generator, so a broker
(infra/docker-compose.yaml) and SLURP_LLM_API_KEY are required either way.
# Scrape a directory of documents into the queue
python -m slurp scraper --local-path ./docs
# Only markdown files
python -m slurp scraper --local-path ./docs --local-extensions .md
# A single file
python -m slurp scraper --local-path ./docs/intro.md
# Then run the worker (it dispatches on each task's connector automatically)
python -m slurp worker --generator-batch-size 1# Serve an auto-refreshing HTML view of the generated QA pairs (default :8077)
python -m slurp render --open --sqlite-database ./data.dbThe page polls the SQLite generations table, so QA pairs appear as the worker
produces them.
python -m slurp skill # print the bundled SKILL.md
python -m slurp skill --install # write it to ./.claude/skills/slurp/SKILL.mdThe scraper discovers Confluence pages and submits them to Kafka
(note the explicit --connector confluence, since local is the default):
# Scrape up to 50 pages from a Confluence space
python -m slurp scraper --connector confluence --confluence-space RESEARCH --confluence-max-pages 50
# Filter by recent pages (last 3 months)
python -m slurp scraper --connector confluence --confluence-space RESEARCH --confluence-months-back 3
# Skip the first 100 pages
python -m slurp scraper --connector confluence --confluence-space RESEARCH --confluence-skip 100
# Run multiple scraper workers
python -m slurp scraper --workers 2 --connector confluence --confluence-space RESEARCHThe worker processes pages from Kafka and generates QA pairs:
# Process pages individually
python -m slurp worker --generator-batch-size 1
# Process pages in batches of 4 for cross-document questions
python -m slurp worker --generator-batch-size 4
# Specify a different model
python -m slurp worker --generator-model "anthropic/claude-3-sonnet"
# Run multiple worker processes
python -m slurp worker --workers 4 --generator-language de--confluence-space: Confluence space key to scrape--confluence-max-pages: Maximum number of pages to fetch (default: 50)--confluence-months-back: Only process pages modified within last N months (0 = no filter, default: 0)--confluence-skip: Number of pages to skip (default: 0)--confluence-concurrency: Number of concurrent requests (default: 4)--confluence-page-batch-size: Number of pages to fetch per batch (default: 50)
--generator-batch-size: Number of documents to process together (default: 1)--generator-model: LLM model to use (default: "google/gemini-2.5-flash-preview-05-20")--generator-language: Language for generated questions (default: "de")--generator-difficulty-ratio: Question difficulty (easy/medium/hard/mixed/balanced)--generator-concurrency: Number of concurrent LLM requests (default: 5)
The system uses SQLite for storing processed documents and generated QA pairs:
task_results: Stores processed Confluence pagesgenerations: Stores generated QA pairs with references to source pages
- Kafka Connection Errors: Ensure Redpanda is running (
docker compose -f infra/docker-compose.yaml ps) - Invalid Configuration: Slurp validates config at startup and prints exactly what is missing or out of range — read the error and see
.env.example - Database Errors: Verify SQLite database permissions and path
- LLM API Errors: Check
SLURP_LLM_API_KEYand your provider's quota; for non-OpenRouter endpoints also set--generator-base-url - HTML Parsing Issues: The HTML parser has been optimized for Confluence pages
- Scraper: Discovers and submits Confluence pages to Kafka
- Worker: Processes pages from Kafka and generates QA pairs
- LLMGenerator: Generates questions and answers using LLMs
- HTMLParser: Cleans and processes HTML content
- SqlitePersistence: Stores results in SQLite database
- KafkaQueueSubmitter: Submits tasks to Kafka
- KafkaConsumer: Consumes tasks from Kafka