Clusy Crawler

Fast, bounded, source-derived web extraction

An Apache-2.0, self-hosted FastAPI service for turning HTTP(S) resources into source-derived Markdown, HTML, and links, plus optional model-assisted schema-constrained JSON.

Quick start · Architecture · API · Evidence · Self-hosting · Research

---

Clusy Crawler combines guarded HTTP/2 fetching, conditional Chromium rendering, a native Rust/PyO3 extraction core, deterministic specialists, bounded recursive discovery, optional Redis caching, and explicit completeness provenance. The default extraction path is local and does not require a model.

• Deterministic by default. The local extraction path needs no model or third-party extraction API.
• Escalation is explicit. Static, browser, and quality images keep their dependency and runtime boundaries visible.
• Results carry context. Route, rendering, completeness, truncation, cache, timing, source identity, and service identity remain inspectable.

Release status: Beta 4 Preview

The deterministic extraction and self-hosting paths are available for evaluation. Pin a source commit or image digest: API and operational compatibility may still change before the first stable release.

Claim boundary

One direct public-repository AEB article-body result is registered as Verified. Broader WebMain, WCXB, and live-provider evaluations remain outside the claim set. Every measured statement below is registry-bound.

One result you can reproduce

Verified evidence — Article Extraction Benchmark · article_body · 181 pages. Clusy F1 0.972127; exact Trafilatura 2.1.0 F1 0.957546; F1 delta +0.014581; F1 delta CI95 low +0.005547; F1 delta CI95 high +0.025336; paired-bootstrap win fraction 0.9996; machine-local in-memory throughput 173.97 pages/s.

Lineage boundary. The measured candidate uses a pinned rs-trafilatura descendant. This is a same-family implementation/version comparison with exact Python Trafilatura 2.1.0, not evidence for an independently originated extraction algorithm.

The complete receipt and the limits of this Beta 2 comparison are versioned in the repository. Beta 4 carries the Beta 3 runtime and source-serialization hardening plus documentation and release-integrity corrections, without rewriting that historical result.

Reproduce the exact historical run · Frozen protocol · Compact report · Registry entry

Have a difficult page? Open a minimal failing case.

Quick start

The default Compose stack runs the browser-capable image and a bundled Redis. It binds the API to loopback only.

git clone https://github.com/clusy-io/clusy-crawler.git
cd clusy-crawler
cp .env.example .env

docker compose up --build --detach
curl --fail http://127.0.0.1:11235/health/ready

Extract one page:

curl --fail --request POST http://127.0.0.1:11235/crawl \
  --header 'content-type: application/json' \
  --data '{"urls":["https://example.com"],"js_render":false}'

.env.example uses local mode with no bearer token. Before sharing the service, set ENVIRONMENT=prod, CRAWLER_API_TOKEN, an independent SERVING_FINGERPRINT_KEY, and an exact GIT_SHA. See docs/SELF_HOSTING.md.

Build from source

Requires Python 3.12+, Rust 1.85, and uv.

uv sync --locked --extra dev
uv run playwright install chromium
uv run uvicorn app.main:app --host 127.0.0.1 --port 11235

For a static-only local process, omit the browser install and disable both Playwright switches:

PLAYWRIGHT_ENABLED=false \
PLAYWRIGHT_JAVA_SCRIPT_ENABLED=false \
uv run uvicorn app.main:app --host 127.0.0.1 --port 11235

Runtime images

Docker target	Contains	Use it for
static-runtime	API, native/Python extraction, PDF support; no Playwright or browser	Lowest-footprint deterministic service
browser-runtime	Static runtime plus Playwright and Chromium	Conditional or explicit JavaScript rendering; Compose default
quality-runtime	Browser runtime plus pinned MinerU-HTML and MinerU-Webkit CPU dependencies	Operator-configured OpenAI-compatible quality backend

No model weights are bundled. Selecting quality-runtime does not enable a backend until its endpoint, API key, and model are configured.

Architecture

request
  │
  ├─ authentication + admission budgets
  ├─ URL / DNS / redirect / robots policy
  ├─ static fetch ───── optional Chromium render
  ├─ specialist or pinned native deterministic extraction
  ├─ optional risk-routed quality backend
  ├─ provenance + output budgets
  └─ Markdown / HTML / links / constrained JSON

Engine lineage

The generic HTML fast path is Trafilatura-family software, not a from-scratch Clusy selector. The PyO3 extension calls vendored rs-trafilatura broad 0.2.2 for the general candidate and a 9261e08-derived article backend for explicit article_body plus bounded adaptive rescue. Below the native confidence gate, the bounded local fallback set includes Python Trafilatura 2.1.0, Readability, Markdownify, documentation-specific extraction, and raw-text rescue. Default asynchronous execution compares eligible candidates in parallel; nonparallel mode tries Trafilatura first. Their exact provenance and current Clusy modifications are documented in native/vendor/NOTICE.md. None of these paths calls a hosted extraction API.

The Clusy codebase adds the service and decision layers around that selector substrate: guarded fetch and render escalation, source-family specialists, deterministic candidate admission and comparison, structure recovery, Document IR and selection receipts, crawl budgets, provenance, and output enforcement. Because the measured candidate uses an rs-trafilatura descendant, the registered AEB result is a same-family implementation/version comparison against exact Python Trafilatura 2.1.0. It measures the complete frozen Beta 2 path, not an independently originated extraction algorithm.

The main design boundaries are:

• Fetch once, escalate deliberately. Ordinary pages stay on the static path. Rendering is conditional or explicit.
• Deterministic result first. Optional quality backends cannot remove the known local fallback.
• Source-derived structure. Deterministic paths replay source text. The optional model may only label source-derived item IDs. Clusy rebuilds the pinned preprocessing, replays the complete selection, admits bounded serializer work, performs the authoritative local serialization, and authenticates a closed process-local receipt. Any failure falls back locally.
• Separate extraction from discovery. Main-content selection and the crawl frontier are independent state machines; indexing and ranking remain outside this service.
• Return provenance. Results report route, verified source-selection identity when applicable, truncation, completeness, cache, render, and per-stage timing state.
• Bound every request dimension. URL count, depth, pages, bodies, output, retries, deadlines, and concurrency are capped.

Read docs/ARCHITECTURE.md for the complete runtime contract.

Capabilities

Fetch and render

• shared asynchronous HTTP/2 connection pool;
• Brotli and Zstandard response decoding;
• manual redirect handling with SSRF validation on every hop;
• private, loopback, link-local, metadata, and other unsafe address rejection;
• bounded retries, body size, deadlines, and concurrency;
• conditional or explicit sandboxed Chromium rendering; and
• source status, redirect, render, and stage-timing provenance.

Extract

• pinned, vendored rs-trafilatura Rust/PyO3 fast-path extraction with documented Clusy modifications;
• confidence-gated Trafilatura, Readability, Markdownify, and documentation fallbacks;
• GitHub repository, file, issue, pull request, commit, and diff specialists;
• PDF and academic-paper extraction;
• optional metadata-only Crossref and configured publisher fallbacks; and
• optional schema/prompt JSON extraction.

Discover

• explicit recursion with max_depth > 0;
• deterministic bounded frontier;
• same-site scope with optional subdomains;
• robots enforcement without a request-side bypass;
• host fairness and crawl-trap budgets; and
• deterministic response ordering.

The built-in frontier is process-local. It is not a durable distributed crawl queue.

Extraction profiles

Profile	Contract	Model required
balanced	Default general main-content extraction	No
article_body	Precision-oriented article-body extraction	No
adaptive	Deterministic result first; optional quality escalation on bounded label-free risk	No
quality	Deterministic result first; attempt the configured quality backend	No

If the optional quality backend is absent, saturated, unavailable, invalid, or slower than its deadline, the deterministic candidate remains authoritative. The quality lane is disabled unless its base URL, API key, and model are all configured. Successful model output is not persisted in Redis unless an immutable backend revision is supplied and its v1 source-serialization receipt was independently replayed and authenticated. Legacy v0 receipts remain readable but are never persisted.

API

Interactive OpenAPI documentation is available at /docs; the schema is at /openapi.json.

Method	Route	Authentication when configured	Purpose
GET	/health	Public	Process liveness
GET	/health/ready	Public	Required component readiness
GET	/health/version	Public	Source, image, pipeline, and non-secret configuration identity
POST	/crawl	Bearer	Batch or bounded recursive crawl
POST	/md	Bearer	One URL to Markdown
POST	/html	Bearer	One URL to source or rendered HTML
POST	/map	Bearer	Bounded URL discovery

Example authenticated request:

curl --fail --request POST http://127.0.0.1:11235/crawl \
  --header 'authorization: Bearer replace-me' \
  --header 'content-type: application/json' \
  --data '{
    "urls": ["https://example.com/docs"],
    "extraction_profile": "balanced",
    "formats": ["markdown", "links"],
    "max_age": 0,
    "store_in_cache": false,
    "js_render": null,
    "max_depth": 0
  }'

Important request limits:

Field	Limit
urls	1–50
URL length	4,096 characters
max_pages	1–100
max_depth	0–10
formats	1–4 unique values
HTML output	At most 5 projected pages
JSON schema	100,000 bytes; depth 20
Extraction prompt	10,000 characters
/map result limit	1–5,000

max_depth=0 processes only the supplied URLs. Recursive requests require max_pages >= len(urls).

Representative response:

{
  "status": "ok",
  "results": [
    {
      "url": "https://example.com/docs",
      "markdown": "# Example\n\nExtracted content.",
      "links": ["https://example.com/docs/next"],
      "metadata": {
        "content_scope": "main_content",
        "extraction_route": "general_html",
        "rendered": false,
        "cache_status": "live",
        "cache_policy": "no_store",
        "cache_read_permitted": false,
        "cache_write_permitted": false,
        "cache_policy_revision": "crawl-cache-policy.v1",
        "completeness_coverage": "source_full"
      },
      "cached": false,
      "error": null
    }
  ],
  "total_pages": 1,
  "service_identity": {
    "schema_version": "crawl-service-identity.v1",
    "revision": "<git-sha>",
    "config_fingerprint": "<64-lowercase-hex>",
    "image_digest": "sha256:<64-lowercase-hex>"
  }
}

max_age=0 bypasses persistent result-cache reads. store_in_cache=false disables persistent result-cache writes. Use both for a live no-store crawl. The receipt describes only Clusy's persistent crawl-result cache; it is not a zero-data-retention promise for process memory, network transit, logs, target sites, browsers, or infrastructure telemetry.

Exact schemas live in app/models/requests.py and app/models/responses.py.

Configuration

Copy .env.example and change only the controls required by your runtime.

Required for production mode:

• ENVIRONMENT=prod;
• exact GIT_SHA;
• non-empty CRAWLER_API_TOKEN; and
• an independent SERVING_FINGERPRINT_KEY with at least 32 characters.

Set IMAGE_DIGEST to the immutable Docker config image ID for a host-local build, or to the OCI manifest digest on a registry-backed platform.

Optional capability	Main settings
Redis	REDIS_URL, TTL, operation timeout, entry cap
Chromium	PLAYWRIGHT_ENABLED, JS_RENDER_MODE, timeout and HTML cap
Quality backend	QUALITY_EXTRACTION_*, immutable backend revision
Structured JSON	ANTHROPIC_API_KEY, model and resource caps
Publisher metadata	ELSEVIER_API_KEY, IEEE_API_KEY
Proxy	HTTP_PROXY, PLAYWRIGHT_PROXY

All validated defaults are defined in app/config.py.

Evidence status

The Verified receipt was produced directly from a clean public source tree identical to the v0.2.0-beta.2 tag tree. Its exact Trafilatura 2.1.0 comparator ran in a separate hash-pinned, label-free environment. The result is bound to a frozen protocol, compact report, deterministic retained raw archive, and exact hashes in bench/evidence/registry.json. See docs/BENCHMARKS.md for the execution boundary, limitations, and evidence-status rules. The throughput value is one local in-memory extraction observation, not a crawler, HTTP-service, stability, or service-level result. No other benchmark, implementation, deployment, or vendor result is authorized for publication by this release.

Development

uv sync --locked --extra dev

uv run ruff check .
uv run mypy app
uv run pytest -q

cargo +1.85 fmt --manifest-path native/Cargo.toml --check
cargo +1.85 clippy \
  --manifest-path native/Cargo.toml \
  --locked --all-targets -- -D warnings
cargo +1.85 test --manifest-path native/Cargo.toml --locked

CI builds and verifies all three container boundaries. Extraction changes require the relevant fixed-protocol benchmark; native performance changes require counterbalanced retain-all evidence and output equivalence unless a quality delta is preregistered.

See CONTRIBUTING.md.

Documentation

Document	Purpose
docs/README.md	Documentation index and status vocabulary
docs/ARCHITECTURE.md	Runtime architecture and failure behavior
docs/SELF_HOSTING.md	Compose, containers, production configuration, upgrades
docs/OPERATIONS.md	Health, observability, release, rollback, incidents
docs/BENCHMARKS.md	Results, artifacts, and claim boundaries
docs/RESEARCH.md	Advanced architecture and SOTA gates
bench/README.md	Benchmark protocol and evidence index
SECURITY.md	Threat model and vulnerability reporting

License

Clusy Crawler is licensed under Apache-2.0. Third-party licenses and provenance are recorded in THIRD_PARTY_LICENSES.md and native/vendor/NOTICE.md.