A native Claude Code plugin for crawling live websites with Crawl4AI and extracting compact Markdown or text with Trafilatura.
cc-crawl4ai gives Claude Code one crawling skill, four runtime-management skills, three specialized agents, and a bundled Python command-line runner. It is deliberately not an MCP server: Claude invokes a local executable through Bash, and the executable launches pinned Crawl4AI and Trafilatura tools without a shell.
Web pages rendered for browsers contain navigation, repeated menus, cookie/UI text, scripts, styling remnants, and other boilerplate that an AI agent usually does not need. Sending all of it into an agent's context wastes tokens, increases cost, and leaves less context for reasoning and source code.
The primary goal of cc-crawl4ai is to extract the useful page content before Claude reads it. Crawl4AI renders the page, Trafilatura removes boilerplate locally, optional BM25 keeps only query-relevant sections, and the plugin returns an artifact path instead of injecting the entire crawl into the conversation.
Claude Code's features overview is served as 503,770 bytes of raw HTML. Each output mode turns that same page into:
| Mode | Output size | Words | Tokens¹ | Reduction vs. regular Markdown |
|---|---|---|---|---|
| Crawl4AI Markdown | 33,971 bytes | 3,959 | 8,269 | baseline |
Crawl4AI fitted Markdown (markdown-fit) |
24,963 bytes | 3,549 | 5,675 | 31.4% fewer |
| Crawl4AI + Trafilatura Markdown | 14,340 bytes | 2,243 | 3,255 | 60.6% fewer |
| Crawl4AI + Trafilatura text | 13,154 bytes | 1,898 | 2,805 | 66.1% fewer |
That is roughly 5,500 fewer tokens for one page in compact text mode. The fitted-Markdown row is the honest intermediate baseline: Crawl4AI's own heuristic filter already removes some boilerplate, and Trafilatura roughly doubles that saving. Results vary by site — content-heavy pages shrink less, and pages with extensive navigation shrink more. BM25 filtering can reduce the artifact further when the agent needs only one topic from a long document.
¹ Methodology. Counted with the cl100k_base tokenizer as a portable proxy; Claude's tokenizer differs, so treat these as indicative rather than exact. Sizes are the bytes of the saved artifact, so they exclude the raw HTML that never reaches the conversation. The common four-characters-per-token rule of thumb understates the benefit here (it estimates 57.8% and 61.3%), because boilerplate-free prose packs more characters per token — 4.09 chars/token for regular Markdown versus 4.67 for Trafilatura text.
Reproduce it with plugin 0.1.2, Crawl4AI 0.9.2, and Trafilatura 2.2.0 (measured 2026-08-11):
URL=https://code.claude.com/docs/en/features-overview
crawl4ai crawl "$URL" --output-format markdown --bypass-cache
crawl4ai crawl "$URL" --output-format markdown-fit --bypass-cache
crawl4ai crawl "$URL" --extractor trafilatura --output-format markdown --bypass-cache
crawl4ai crawl "$URL" --extractor trafilatura --output-format text --bypass-cacheLive pages change over time, so absolute numbers drift; the ordering of the modes is the durable result.
Use it naturally:
Read https://code.claude.com/docs/en/features-overview with Trafilatura and summarize the useful content.
Or request the smallest representation:
Read https://code.claude.com/docs/en/features-overview as compact plain text to minimize agent tokens.
- Reads one page into compact Trafilatura Markdown or plain text.
- Saves regular Crawl4AI Markdown, fitted Markdown, JSON, or complete output.
- Performs bounded BFS, DFS, or best-first multi-page crawls.
- Filters single-page Markdown/text into query-relevant structural chunks with local BM25 scoring.
- Extracts structured JSON through Crawl4AI's LLM or deterministic CSS/XPath strategies.
- Keeps large results in project-local artifacts instead of injecting full pages into the conversation.
- Installs Crawl4AI and Trafilatura in separate, pinned virtual environments.
- Claude Code with plugin and marketplace support.
- Python 3.10 or newer, including the standard-library
venvmodule. - Network access to install Python packages/browser binaries and to crawl target sites.
- Platform libraries required by Crawl4AI and its browser runtime.
- About 2 GB of free disk space for isolated environments and browser binaries.
- A separately configured Crawl4AI LLM provider only for question mode and LLM JSON extraction.
The pinned compatibility set for plugin 0.1.2 is Crawl4AI 0.9.2 and Trafilatura 2.2.0. The runner handles POSIX and Windows virtual-environment layouts/process termination, but its process-group tests are POSIX-only and browser availability still depends on upstream host support. See Installation.
Run these commands inside Claude Code:
/plugin marketplace add romek-rozen/cc-crawl4ai
/plugin install cc-crawl4ai@romek-plugins
If requested, run /reload-plugins. Then install the isolated Python and browser runtime:
/cc-crawl4ai:crawl4ai-install
The install skill asks before downloading dependencies. A first installation normally takes 5–15 minutes, depending on internet speed and local caches, and may download more than 500 MB of packages and Chromium binaries. The installer displays numbered stages and elapsed time throughout the process. Its recommended runtime is persistent ${CLAUDE_PLUGIN_DATA}, so updating the plugin does not discard it.
Check the installation:
/cc-crawl4ai:crawl4ai-status
/cc-crawl4ai:crawl4ai-test
The test accesses https://example.com and verifies the complete Crawl4AI → Trafilatura pipeline.
For project/custom scopes, update and removal instructions, and exit-code details, read Installation.
Ask Claude naturally:
Read https://example.com as compact Markdown and summarize it.
Crawl https://docs.example.com broadly, with a limit of 10 pages.
Find the sections about refresh tokens at https://example.com/docs.
Extract product names and prices from https://shop.example.com as JSON.
Or invoke the main skill directly:
/cc-crawl4ai:crawl4ai https://example.com
The default compact-page command produced by the skill is equivalent to:
"${CLAUDE_PLUGIN_ROOT}/bin/crawl4ai" crawl "https://example.com" \
--project-root "${CLAUDE_PROJECT_DIR}" \
--runtime-root "${CLAUDE_PLUGIN_DATA}" \
--extractor trafilatura \
--output-format markdown${CLAUDE_PLUGIN_ROOT}, ${CLAUDE_PLUGIN_DATA}, and ${CLAUDE_PROJECT_DIR} are Claude Code runtime placeholders. The shell example is mainly for debugging a loaded plugin; use a real path when running the bundled CLI directly from a checkout.
| Component | Purpose |
|---|---|
/cc-crawl4ai:crawl4ai |
Route a crawl, scrape, question, BM25 filter, or extraction request. |
/cc-crawl4ai:crawl4ai-install |
Install or update pinned Python runtimes and browser dependencies. |
/cc-crawl4ai:crawl4ai-status |
Probe both executables and report browser verification separately. |
/cc-crawl4ai:crawl4ai-test |
Smoke-test the real pipeline against example.com. |
/cc-crawl4ai:crawl4ai-clear-cache |
Remove only project .crawl4ai/cache and .crawl4ai/robots. |
cc-crawl4ai:crawl4ai-scrape |
Specialized one-page Trafilatura agent. |
cc-crawl4ai:crawl4ai-crawl |
Specialized bounded deep-crawl agent. |
cc-crawl4ai:crawl4ai-extract |
Specialized structured-JSON extraction agent. |
Management skills cannot be invoked automatically by the model. Install and test workflows request confirmation before network downloads/access, and cache clearing requests confirmation before deletion.
| Goal | Required options | Important constraints |
|---|---|---|
| Compact page | --extractor trafilatura |
One page; Markdown/md/text only. |
| Local relevance filter | --bm25-query "terms" |
One page; Markdown or text; no question/extraction/deep crawl. |
| Deep crawl | --deep-crawl bfs --max-pages 10 |
Explicit positive page cap required; Markdown, fitted Markdown, or all. |
| LLM JSON | --output-format json --json-extract "instruction" |
Requires Crawl4AI provider configuration. |
| CSS/XPath JSON | --output-format json --schema-path ... --extraction-config ... |
Both files required; no LLM required. |
| Question | --question "..." |
Single page; may require a configured provider. |
See Usage for complete examples and extraction configuration, and CLI reference for every option, incompatibility, runtime lookup rule, artifact path, and exit code.
By default, output is stored beneath the active project:
.crawl4ai/outputs/<domain>/<format>/<local-timestamp>-<url-slug>.<extension>
Trafilatura uses the trafilatura/ format directory and also saves sibling *.raw.html. Existing names are never overwritten: the runner adds a deterministic suffix when necessary. An explicit relative --output-file is resolved under the project root; an absolute path is accepted as-is.
The runner prints saved paths. Claude's skill/agents then read only the relevant artifact. Outputs, raw HTML, runtime state, and upstream crawler caches can contain sensitive information; protect the project and runtime directories accordingly.
Claude Code
└── cc-crawl4ai plugin
├── skills/ routing and managed workflows
├── agents/ scrape, deep-crawl, extraction specialists
└── bin/crawl4ai validation and orchestration
├── crwl subprocess Crawl4AI 0.9.2
├── trafilatura process Trafilatura 2.2.0 (optional)
└── local BM25 filter no external model (optional)
There is no resident service, npm runtime package, or MCP transport. package.json is private repository metadata and validation convenience only.
- Installation and marketplace lifecycle
- Usage and examples
- CLI reference
- Development, architecture, and releases
- Troubleshooting
- Security and privacy
- Plugin-local README
npm test
npm run validate
claude plugin validate plugins/cc-crawl4ai
claude plugin validate .See Development for local loading, version synchronization, dependency compatibility testing, and the manual marketplace release workflow.
Crawling is network access to untrusted content. This plugin validates modes and URLs, uses argv-based subprocess execution with shell=False, bounds diagnostics, and terminates child process groups on timeouts where supported. It does not provide an SSRF sandbox, robots-policy enforcement, authorization to scrape, prompt-injection isolation, or secret redaction. Read Security before using it in sensitive networks.
Please report vulnerabilities privately as described in that manual rather than opening a public issue.
If this project helps you, you can support its development through GitHub Sponsors or Patreon. Funding links are also configured in .github/FUNDING.yml.
MIT © Roman Rozenberger