site-crawler

Open-source SEO monitoring. GSC + Site Crawler + Core Web Vitals in one dashboard.

Generate clean, AI-ready llms.txt files for your website or docs. Supports crawling, sitemaps, static builds, and framework-aware adapters (Next.js, Vite, Nuxt, Astro, Remix). Includes Markdown/MDX docs mode and robots.txt generator for LLM and search crawlers.

Free Chrome extension for comprehensive Shopify SEO analysis. 50+ checks, real Core Web Vitals, site crawler, Google rank tracker, competitor comparison, internal link map, and PDF reports. No API keys, no signup, no cost.

Precision web screenshot capture — paste a URL, capture desktop or mobile renders, discover navigation pages, bulk-export as ZIP, and integrate with a single GET request.

Omnipotent Master SEO & Security Auditor (Cyber Fusion Engine v4.0)

grandstage catalog data scraper

Crawl an entire website and surface every broken link — self-contained PHP + cURL, dark-themed HTML dashboard + CSV export.

Crawlcat public REST API: OpenAPI contract, multi-language code examples, and docs for site SEO audits — crawl sites, export Copy for AI markdown, and integrate with Cursor, Claude, or ChatGPT via MCP.

Site crawler used in Ethora platform as an option to import your specific business data into your AI agent chat bot.