Crawlr
crawlr.ai
- Free plan
- Web
- API
What Crawlr is
Crawlr is a website crawler and technical SEO audit tool for finding issues in site structure, crawl efficiency, and technical setup. It discovers URLs through links, sitemaps, canonicals, and JavaScript, with controls for robots.txt, rate limits, inclusion and exclusion rules, and crawl depth. Its analysis covers response codes, indexability, metadata, headings, word counts, and internal and external links. Results can be viewed in graphs, directory trees, 3D site views, and content-cluster visualizations. Crawls can be scheduled daily, weekly, or on a custom schedule, with email completion notices. The API can start crawls, check progress, return structured JSON, retrieve historical data, and emit webhook events. The free plan includes 500 tokens per month, one concurrent crawl, CSV and JSON exports, and 15-day crawl history. Paid plans start at $8/mo (annual). Crawlr is available on the web and through an API; unauthenticated sandbox crawls are capped at 25 pages.
Who it suits
Crawlr suits site owners, SEO practitioners, and teams that need to inspect site structure or run scheduled crawls. Its listed plans also address single-domain and portfolio use, growing teams and agencies, and high-volume SEO or AI pipelines.
Good to know, and what to watch
- Discovers URLs through links, sitemaps, canonicals, and JavaScript
- Analyzes indexability, metadata, headings, and links
- Offers scheduled crawls with email notifications
- API returns structured JSON and historical data
- Free plan includes CSV and JSON exports
- Unauthenticated sandbox crawls have a 25-page maximum
- Free plan allows one concurrent crawl
- Paid plans start at $8/mo (annual)
Plans
Every plan and priceFree
Starter
Standard
Growth
Free plan
Getting started
- Visit https://www.crawlr.ai/.
- Start with the Free plan or select a paid plan.
- For API use, authenticate with an API key and keep it out of client-side code, public repositories, and logs.
- Set crawl controls such as robots.txt compliance, rate limits, URL rules, and crawl depth.
- Run a crawl or schedule it daily, weekly, or on a custom schedule.
Limits to check
The Free plan is free and includes 500 tokens per month, one concurrent crawl, and 15-day crawl history. Starter is $8 a month, billed yearly with two months free; it provides 5,000 tokens/month, five concurrent crawls, a 5,000-page crawl maximum, and three-month history. Unauthenticated sandbox crawls have a 25-page maximum.
The verdict
Crawlr combines technical SEO checks, crawl controls, visual reports, scheduling, and API access. The free plan provides a way to begin, while higher crawl capacity and export options sit on paid plans.
Questions people ask
Does Crawlr have a free plan?
Yes. The Free plan is free and includes 500 tokens per month, one concurrent crawl, CSV and JSON exports, and 15-day crawl history.
What does Crawlr cost beyond the free plan?
Paid plans start at $8/mo (annual). Starter is listed at $8 a month, billed yearly with two months free.
Which platforms does Crawlr support?
Crawlr is available through web and API.
What can the API do?
It can start crawls, check status, retrieve structured JSON, access historical data, and handle webhook events.
What integrations are listed?
Google PageSpeed Insights is listed for Starter, and webhook notifications (Beta) are listed for Growth.
What the maker publishes
- AI-oriented output
- Crawlr provides token-aware markdown summaries intended for LLM and RAG ingestion. source
- API
- The API supports starting crawls, checking status, retrieving structured JSON results, accessing historical data, and webhook events. source
- API access
- Yes source
- API capabilities
- The API can start crawls, poll status, retrieve JSON results, access historical crawls and emit webhook events. source
- API design
- The maker describes a REST API using API-key authentication, standard HTTP methods, documented JSON structures, rate limits and descriptive errors. source
- Company listing
- A LinkedIn page named Crawlr Labs describes a privately held technology company with 11-50 employees focused on competitor monitoring; the page does not identify crawlr.ai's SnappyTech maker. source
- Crawl controls
- Crawl settings include robots.txt compliance, rate limiting, include and exclude rules, and crawl depth controls. source
- Crawl graphs
- Interactive 2D force graphs, 3D graphs, tree views and filtered views visualize URL relationships and site structure. source
- Data collected
- The service collects account details, billing information, usage logs, device information and crawl data such as URLs, content, metadata, links, status codes and performance metrics. source
- Data retention
- Personal data is retained while an account is active or needed for service delivery, while crawl-data retention follows the subscription plan. source
- Deployment
- The official pages describe Crawlr as cloud-based and provide no self-hosting option. source
- DIQ score
- Its Domain Intelligence Quotient combines indexability, technical health, content quality, link structure and mobile or speed signals into a score from 0 to 100. source
- Documentation depth
- The API page provides endpoint-level capabilities, data-format descriptions, use cases and authentication, rate-limit and error-handling notes, but no separate public SDK is stated. source
- Exports
- The Free plan includes CSV and JSON exports; paid plans add LLM-ready markdown exports and higher-tier bulk export capabilities. source
- Free plan
- Yes source
- Full-site crawling
- It crawls entire websites, including subdomains, parameterized URLs and XML-sitemap URLs. source
- HTTP analysis
- The crawler tracks 2xx, 3xx, 4xx and 5xx responses and reports redirect chains, loops, soft 404s and broken links. source
- Independent listing status
- G2 says Crawlr has not yet been reviewed and lacks enough reviews to provide buying insight. source
- Independent pricing availability
- G2 states that pricing details are unavailable on its listing and directs buyers to the vendor site. source
- Independent product description
- G2 describes Crawlr as a free web-scraping tool for extracting and managing website data, which differs from crawlr.ai's technical-SEO positioning. source
- Independent seller identity
- G2 attributes the listed Crawlr product to Crawlr Labs. source
- Indexability analysis
- It checks noindex, nofollow and canonical directives and flags conflicts such as noindex combined with another canonical target. source
- Integration
- Google PageSpeed Insights is listed as the named third-party integration for performance analysis. source
- Integrations
- The pricing page lists Google PageSpeed Insights for the Starter plan and webhook notifications (Beta) for Growth. source
Checked against the maker's pages on 15 Oct 2026.