Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

Crawlr

crawlr.ai

  • Free plan
  • Web
  • API
7.8 Club score Visit crawlr.ai

What Crawlr is

Crawlr is a website crawler and technical SEO audit tool for finding issues in site structure, crawl efficiency, and technical setup. It discovers URLs through links, sitemaps, canonicals, and JavaScript, with controls for robots.txt, rate limits, inclusion and exclusion rules, and crawl depth. Its analysis covers response codes, indexability, metadata, headings, word counts, and internal and external links. Results can be viewed in graphs, directory trees, 3D site views, and content-cluster visualizations. Crawls can be scheduled daily, weekly, or on a custom schedule, with email completion notices. The API can start crawls, check progress, return structured JSON, retrieve historical data, and emit webhook events. The free plan includes 500 tokens per month, one concurrent crawl, CSV and JSON exports, and 15-day crawl history. Paid plans start at $8/mo (annual). Crawlr is available on the web and through an API; unauthenticated sandbox crawls are capped at 25 pages.

Who it suits

Crawlr suits site owners, SEO practitioners, and teams that need to inspect site structure or run scheduled crawls. Its listed plans also address single-domain and portfolio use, growing teams and agencies, and high-volume SEO or AI pipelines.

Good to know, and what to watch

  • Discovers URLs through links, sitemaps, canonicals, and JavaScript
  • Analyzes indexability, metadata, headings, and links
  • Offers scheduled crawls with email notifications
  • API returns structured JSON and historical data
  • Free plan includes CSV and JSON exports
  • Unauthenticated sandbox crawls have a 25-page maximum
  • Free plan allows one concurrent crawl
  • Paid plans start at $8/mo (annual)

Free

Free

Starter

$8/mo

Standard

$73/mo

Growth

$291/mo

Free plan

Getting started

  1. Visit https://www.crawlr.ai/.
  2. Start with the Free plan or select a paid plan.
  3. For API use, authenticate with an API key and keep it out of client-side code, public repositories, and logs.
  4. Set crawl controls such as robots.txt compliance, rate limits, URL rules, and crawl depth.
  5. Run a crawl or schedule it daily, weekly, or on a custom schedule.

Limits to check

The Free plan is free and includes 500 tokens per month, one concurrent crawl, and 15-day crawl history. Starter is $8 a month, billed yearly with two months free; it provides 5,000 tokens/month, five concurrent crawls, a 5,000-page crawl maximum, and three-month history. Unauthenticated sandbox crawls have a 25-page maximum.

The verdict

Crawlr combines technical SEO checks, crawl controls, visual reports, scheduling, and API access. The free plan provides a way to begin, while higher crawl capacity and export options sit on paid plans.

Questions people ask

Does Crawlr have a free plan?

Yes. The Free plan is free and includes 500 tokens per month, one concurrent crawl, CSV and JSON exports, and 15-day crawl history.

What does Crawlr cost beyond the free plan?

Paid plans start at $8/mo (annual). Starter is listed at $8 a month, billed yearly with two months free.

Which platforms does Crawlr support?

Crawlr is available through web and API.

What can the API do?

It can start crawls, check status, retrieve structured JSON, access historical data, and handle webhook events.

What integrations are listed?

Google PageSpeed Insights is listed for Starter, and webhook notifications (Beta) are listed for Growth.

What the maker publishes

AI-oriented output
Crawlr provides token-aware markdown summaries intended for LLM and RAG ingestion. source
API
The API supports starting crawls, checking status, retrieving structured JSON results, accessing historical data, and webhook events. source
API access
Yes source
API capabilities
The API can start crawls, poll status, retrieve JSON results, access historical crawls and emit webhook events. source
API design
The maker describes a REST API using API-key authentication, standard HTTP methods, documented JSON structures, rate limits and descriptive errors. source
Company listing
A LinkedIn page named Crawlr Labs describes a privately held technology company with 11-50 employees focused on competitor monitoring; the page does not identify crawlr.ai's SnappyTech maker. source
Crawl controls
Crawl settings include robots.txt compliance, rate limiting, include and exclude rules, and crawl depth controls. source
Crawl graphs
Interactive 2D force graphs, 3D graphs, tree views and filtered views visualize URL relationships and site structure. source
Data collected
The service collects account details, billing information, usage logs, device information and crawl data such as URLs, content, metadata, links, status codes and performance metrics. source
Data retention
Personal data is retained while an account is active or needed for service delivery, while crawl-data retention follows the subscription plan. source
Deployment
The official pages describe Crawlr as cloud-based and provide no self-hosting option. source
DIQ score
Its Domain Intelligence Quotient combines indexability, technical health, content quality, link structure and mobile or speed signals into a score from 0 to 100. source
Documentation depth
The API page provides endpoint-level capabilities, data-format descriptions, use cases and authentication, rate-limit and error-handling notes, but no separate public SDK is stated. source
Exports
The Free plan includes CSV and JSON exports; paid plans add LLM-ready markdown exports and higher-tier bulk export capabilities. source
Free plan
Yes source
Full-site crawling
It crawls entire websites, including subdomains, parameterized URLs and XML-sitemap URLs. source
HTTP analysis
The crawler tracks 2xx, 3xx, 4xx and 5xx responses and reports redirect chains, loops, soft 404s and broken links. source
Independent listing status
G2 says Crawlr has not yet been reviewed and lacks enough reviews to provide buying insight. source
Independent pricing availability
G2 states that pricing details are unavailable on its listing and directs buyers to the vendor site. source
Independent product description
G2 describes Crawlr as a free web-scraping tool for extracting and managing website data, which differs from crawlr.ai's technical-SEO positioning. source
Independent seller identity
G2 attributes the listed Crawlr product to Crawlr Labs. source
Indexability analysis
It checks noindex, nofollow and canonical directives and flags conflicts such as noindex combined with another canonical target. source
Integration
Google PageSpeed Insights is listed as the named third-party integration for performance analysis. source
Integrations
The pricing page lists Google PageSpeed Insights for the Starter plan and webhook notifications (Beta) for Growth. source

Checked against the maker's pages on 15 Oct 2026.