How Lightcrawl crawls your site

When someone adds a site to Lightcrawl, we crawl it to find its pages, then audit each page for performance, SEO and quality. This page explains what that looks like from your side and how to control it.

What we crawl

We start at the URL that was added and find the rest of the site two ways: your sitemap.xml (including sitemaps listed in robots.txt), and the internal links on the pages we read. We stay on the exact host that was added, treat URLs that differ only in query parameters as the same page, and skip images, video and PDFs. A site can also be set to a single page, in which case we fetch only that URL.

We only crawl sites that a Lightcrawl customer has added to their workspace.

How fast we crawl

By default we pace ourselves the way search engines do: start gently, speed up only while your server is clearly coping, and back off as soon as it isn't.

• We start at one request per second and speed up gradually, to at most four per second, while your pages keep answering quickly and without errors.
• If your pages start answering noticeably slower than they did at the start of the crawl, or return server errors (5xx) or time out, we slow down straight away and only speed up again once things recover.
• If your server answers 429, or 503 with Retry-After, or your bot protection starts challenging us partway through, we pause, slow down (as far as one request every 20 seconds), and retry. A site that asked us to slow down stays at the slower pace for the rest of the crawl. If it keeps refusing, we stop and keep the pages we already read.
• If your robots.txt sets a Crawl-delay, we wait at least that long between requests.
• Every request counts toward the pace. Besides pages, we check that the images and PDFs a page links to still exist, with a light HEAD request. Each one is checked once per crawl, however many pages use it.
• Lightcrawl customers can choose a slower fixed pace per site. A fixed pace still backs off the same way. Crawling faster than four requests per second is only available to a site's verified owner.

We send cache-revalidation headers so we read your current pages, not a stale cached copy.

robots.txt

We read and obey robots.txt before crawling. Rules for User-agent: lightcrawl (or *) apply to us. If it disallows the page that was added, the crawl stops with a "blocked by robots.txt" result rather than an empty crawl.

How to opt out

To stop Lightcrawl from crawling your site, add this to your robots.txt:

User-agent: lightcrawl
Disallow: /

No allowlist rule overrides it. The one exception is a verified owner of the site, who can choose to crawl their own site anyway from inside Lightcrawl. That requires proving ownership and never applies to anyone else's site.

Where requests come from

Our crawler runs in United States datacenters (ord, iad) from a small set of fixed IP addresses, over IPv4 or IPv6. If you restrict traffic by country, allow the United States or our addresses. The addresses, our request header and our User-Agent are listed on Allow Lightcrawl through your firewall.

We do not rotate identities, use residential proxies, or try to get past a challenge. If your site blocks us, we stay blocked until you allow us.

If a crawl comes back empty

Usually either robots.txt disallows us, or a CDN or firewall is turning our crawler away. Lightcrawl tells you which, with a link to the fix. For the second case, see Allow Lightcrawl through your firewall, or test your site with the access checker.