Docs

Documentation

Guides for getting the most out of Lightcrawl.

Guides for getting the most out of Lightcrawl: how our crawler works, and how to let it through if your site blocks automated traffic.

Crawler

Allow Lightcrawl through your firewall: Our IP addresses, header, and User-Agent, and how to allow them in Cloudflare, Akamai, Imperva, AWS WAF, and more.
Check if we can crawl your site: Enter your URL and we will tell you whether anything is turning our crawler away.
How Lightcrawl crawls your site: Crawl scope, request rate, sitemaps, and how we honor robots.txt.