Docs
Documentation
Guides for getting the most out of Lightcrawl.
Guides for getting the most out of Lightcrawl: how our crawler works, and how to let it through if your site blocks automated traffic.
Crawler
- Allow Lightcrawl to crawl your site: Recognize our crawler and allowlist it in your CDN, WAF, or firewall.
- How Lightcrawl crawls your site: Crawl scope, request rate, sitemaps, and how we honor robots.txt.