Docs
Documentation
Guides for getting the most out of Lightcrawl.
Guides for getting the most out of Lightcrawl — how our crawler works, and how to let it through if your site blocks automated traffic.
Crawler
- Allow Lightcrawl to crawl your site — Recognize our crawler and allowlist it in your CDN, WAF, or firewall.
- How Lightcrawl crawls your site — Crawl scope, request rate, sitemaps, and how we honor robots.txt.