Allow Lightcrawl through your firewall

Lightcrawl identifies itself three ways. Add an allow rule for them in your CDN or firewall and our crawler gets through. Most people only need the first one.

Which should I use?

• Allow the IP addresses. This works in every firewall, covers every request we make, and is the only fix for rules that block datacenter or low-reputation IPs. It is also the one that cannot be faked.
• Allow the header too if your firewall can match request headers. It is a good second rule, but anyone can send a header, so do not rely on it alone for anything security-sensitive.
• Avoid matching only the User-Agent. Our Lighthouse and screenshot passes send a plain Chrome User-Agent, so a User-Agent rule lets the crawl through and can still block your performance scores.

Wherever you add the rule, put it at your outermost layer (usually your CDN) and above your bot-blocking rules. Firewalls stop at the first rule that matches, so an allow rule below a block rule never runs.

Check that it worked

1. Run the access checker. It requests your homepage from our crawler's own network and should say your site is reachable.
2. Re-run the crawl from your Lightcrawl dashboard. The "blocking our crawler" banner clears once we can read your pages.

Still blocked? See Still blocked below.

Instructions for your platform

Jump to yours:

• Cloudflare
• Akamai
• Imperva / Incapsula
• AWS WAF + CloudFront
• Fastly
• Ezoic
• Sucuri
• Wordfence
• DataDome, HUMAN, Kasada
• Vercel and Netlify
• Managed WordPress hosts
• nginx, Apache, ModSecurity
• Shopify, Wix, Squarespace

Cloudflare

Go to Security → WAF → Custom rules, create a rule, and use Edit expression to paste:

ip.src in {209.71.89.141 2a09:8280:e626:1:0:108:20b5:0 209.71.81.125 2a09:8280:e618:1:0:108:20b5:0}

Set the action to Skip and tick everything it offers: remaining custom rules, managed rules, rate limiting, Super Bot Fight Mode, and Browser Integrity Check. Drag the rule to the top.

• Bot Fight Mode (the free-plan toggle under Security → Bots) ignores custom rules. Turn it off, or add each IP under Security → WAF → Tools → IP Access Rules with the action Allow.
• I'm Under Attack mode challenges everyone. Set the security level back to Medium or High while we crawl.

Akamai

In Security → Security Configurations, open the configuration for your hostname:

• App & API Protector / Kona: add our IPs to a Network List and reference it from an exception on the firewall policy.
• Bot Manager: define a custom bot matching the X-Lightcrawl-Crawler header, put it in a custom category, and set that category to Allow.

Changes go through staging then production, so give it a few minutes.

Imperva / Incapsula

Add our IPs under Settings → General → Allowlists. Imperva blocks unfamiliar datacenter traffic aggressively, so the IP allowlist is usually what fixes it. Optionally also allow the lightcrawl/1.0 User-Agent under Security → Bot Access Control.

AWS WAF + CloudFront

AWS keeps IPv4 and IPv6 apart, so create two IP sets under WAF → IP sets:

IPv4:  209.71.89.141/32 209.71.81.125/32
IPv6:  2a09:8280:e626:1:0:108:20b5:0/128 2a09:8280:e618:1:0:108:20b5:0/128

In the web ACL on your distribution or load balancer, add a rule matching either IP set with the action Allow, at a lower priority number than your Bot Control, IP reputation, and rate-based rules.

Fastly

In the Next-Gen WAF, add a site rule matching our IPs (or the X-Lightcrawl-Crawler header) with the action Allow, above your bot signals. Also check Fastly rate limiting: a polite crawl can still trip a low per-IP threshold on a large site.

Ezoic

Ezoic can answer automated traffic with an empty page (HTTP 202). It checks the source IP, so add our IP addresses to the crawler allowlist in your Ezoic dashboard (under Settings → Security or Cloud settings, depending on account age). If it offers a "bypass Ezoic for this crawler" option, use that.

Sucuri

Add our IPs under Settings → Access Control → Allowlist IP Addresses. Then check the Security tab: aggressive bot blocking and emergency DDoS protection can still challenge us.

Wordfence

Go to Wordfence → Firewall → All Firewall Options and add our IPs to Allowlisted IP addresses that bypass all rules. That covers the firewall, rate limiting, and crawler heuristics at once.

DataDome, HUMAN, Kasada, and similar

Bot-management products sit outside your WAF, so add an allow rule for our IPs in the vendor's own console, with the action Allow rather than "monitor". If your plan does not allow that, ask their support to allow Lightcrawl and send them this page.

Vercel and Netlify

On Vercel, add a rule in Project → Firewall → Custom Rules matching our IPs (or the x-lightcrawl-crawler header) with the action Allow, above any bot or rate-limit rules. Attack Challenge Mode overrides custom rules, so turn it off while we crawl. On Netlify, check traffic and rate-limit rules and any edge function that filters User-Agents.

Managed WordPress and app hosts

WP Engine, Kinsta, Cloudways, SiteGround, Flywheel and similar hosts run their own bot filter in front of WordPress, so a plugin setting cannot reach it. Open a support ticket asking them to allowlist our IP addresses for your domain and link this page.

nginx, Apache, and ModSecurity

If your site sits behind a CDN, allowlist there instead: your server only sees the CDN's address. Otherwise, exempt our IPs from your bot rules.

nginx:

# Requests from Lightcrawl skip your bot rules.
geo $lightcrawl {
    default 0;
    209.71.89.141 1;
    2a09:8280:e626:1:0:108:20b5:0 1;
    209.71.81.125 1;
    2a09:8280:e618:1:0:108:20b5:0 1;
}

# Then, in the rule that blocks bots:
#   if ($lightcrawl) { ...skip... }

ModSecurity:

SecRule REMOTE_ADDR "@ipMatch 209.71.89.141,2a09:8280:e626:1:0:108:20b5:0,209.71.81.125,2a09:8280:e618:1:0:108:20b5:0" \
    "id:10001,phase:1,pass,nolog,ctl:ruleEngine=Off"

Apache: add a Require ip line for each address ahead of your bot rules. Also check fail2ban or other log-based banning, which can mistake a crawl of a large site for a scan.

Shopify, Wix, Squarespace, and other hosted platforms

You cannot add firewall rules on these, and they rarely block us. If a crawl comes back empty, check for a password-protected or "coming soon" store, a robots.txt rule, or an app that adds bot filtering.

Still blocked?

The usual reasons, most common first:

• Only the header or User-Agent is allowed. Many blocks are based on the IP address and never look at headers. Allow the IP addresses.
• A second layer took over. Rate limiting, a separate bot-management product, "under attack" modes, geo restrictions, or your host's own filter each need their own exception.
• The rule is too narrow. It must cover the whole site, including /robots.txt and your sitemaps, which we fetch first.
• Every non-browser request is refused. Some products judge the connection itself (its TLS fingerprint) before reading any header. Only an IP allowlist gets past that.

If none of that helps, email support@lightcrawl.com with your domain and roughly when the crawl ran. We will tell you exactly what response we got.

Technical reference

For writing precise rules, reading logs, or verifying who we are.

What each kind of request sends

Lighthouse and screenshots use a stock Chrome User-Agent on purpose: they measure your page as a visitor gets it, and a site that treats identified bots differently would skew the scores. Audits run through Google PageSpeed Insights come from Google, with Google's own User-Agent, so nothing you allowlist for Lightcrawl affects them.

Full User-Agent

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 lightcrawl/1.0 (+https://lightcrawl.com/bot)

The Chrome version changes as we keep it current. Match lightcrawl/1.0, not the whole string.

Why IPv6 is on the list

Our crawler connects over IPv6 whenever your site publishes an IPv6 address (most CDNs, including Cloudflare, do by default) and falls back to IPv4 otherwise. You cannot predict which a given request will use, so allow both. Google, Bing and other major crawlers publish IPv6 ranges for the same reason.

Machine-readable lists

• /ips.json: the IP addresses as CIDR prefixes, in the {creationTime, prefixes[]} format Google, Bing and OpenAI publish. Point allowlist automation here.
• /bot.json: name, User-Agent token, header, IPs, and contact.

These addresses are static. If they ever change, both files and this page change with them, and creationTime in /ips.json moves.

Test by hand

The checker does this for you from our own network. To test your rules from your side:

curl -sS -o /dev/null -D - \
  -H 'X-Lightcrawl-Crawler: 1' \
  -A 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 lightcrawl/1.0 (+https://lightcrawl.com/bot)' \
  https://example.com/

Real HTML with a 200 means you are fine. A 403, 406, 429, or an empty 200/202 where a browser gets the page means something is blocking. A 5xx a browser also gets is a site outage, not a block. Your own machine is not on our IP list, so this tests header and User-Agent rules only.

Which layer is blocking

The response headers name it:

• cf-ray, cf-mitigated, server: cloudflare: Cloudflare
• x-ezoic-cdn, x-middleton-response: Ezoic
• server: AkamaiGHost, x-akamai-transformed: Akamai
• x-sucuri-id, x-sucuri-cache: Sucuri
• x-iinfo, or an incap_ses/visid_incap cookie: Imperva
• x-amz-cf-id: CloudFront, usually with AWS WAF
• None of these: your own server, a security plugin, or your host's bot filter

If more than one layer is involved, fix the outermost first.

How we crawl, and opting out

We obey robots.txt, pace our requests, and never try to get around a block. To stop us entirely, add User-agent: lightcrawl / Disallow: / to your robots.txt; no allow rule overrides it. Crawl rate, scope and the rest are in How Lightcrawl crawls your site.