Strict server firewalls and CDNs can block Indexing Insight's scheduler.
We fetch your sitemaps with a user-agent containing indexinginsight.
You will need to ask your IT or dev team to whitelist our user-agent so that our scheduler can fetch your XML Sitemaps.
Our tool will only crawl your XML sitemaps and we will only fetch and download the XML sitemap files selected in the tool.
We do not crawl your website or the important pages that you submit.
How to whitelist Indexing Insight in Cloudflare
The most common reason our tool gets blocked is because of Cloudflare.
Go to your Domain → Security → WAF → Custom rules and select Create rule.
Give it a descriptive name (e.g. "Allow Indexing Insight UA").
Under When incoming requests match, set:
Field: User Agent
Operator: wildcard
Value: indexinginsight
Under Choose action, select Skip, then tick which features this user-agent should bypass (e.g. Managed Rules, rate limiting rules, Super Bot Fight Mode, etc).
Make sure the rule is set to First (rules are taken into account from top to bottom).
Deploy the new rule to your Cloudflare account.
