Your Firewall Blocks the AI Crawlers Marketing Needs

A marketing team spends four months rewriting service pages for AI answer engines. Clean entity facts, consistent pricing, tight schema. Three weeks after launch, the pages still do not appear in any assistant's answers. The content is fine. The problem is that a security vendor shipped a default bot rule in a quarterly update, and the retrieval crawlers that feed those assistants now get a 403 before they ever see the page.

Nobody made that decision. That is the whole issue. Crawler access is now a business permission with revenue attached, and in most organisations it has no owner, no approval record, and no verification step. It gets set by whoever touched the CDN last.

Four places your AI crawler policy is actually written

Most owners assume crawler access lives in one file. It lives in four, and each one can override the others depending on request order.

• robots.txt. A request, not a wall. Well-behaved crawlers read it and obey. It is the only layer marketing usually controls, and the only layer that cannot actually enforce anything.
• CDN or WAF bot rules. Cloudflare, Akamai, Fastly, AWS WAF. This layer enforces. It returns 403s, challenges, or silent blocks before the request reaches your server. It is owned by IT or an outsourced security provider, and it ships with defaults that change without notice.
• Server and application rate limits. Nginx limits, host-level throttles, managed WordPress bot protections. These do not block by name. They block by frequency, so a crawler that behaves fine at ten requests a minute gets cut off at eleven, and the failure looks random.
• Platform and plugin settings. Security plugins, SEO plugins, hosting dashboards. One toggle labelled something like "block AI scrapers" can rewrite robots.txt and add server rules simultaneously, and the person who flipped it was solving a bandwidth complaint.

Each layer was configured by a different person at a different time for a different reason. None of them logged a business justification. That is how a site ends up allowing GPTBot in robots.txt while the WAF blocks it at the edge, which means you get the training exposure you probably did not want and lose the retrieval you did.

Blocking by name stopped matching how AI companies operate

Search Engine Journal reported on September 18, 2026 that Cloudflare launched Bot Preference Sync, which lets sites set AI crawler policy by purpose category rather than crawler by crawler. The feature is useful. What it exposed is more useful. When most sites map their existing allow and block rules onto purpose categories, the result describes no coherent position at all. They block a search retrieval bot and permit a training bot from the same company, because someone copied a blocklist from a forum post in 2024.

Three purpose categories matter now:

1. Training. The crawler ingests your content to improve a model. No direct traffic, no citation, no near-term revenue. Blocking this is a defensible business choice.
2. Search retrieval. The crawler builds the index an assistant queries when a user asks a question. Blocking this removes you from AI answers entirely.
3. Live agent fetch. A user asked a question, and the assistant fetches your page in real time to answer it. This is the closest thing to a warm lead in AI discovery.

SEJ reported in the same week that Cloudflare had split training access from search access, and separately that Google is piloting AI-mediated payment flows. Read those two together. A single blanket block written in a security review can remove a business from organic AI answers and from emerging paid AI surfaces in one move, and the person who wrote it was thinking about bandwidth.

The crawler access register

The fix is administrative, not technical. You need one document that says what is allowed, who approved it, when, and why. Keep it in the same place as your change log. One line per agent.

Each line carries six fields: agent name, purpose category, decision, approver, date, business reason. Written out, a line looks like this: OAI-SearchBot / search retrieval / allow / J. Okafor, Head of Marketing / 2026-09-22 / feeds ChatGPT answers for commercial service queries.

A workable starting register for most businesses:

• Allow, search retrieval: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot. These decide whether you exist in AI answers.
• Allow, live agent fetch: ChatGPT-User, Perplexity-User, Claude-User. These fire when a real person is mid-question.
• Owner decision, training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent. Allow or block, but record the reason either way.
• Default block: unidentified scrapers and high-volume crawlers with no stated purpose.

Agent names change. Verify current user agent strings against each provider's published documentation before you write the register, then set a quarterly review date. For multi-site groups, the register lives once at the group level and each site records only its exceptions, so a franchise or brand portfolio does not end up with twelve contradictory policies nobody can reconcile.

Verify that the policy you wrote is the policy running

A register is a claim. Verification turns it into evidence. Run three checks, in this order.

Read your server logs for the last 30 days. Filter by user agent and record which AI agents actually requested pages, how many requests each made, and what status code they received. A crawler you believe you allow but that appears zero times in 30 days is being blocked somewhere upstream. A crawler returning 403s or 429s tells you which layer is doing it: 403 points to the WAF, 429 points to a rate limit.

Run the snippet retrieval test. Chris Green published a free version of this on September 18, 2026. Take a distinctive sentence from the page you care about, one that appears nowhere else on the web, paste it into two chatbots, and ask for an exact match and the source URL. If the assistant can return the sentence and the URL, the page is reachable. If it cannot, retrieval is broken regardless of what your robots.txt says. Use two assistants, because one provider's index lag is not the same as a block.

Record expected versus live. SEOGOD's W39.1 standard requires both halves on every approved change. One line per agent: expected status from the register, live status from logs and the snippet test, date checked, who checked it. That line is what you produce six months later when someone asks why organic AI referrals dropped in March.

Crawler access governance checklist

• Pull the current robots.txt and read the AI agent directives line by line.
• Export current bot management rules from the CDN or WAF, including vendor defaults you did not write.
• List server and host-level rate limits that apply to non-browser traffic.
• Audit security and SEO plugins for any AI blocking toggle, and note its current state.
• Reconcile all four layers against a single purpose-category policy.
• Name one accountable owner for crawler access, with a named deputy.
• Write the access register with approver and date on every line.
• Pull 30 days of logs and record which agents actually fetched pages.
• Run the snippet test on your three highest-value pages in two assistants.
• File the expected versus live proof line with the register.
• Add crawler access to the pre-launch checklist for every CDN migration, security update, and plugin install.
• Set a quarterly review to catch renamed agents and new vendor defaults.

What to fix first

Do these three in order, in one sitting.

1. Run the snippet test on your single highest-revenue page. Five minutes, no tooling, no access required. If two assistants can quote it back with the URL, retrieval works and you have time to do the rest properly. If neither can, you have an active revenue leak.
2. Check the WAF, not robots.txt. Enforcement happens at the edge. Pull the bot management rules and look specifically for vendor-default AI categories enabled during a platform update. This is where the silent blocks live.
3. Name the owner today. Before you touch a single rule, write down who signs off on crawler access changes. An unowned permission reverts to vendor default at the next update, and you will be running this audit again in six months.

Access is a governed permission now

Marketing owns the consequence of crawler policy and security owns the controls. That split is why the decision keeps getting made by accident. The change-control rule closes it: no security update, CDN migration, or plugin install goes live without the crawler access owner signing off and the verification loop rerunning afterward. Expected status, live status, date, initials.

That is the same discipline SEOGOD applies to every other change in the discovery stack, where an approved patch is not finished until live evidence confirms it. If you want to see which layer is currently blocking you before you build the register, the free SEO tools will show you what a crawler sees, and a free audit will tell you whether your highest-value pages are reachable at all.