no_ai_crawler_policy
Umbodi adds the owner's chosen policy for AI crawlers to robots.txt in a fenced block, leaves the site's own rules untouched, publishes it as a pull request, and checks the file is live.
Fixed by Umbodi on its own site · planted drill
Answer engines read a site through crawlers such as GPTBot and ClaudeBot. A robots.txt that names none of them leaves them on a default nobody chose — allowed or blocked by accident.
Fires when robots.txt names none of GPTBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot. Severity: info.
The agent writes the owner's policy — allow or block — for each AI crawler inside a fenced block it owns, and reads the existing file first so every hand-written rule is kept. With no policy chosen, it stops and asks.
The change reaches the site as a pull request that adds or edits a site file at the repository's web root, opened by the Umbodi GitHub App. Generated blocks sit inside fences the agent owns, so the site's hand-written lines are never rewritten.
The value is derived from the site's own data, so the change is stamped mechanical and no score bar applies. It still applies only when the owner has armed unattended acting and set this category's autonomy dial to automatic; merging needs a separate opt-in.
One proposal in ten is held back as a control group — chosen by a fixed hash per project, not by hand — and is never applied unattended, so the effect of the fixes can be measured against the ones not made.
After the pull request merges and the production deploy appears, the agent fetches the live file and compares it with the merge's own diff: every line it added must be there, and a change that only removed lines must have them gone. If not, the file is re-read up to five times, about two and a half hours, and then the agent reverts its own merge.
It stops, and says why, when the live file is generated by the site's own code rather than served from the repository — a pull request would change nothing — when the file in the repository already says the same, when a robots.txt change needs a policy the owner has not chosen, or under the same merge and deploy rules as page changes. It never reverts on a file it cannot fetch.
We planted this defect on our own site and let the agent find it. Each stage below is a timestamp read from the receipts, not a claim typed into this page.
| Page | planted | found | proposed | approved | published | merged | verified |
|---|---|---|---|---|---|---|---|
| /robots.txt | 2026-10-08 09:16 | 2026-10-08 09:23 | 2026-10-08 09:23 | 2026-10-08 09:23 | 2026-10-08 09:23 | 2026-10-08 09:40 | 2026-10-08 10:00 |
robots.txt- User-agent: GPTBot - User-agent: ClaudeBot - User-agent: PerplexityBot 5 line(s) removed, 0 added
robots.txt+ # umbodi:generated:start + # Added by Umbodi — the rules above are yours and are left untouched. + # Answer engines are allowed, deliberately: being cited is distribution. 0 line(s) removed, 27 added
This receipt was recorded before 11 October 2026, when the after-deploy check confirmed only that the changed file loaded. The defect was re-checked on the live site on 11 October with the audit rule itself, and is fixed.
Last verified 2026-10-08 10:00 UTC
That the full chain ran with no human decision inside it: the agent found the defect, decided the fix, published it, merged it and verified the live site.
It does not prove a ranking or citation change. Umbodi never infers one from a fix: rank is measured separately, on Google, and reported as measured. One receipt also shows the path works on this site, not that it works on every site.
Umbodi adds the owner's chosen policy for AI crawlers to robots.txt in a fenced block, leaves the site's own rules untouched, publishes it as a pull request, and checks the file is live.
Yes. We planted it on our own site and the agent found, fixed, merged and verified it with no human decision in the loop. The receipt is on this page.
The receipts page lists every defect type the agent repairs on its own, and which of them it has repaired live on this site.