robots_blocks_crawlers
Umbodi lifts a site-wide block on a major search crawler, keeps every other rule, and acts only when the owner has opted in to crawler-policy changes; it then checks the file is live.
Not fixed by Umbodi on its own site — it cannot be planted there
A Disallow of the whole site for Googlebot, Bingbot or every crawler asks the search web to forget the site. Left over from a staging build, it removes the site from search.
Fires when robots.txt disallows a major crawler — Googlebot, Bingbot or all crawlers — from the whole site. Severity: error.
The agent removes the site-wide block for that crawler inside a fenced block it owns and leaves the site's other rules untouched. Writing a new robots.txt needs the owner's chosen policy for AI crawlers (allow or block); without it the agent stops and asks.
The change reaches the site as a pull request that adds or edits a site file at the repository's web root, opened by the Umbodi GitHub App. Generated blocks sit inside fences the agent owns, so the site's hand-written lines are never rewritten.
The value is derived from the site's own data, so the change is stamped mechanical and no score bar applies. It still applies only when the owner has armed unattended acting and set this category's autonomy dial to automatic; merging needs a separate opt-in.
Because this changes what crawlers may read, there is one more gate: the owner must have opted the project in to crawler-policy changes. Without that, the agent proposes and waits.
One proposal in ten is held back as a control group — chosen by a fixed hash per project, not by hand — and is never applied unattended, so the effect of the fixes can be measured against the ones not made.
After the pull request merges and the production deploy appears, the agent fetches the live file and compares it with the merge's own diff: every line it added must be there, and a change that only removed lines must have them gone. If not, the file is re-read up to five times, about two and a half hours, and then the agent reverts its own merge.
It stops, and says why, when the live file is generated by the site's own code rather than served from the repository — a pull request would change nothing — when the file in the repository already says the same, when a robots.txt change needs a policy the owner has not chosen, or under the same merge and deploy rules as page changes. It never reverts on a file it cannot fetch.
Our site has nowhere this defect can be planted without breaking something real, so there is no live receipt. The repair path is proven in the test suite only. The reason, as the code states it:
the plant is "Disallow: / on the live robots.txt" for up to 52 hours — telling every crawler to stop reading umbodi.ai. On a site whose entire product is search and AI-answer visibility that is not a worse snippet, it is the worst available outcome, and it would also suppress the rank measurements every other drill depends on. Measured 2026-10-07: our robots.txt currently disallows 2 admin prefixes and allows everything else.
Last verified 2026-10-11 08:43 UTC
Only what the test suite proves: the route from finding to verified repair works against fixtures. Nothing on this page claims it has happened live.
It does not prove a ranking or citation change. Umbodi never infers one from a fix: rank is measured separately, on Google, and reported as measured. One receipt also shows the path works on this site, not that it works on every site.
Umbodi lifts a site-wide block on a major search crawler, keeps every other rule, and acts only when the owner has opted in to crawler-policy changes; it then checks the file is live.
Not live. Our site has nowhere it can be planted without breaking something real, so the repair is proven in the test suite only, and this page says why.
The receipts page lists every defect type the agent repairs on its own, and which of them it has repaired live on this site.