HyperlambdaCan AI read your site?
Give it a URL. It reads your robots.txt and works out which AI crawlers you allow and which you turn away — citing the exact rule that decides each one — then checks whether your content survives without JavaScript, whether you publish an llms.txt, and what the page costs a model to read.
Reads /robots.txt, /llms.txt and the page itself. Usually a second or two.
What this can and cannot tell you
The crawler verdicts come from your published robots.txt, evaluated the way crawlers are specified to read it: an exact user-agent group beats the * group outright, the longest matching path wins inside it, Allow beats Disallow on a tie, and an empty Disallow: means allow everything. No robots.txt at all means allowed. Each verdict shows the rule it came from, so you can check the working rather than trust a badge.
What it cannot see is anything decided above your site — a firewall or CDN rule that turns crawlers away without ever mentioning it in a file. So a clean result here means no block found in robots.txt, not a guarantee that every crawler gets through. This tool also never pretends to be a crawler: it evaluates the rules you published rather than spoofing a user agent to slip past them.
Haven't got an llms.txt? Generate one from your sitemap. Want the page itself as clean model input? URL to Markdown. Built with the Hyperlambda Generator, callable by your own agents over MCP.
The whole backend, printed
The backend for this one is deliberately tiny, and worth being honest about: the robots.txt reasoning you see above runs in your browser, not on the server. All the server does is fetch a URL on your behalf — which the browser cannot do for another site's robots.txt. This is that endpoint in full:
.arguments
url:string
validators.mandatory:x:@.arguments/*/url
http.get:x:@.arguments/*/url
yield
status:x:@http.get
content:x:@http.get/*/contentDescribe an endpoint in a sentence, and the Hyperlambda Generator writes and deploys it — then it is a tool your own agents can call over MCP.
More free tools
Checking whether AI can read your site is one of nine free tools here, all of them built the same way and callable by your own agents. See all of them — no signup on any of it.
Questions
Should I be blocking AI crawlers?
That is a business decision, not a technical one, and both answers are defensible. Blocking protects content you sell or licence; allowing is how you get cited when someone asks an assistant about your field. What is rarely deliberate is the third case — blocking by accident, through a rule someone added years ago or a default that shipped with a plugin. This tool exists to make that visible.
My content is missing without JavaScript. Does that matter?
For AI crawlers, yes. Google renders JavaScript before indexing; most AI crawlers currently do not, and simply take whatever HTML the server returns. If that HTML is an empty shell, you are invisible to them no matter what your robots.txt says.
Why does the page show what a crawler receives?
Because a word count is arguable and the actual text is not. If the extract below your results is three words and a cookie banner, that is precisely what a model gets when it reads your page.
Is this the same as a robots.txt tester?
It uses the same rules, but it asks a narrower question: not "is this path crawlable" but "can these named AI agents read this site, and does anything else stop them". The agent list, the JavaScript check and the llms.txt check are the parts a general tester will not tell you.