For many years we had two audiences for our websites: humans, and bots. Humans needed an intuitive, user-friendly interface that helped them find the content or actions they were looking for; and bots like Googlebot needed clean, semantic, machine-readable files that they could glide through to understand what a site was about.

Now, we’re at a different stage of the web, and there’s a new kind of visitor who also needs attention: AI agents. Yes, AI agents are just another type of bot, but the way they operate and “think” is different, and therefore their needs are unique.

With agents in the picture, we’re seeing that more and more requests to websites are coming from software sent there by agents to read pages, answer questions, and do things on behalf of people. In my last post, I wrote about why WordPress was born ready for this: feeds, a REST API, structured content, and two decades of letting other software read and write a site prepared WordPress to be a leading solution for the agentic web. In this post we’ll take a look at the web as a whole: how it’s changing, who’s writing the new rules, and what agent ready actually means. In the next post, I’ll get into the actions you can take already so your website is agent ready and ahead of the curve.

Bots now outnumber people

In June 2026, Cloudflare CEO Matthew Prince published findings showing that automated traffic had passed human traffic. Cloudflare Radar put bots at about 57% of requests for HTML pages on Cloudflare’s network, versus about 43% from people, and the trend continued over the following months. Prince had initially projected that this shift would only happen in 2027, but the agentic web is moving faster than expected.

People are visiting less

While this data does leave some open questions (like it’s not always possible to differentiate a bot from a person, and a lot of that bot traffic is crawlers, not agents doing tasks on behalf of humans), what is clear is that people are visiting sites less. LLMs and agents now answer many questions directly, so a person can get what they need from a site without actually viewing it, which means that in more and more cases agents are visiting, and humans aren’t.

We’ve built for bots before

Optimizing sites for this new reality reminds me a lot of how we built our sites for SEO: making sure we had readable URLs (like WordPress’ “pretty permalinks”), optimized title tags, sitemaps, alt text on images, and readable content offered primarily in indexable HTML. We were building for bots then too, ones that crawled pages and indexed them, and the bots we’re building for now, agents, do the same but also read, answer questions intelligently, and take actions.

Who’s writing the rules for the agentic web?

From the inception of the web, it has adhered to openness. It’s thanks to this open approach, and standards crafted over years to keep it that way, that the web developed into the wild and wonderful thing that it is. Just as no one owns the web, nobody owns the agentic web either, and its rules are also being written in the open by a mix of standards bodies, browser makers, AI companies, platforms, and individual developers.

Standards bodies and browser makers

Some of that work happens in more formal settings: the W3C’s Web Machine Learning Community Group is developing WebMCP, which engineers at Google and Microsoft first drafted together in August 2025; and the Internet Engineering Task Force (IETF) has a working group on AI Preferences, a shared vocabulary for what AI may do with content, and another on bot identity that we’ll look at below.

Browser makers are also exploring these agentic directions, and each of them takes a stand on new proposals: Chrome and Edge are both testing WebMCP with real sites through origin trials; Safari and Firefox don’t support it.

Apple’s WebKit team and Mozilla, have published formal positions on it: WebKit opposes it, and Mozilla’s position is labeled neutral.

Grassroots proposals

A lot of this work also starts less formally and in a more grassroots way. Jeremy Howard proposed llms.txt on a website of its own in 2024, while Cloudflare published Content Signals and a draft for listing Agent Skills on GitHub. Anthropic released MCP as an open protocol in late 2024, and it was adopted quickly by the other AI companies.

Audits and checkers

The tools that check sites shape what gets adopted, too. In May 2026, Google’s Lighthouse added an Agentic Browsing category that checks if a page works well for agents: whether its buttons and fields have names an agent can read, whether the layout stays still so an agent doesn’t click the wrong thing, whether the site has an llms.txt file, and whether the page registers WebMCP tools and marks up its forms for them.

In September 2026, Lighthouse 13.5 added a check for AI catalogs, which are files at /.well-known/ai-catalog.json that list a site’s agent services, such as its MCP servers. Instead of a score out of 100, the Agentic Browsing category shows how many of its checks a page passes. Note that the whole category is currently labeled as experimental. Chrome’s DevTools for agents can also run Lighthouse from inside a coding agent, so the agent can audit a page, fix what it finds, and run the audit again.

Cloudflare’s Is It Agent Ready? scores a site’s agent files and robots.txt, and gives you a prompt for your coding agent to fix each failing check. The Website Specification collects all of these checks, and many more, into one checklist that you can run against your site, with each item tagged required, recommended, optional, or avoid.

However, despite all this work, no new standards have emerged as officially required, and it’s all still very much in flux.

What agent ready means

OK, we know agent interactions with our websites are increasing, but how do we get ready, or “agent ready”, for them? 

Let’s look at how sites are built today to try to understand it. A web page is a set of instructions for “drawing” something. The HTML says where the heading goes, the CSS says how it looks, and JavaScript often builds half the page after it loads. A person looking at the page can see a product, a price, and an “Add to cart” button, while an agent sees thousands of nested elements and has to work out which number is the price and which is the shipping cost, the old price, or the review count.

Agents deal with this in one of two ways:

  1. Tools like the web tools in ChatGPT and Claude fetch a page to answer a question: they download the HTML and convert it to markdown text, discarding the layout.
  2. Browser agents, like Gemini in Chrome, Claude in Chrome, or ChatGPT’s agent mode, read the page’s accessibility tree or take screenshots, then click through the site the way a slow, careful person would.

Both depend on the agent reading the page correctly, but they don’t (or can’t) always.

Why can’t tools or agents always accurately capture and understand web pages? There are many reasons, including all the things standing between an agent and the content like cookie banners, CAPTCHAs, bot protection that blocks anything that doesn’t look like a normal browser, content that only appears after scrolling, and forms whose error messages are red text with nothing in the code indicating which field they’re referring to. A person can mostly work around all of these barriers relatively easily, but an agent often gets stuck, or gives up and fills in the gap with a (bad) guess.

Read, query, act

An agent ready site makes three things easy for an agent: reading, querying, and acting.

Reading means getting the content without having to dig it out of convoluted navigation, ads, and theme markup. In addition, all of that stuff related to the visual appearance of a page can often make it several times the size of its actual content. Since agents burn tokens trying to find what they’re looking for, these larger pages can end up being expensive, so offering the content to an agent in an efficient, optimized way saves money, and also saves room in their context windows.

Querying means asking a specific question, like “when is the next event?” or “is this in stock in size 9?”, and getting a direct, structured answer rather than having to search through a page.

Acting means submitting a form, booking a time, or adding something to a cart.

Reading and querying are mostly solved, and a lot of what makes a site useful for agents is work you may already be doing for accessibility. An agent that falls back to browsing reads the accessibility tree, the same structure a screen reader uses, so real headings and buttons that are actually real buttons help a blind visitor and an agent through the same code. Making a site accessible for people is reason enough on its own, but it has never been the only benefit: semantic markup has helped with search for twenty years, and agents are the newest type of bot that is assisted by it.

Acting is where the web has the furthest to go: an agent usually has to find a form’s fields on screen and type into them, which breaks whenever the design changes. Labeled form fields help, since an agent can find a field by its label instead of its position on the page, but a named action with a clear description and typed inputs, like “search products by keyword and maximum price,” keeps working however the page looks.

Knowing which agents to let in

Being agent ready isn’t just about letting all agents in, but rather letting the right agents in, and that really matters once agents aren’t only reading sites, but start acting on them. A site would be interested in welcoming a shopper’s assistant, but would also want to block a scraper, and to do that it would need to be able to tell them apart.

In addition, site owners have the right to know who and what exactly is crawling and accessing their site, but that’s hard to do today, because a bot only introduces itself with its user-agent string, which is a line of text sent with every request that the bot writes itself. Nothing prevents a script from calling itself GPTBot or Chrome, and I’ve seen this in action on my own site, where one address called itself Chrome on a Mac, then node, then GPTBot, all within 12 seconds.

When names can’t be trusted, sites end up blocking based on guesses, which can backfire. For example, an agent readiness scanner tested my site by pretending to be OpenAI’s GPTBot. Cloudflare noticed the request didn’t actually come from OpenAI and rightfully blocked it, but the scanner then reported that my site blocks AI agents – a problem that wouldn’t have happened if it had been honest about who it was!

It is possible to identify a lot of bots by comparing a request’s address with the address ranges that OpenAI, Anthropic, Google, and others publish for their crawlers. However, while that works for crawlers running on a company’s own servers, it won’t for an agent running in someone’s browser or on their own laptop. The fix being worked on is Web Bot Auth, an IETF draft in which an agent signs each request with a private key and the site checks the signature against a public key that the agent’s company publishes, so a bot trying to pretend it’s associated with a particular company won’t pass.

Web Bot Auth is already in use: OpenAI’s ChatGPT agent signs its requests this way. In October 2025, Visa and Cloudflare announced the Trusted Agent Protocol, which builds on Web Bot Auth so stores can recognize shopping agents acting for real customers.

Agent ready doesn’t mean open season

Note that “agent ready” doesn’t mean it has to be open season for AI agents on your content. Some site owners think that “make your site readable by AI” means “let AI take all your stuff,” but being hard for agents to read doesn’t stop your content from being used, and mostly just leaves you out of the answers people now get instead of visiting. Letting agents read your site and letting AI companies train on it are separate decisions, and you can decide to do either or both, as I’ll show in my next post.

Building for both audiences

As I mentioned earlier, websites have always had two audiences: people, and the bots that index pages for them. Agents are bots, but because they reason about what they read, in some ways they behave more like a human audience. They sit somewhere in between, reading and acting on a site the way software does, but on behalf of a specific person who wants an answer or needs something done.

SEO already taught us how to build for these types of audiences, and what to avoid. The tricks aimed only at bots, like keyword stuffing and cloaking, which meant showing search engines different content from what people saw, worked for a while but eventually got sites penalized. The sites that did well in the long run were first and foremost good for people, which usually meant they were also easy for machines to read (a well built, semantic and accessible site is easier for bots to read too).

I expect agent readiness to go the same way. Sites that are well built and already have machine-readable content (like WordPress, as I wrote in my previous post) will be easier for agents to use, and agent files and markdown versions of pages need to match what people see, not cloak a different message for the bots.

A chance to shape the standards

The world of SEO also shows how standards for bots come together. For years each search engine did things its own way, until Google, Yahoo, and Microsoft agreed on the sitemaps protocol in 2006 and on schema.org in 2011. Both were published for anyone to use, but the search engines worked them out among themselves first. Agents are in a similar stage now, with one difference: much of the work is happening in public while it’s still a draft. Anyone can comment on a GitHub issue, join a W3C community group, or post to an IETF mailing list. The WordPress community runs a big share of the sites these specs are about, and we can take part while the drafts are still works in progress.

In my next post, I’ll go through the agent ready approaches that are catching on, from llms.txt and markdown versions of pages to MCP servers, and what you can do on your own site now.