I use “AI agent optimization” here to mean making your website easier for AI systems to find, understand and use. That includes the assistants that search the web and cite pages in their answers, and the browser agents that click buttons, fill out forms and complete tasks on a person’s behalf. The phrase is also used to mean using AI agents to do your SEO. That’s a different subject.
The short answer is that most of it is work that was always good practice. Put real text where people and machines can read it. Make sure your important content is in the page itself, not only loaded later by scripts. Use proper buttons, links and labelled form fields. Say plainly what you do, who you do it for and what it costs. And don’t block the bots you want to be found by. Google’s own guide for developers puts it in one line: “Everything we suggest to make a site ‘agent-ready’ also makes sites better for humans.”
This article is for owners and leadership teams who keep hearing that their website needs to be “ready for AI agents” and want to know what that actually involves, and how urgent it really is.
What I said the day ChatGPT’s agent launched
I made a video the day OpenAI released ChatGPT agent in July 2025. I’d spent a lot of time with OpenAI’s earlier tool, Operator, and as I said at the time, it “was a pretty freaking horrendous experience honestly. What it did do was it gave me that vision for the things to come.”
@iamrodneywarner ChatGPT Agent Rollout: Why You Need AI Optimization Now OpenAI just combined deep research with browser automation, creating AI that can think AND act on websites. But most sites fail basic AI usability tests. Here's what AI Agent Optimization means and why you have weeks, not months, to fix it. #ChatGPTAgent #AIOptimization #AIAgentOptimization #BizTok #AIRollout #WebsiteOptimization #AEO ♬ original sound – iamrodneywarner
The core of what I said was that there are two jobs: “You need to make sure that AI can find you and you also need to make sure that AI can use your website.” And that “all these things that the AI agents need aren’t new. Web and accessibility experts have been preaching this stuff forever.”
That part has held up well. The products and the pace have moved around a lot more.
What happened in the year since
Here’s how the last year went, in order:
- July 2025. OpenAI launched ChatGPT agent, which combined its research tool with the ability to operate a browser.
- September 2025. OpenAI launched Instant Checkout, letting people buy some products without leaving ChatGPT.
- October 2025. OpenAI launched Atlas, its own web browser with an agent mode built in.
- January 2026. Google added an “auto browse” feature to Chrome that can click, scroll and fill out forms, for its paying US subscribers.
- March 2026. OpenAI ended Instant Checkout and moved purchases back to retailers’ own checkouts. Walmart, one of the biggest participants, said purchases made inside ChatGPT converted at about a third of the rate of shoppers it sent to its own site.
- Summer 2026. OpenAI retired both Atlas and ChatGPT agent, folding browsing into the ChatGPT app, a Chrome extension and a cloud browser that runs on its servers.
Meanwhile other agents kept arriving. Anthropic’s Claude has an extension for Chrome, Perplexity has its Comet browser, and Microsoft has been testing a Copilot mode in Edge. Earlier this year I wrote to my team that I’d been “playing around with Claude’s Chrome extension and Perplexity’s new Comet browser. Pretty freaking wild stuff.”
So agents are real and getting more capable. They just haven’t become the main way most people use the web, and the specific products have changed faster than any website could chase them. That’s the best argument for doing the durable basics and skipping anything built around one company’s tool.
The two jobs: be found, and be usable
It helps to separate the two kinds of AI visitor, because they need slightly different things.
AI search tools (ChatGPT search, Perplexity, Google’s AI Overviews and AI Mode, Copilot) use search indexes, their own crawlers and sometimes a live fetch of a page to find information for their answers. Google’s AI features, for example, are built on its regular Search index and ranking systems. For these, the job is being found: your pages have to be crawlable, indexed and clear about what you do.
Browser agents (Chrome’s auto browse, Claude in Chrome, Comet and the like) drive a real browser to complete a task: compare options, fill out a form, book something. For these, the job is being usable: the buttons, links and forms on your site have to work for something that isn’t a person with a mouse.
Most of the work serves both, which is why it’s worth doing once and doing properly.
How AI agents actually read your website
Google’s guidance for developers describes three ways an agent sees a page:
- Screenshots. It takes a picture of the page and uses a vision model to work out what’s on it.
- The raw HTML. It reads the code that makes up the page.
- The accessibility tree. This is the structured version of the page that the browser builds for screen readers and other assistive technology: what’s a heading, what’s a button, what each form field is for. Google calls it “a high-fidelity map that ignores the visual ‘noise’ of CSS.”
That third one is why accessibility keeps coming up. Agents and assistive technology benefit from many of the same things: real text, labelled fields, proper headings and controls that are what they look like. OpenAI said something similar about its Atlas browser while it existed, that it used the labels and roles that support screen readers to understand a page. Accessibility and agent readiness aren’t the same thing, but a site built well for accessibility starts with a lot of the right ingredients.
There’s one difference worth knowing about JavaScript. Browser agents run a full browser, so they see what a person sees after the page loads. Crawlers may not. In Vercel’s December 2024 testing, the OpenAI, Anthropic and Perplexity crawlers it measured didn’t run JavaScript, while Google’s did. Those systems can change, which is a good reason not to design around any one crawler’s current behavior. The durable rule is simpler: don’t make your most important information depend on scripts running in the browser. Your services, prices and contact details belong in the HTML your server sends.
Job one: let the right bots in
Each AI company runs more than one bot, and they do different jobs. Blocking one isn’t the same as blocking another.
| Bot | Company | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages to show in ChatGPT search results |
| GPTBot | OpenAI | Collects content that may be used to train models |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user asks it to |
| Claude-SearchBot | Anthropic | Improves Claude’s search results |
| ClaudeBot | Anthropic | Collects training data |
| Claude-User | Anthropic | Fetches a page when a Claude user asks |
| PerplexityBot | Perplexity | Indexes sites for Perplexity’s answers |
| Perplexity-User | Perplexity | Fetches a page when a user asks |
| Googlebot | Crawls for Search, including AI Overviews and AI Mode | |
| Google-Extended | A setting, not a crawler: controls use in Gemini training and grounding |
Three things matter for a business decision.
The search bots and the training bots are separate. You can decline to have your content used for training and still be found in AI search. On my AI SEO page I put it this way: “If your site blocks the crawlers an assistant uses for search, your own pages can’t be among its sources.” For most B2B companies that want to be recommended, blocking the search bots is a mistake.
Google-Extended doesn’t control AI Overviews. Those come from Googlebot, like the rest of Search. Blocking Google-Extended keeps your content out of Gemini’s training and grounding, not out of Google’s AI answers.
The block may not be in robots.txt at all. Robots.txt is the file that tells bots what they may crawl, and it’s the first place to look. But as my agency’s team has written, some hosting platforms and security services now block AI crawlers by default, often without the site owner knowing, and those blocks are invisible from the front end of your website. Cloudflare, which sits in front of a large share of websites, has changed its AI bot controls more than once. It now sorts AI traffic into search, agent and training categories, and since September 2026 new domains block agent and training bots by default on pages that carry advertising, while search stays allowed. Settings like these change, so don’t assume robots.txt tells the whole story. Check robots.txt first, then the bot settings in your hosting, CDN or firewall.
For what it’s worth, the robots.txt on my own site blocks nothing but the WordPress admin area. My site also sits behind a CDN, which is exactly why the second check matters.
Job two: make your site usable by a machine
This is the list I’d hand a web team. None of it is new, which is the point.
- Real text, not text in images. Phone numbers, addresses, prices and service names should be text a machine can read.
- Content in the HTML. What you do, who you serve, what it costs and how to reach you should be in the page the server sends, not only in content loaded by scripts, tabs or pop-ups.
- Buttons that are buttons and links that are links. A clickable box built from generic code looks like a button to a person and like nothing at all to the accessibility tree.
- Labelled forms. Every form field needs a proper label, so an agent knows which box is the email and which is the company name. A form that only works after several layers of script is hard for agents and assistive technology alike.
- Links that say where they go. “Click here” and “Learn more” tell a machine nothing. “See pricing for website projects” tells it everything.
- A stable layout. Agents that work from screenshots get confused when things move around as the page loads, and Google specifically warns against invisible overlays that sit on top of buttons.
- Say the obvious out loud. I’ve written before that you should “state your positioning as a fact that a machine can index.” Who you serve, what you do, where you work and roughly what it costs. On my own site, prices are written in plain text on the Work With Me page, so an agent asked what I charge can answer from the page.
- Facts that match everywhere. Agents and AI search tools compare what your site says with what other sources say. If your company profiles disagree on your city, your founding year or what you do, the machine’s picture of you gets muddier. If you serve several areas from one office, say so plainly, so service-area pages aren’t mistaken for offices.
Structured data helps, but it isn’t magic
Structured data (often called schema markup) is code that labels the facts on a page for machines: this is a business, this is its address, this is a service, this is its price range. It’s worth having. On my site, each service page says in structured data what the service is, who provides it and where.
But be clear on what it does. Google’s May 2026 guide says that “structured data isn’t required for generative AI search, and there’s no special schema.org markup you need to add,” while still recommending it as part of good SEO. My agency’s own technical SEO page calls it “defensible best practice for entity clarity,” and says the verdict on whether it dramatically moves AI citations is still out. Do it because it’s good hygiene. Don’t buy it as the thing that gets you recommended.
One practical warning: after structured data is added across a site, check a few individual pages. Template defaults have a way of leaking onto pages they don’t describe.
What about llms.txt?
llms.txt is a proposed file that summarizes a website for AI tools, a bit like a sitemap written for language models. It’s been heavily promoted, and the evidence for it is thin.
Google Search says plainly that you don’t need it: its guide says you don’t need “new machine readable files, AI text files, markup, or Markdown” to appear in its AI features. Chrome’s newer Lighthouse checks can look at an llms.txt file if you have one, but that isn’t the same as Google expecting it.
My position for an owner is simple: don’t make it a priority. If your team can add it cheaply, fine. I wouldn’t pay anyone meaningful money for it, and I wouldn’t let it jump ahead of the basics above.
What about WebMCP?
WebMCP is more interesting than llms.txt, and earlier. It’s a proposed standard that would let a website tell an AI agent directly which actions it can perform, like submitting a form, searching inventory or starting a booking, without the agent having to work it out from the screen. Chrome is experimenting with it, and its new agent-readiness checks in Lighthouse already look for it.
For a typical B2B marketing site, I wouldn’t build around it yet. If you run ecommerce, software or another site where an agent completing a transaction really matters, have your development team keep an eye on it.
What I wouldn’t overreact to yet
The new acronyms
Is SEO dead now that AI answers the question? No. Google’s own guide is direct about AEO (answer engine optimization) and GEO (generative engine optimization): “From Google Search’s perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO.” What has changed is where the decision happens. As I put it in a LinkedIn post this year, “By the time someone lands on your site, they’re not discovering you. They’re confirming you.”
Claims that AI visitors convert better
The evidence conflicts. Adobe’s US retail data found traffic from generative AI tools converting worse than other traffic early in 2025 and better by that year’s holiday season. A study of 973 ecommerce sites found ChatGPT referrals converting worse than organic search. Anyone quoting one number as settled is overselling. For a B2B company, the better question is whether the people who arrive can quickly confirm you’re the right fit. As my AI SEO page says, “People aren’t making $100,000 purchasing decisions just because ChatGPT told them so.”
Agentic commerce, unless you sell online
AI agents buying products for people is real, and still moving around. OpenAI’s in-chat checkout lasted about six months, and Walmart said purchases made inside ChatGPT converted at about a third of the rate of shoppers it sent to its own site. Google announced its Universal Commerce Protocol in January 2026 with retailers including Shopify, Etsy, Wayfair, Target and Walmart, and it powers checkout inside Google’s AI experiences for eligible US retailers. If you run ecommerce, keep your product data clean wherever it’s syndicated and watch these programs. For most B2B and service companies, there’s nothing to act on yet.
Three things your web team should be able to show you
You don’t need to learn any of this in depth. You need to ask for three pieces of evidence. The word that matters is “show”: it turns questions that can be hand-waved into something you can see.
- “Show me which AI bots we allow and block, and where.” That means search bots, user-triggered fetchers and training bots, in robots.txt and in the hosting, CDN or firewall settings. “I don’t think we block anything” isn’t an answer.
- “Show me our important pages without JavaScript.” Ask them to load the page with scripts turned off, or show you what a crawler receives. Your services, prices, proof and contact options should all be there.
- “Run Lighthouse and show me what fails.” Lighthouse is built into Chrome. Ask for the accessibility audit and, where it’s available, the newer Agentic Browsing checks, then what it would take to fix the important items.
If the answers come back as a large new “AI optimization” project with a new acronym and a big invoice, be skeptical. Google’s own guide suggests weighing outside AEO and GEO advice against its official guidance. Most of what this takes is the kind of work a competent web team should already be doing.
How urgent is this, really?
Less urgent than the launch-day headlines suggested, and more lasting.
Google’s guide frames agent readiness as something to look at “if this is something that’s relevant to your business and you have extra time.” That’s a fair read for most companies. The bot settings and the content-in-the-HTML question are worth checking this month, because a mistake there can quietly keep you out of AI answers. The rest is good website practice that pays off for human visitors at the same time, so it belongs in your normal site work.
Common questions about AI agent optimization
What is an AI browser agent?
An AI tool that operates a web browser on someone’s behalf: it can open pages, click, scroll, fill out forms and complete multi-step tasks. Examples include Chrome’s auto browse, Claude in Chrome and Perplexity’s Comet.
What is the accessibility tree?
The structured version of a page that a browser builds for screen readers and other assistive technology. It identifies headings, links, buttons and form fields and what each one is for. AI agents use it to understand and operate a page, which is why accessibility work helps with agents too.
Should we block AI crawlers?
For most B2B companies that want to be recommended, leave the search crawlers alone. You can block training crawlers like GPTBot and ClaudeBot separately if you don’t want your content used for training.
How do I optimize my website for ChatGPT?
Make sure OAI-SearchBot can reach your site, put your key content in the HTML, state clearly what you do and for whom, and keep your facts consistent across the web. ChatGPT also relies on other sources about you, so the site is only part of it.
Do I need llms.txt?
Not to appear in Google’s AI features, according to Google. It’s optional and cheap, with no proven benefit yet.
Where to go from here
You probably don’t need an “AI agent optimization” project. You need a website that machines can reach, understand and operate without guessing.
Check the bot controls in both places. Keep important information in readable HTML. Build forms, buttons and links properly. Make the facts about your company explicit. Then keep an eye on newer standards like WebMCP as they mature.
If you’d like a second set of eyes, a technical check of whether AI search tools can reach and read your site is part of my AI SEO consulting work. You can start a conversation on Work With Me.
