Category: AEO

I’m collecting data on AI crawlers and search spiders and learning what they like to eat. It’s true that few are eating LLMS.txt, but it doesn’t tell the full story.

To have good SEO, you might not need to add structured content, but some formats, like JSON-LD, question-answer pairs, and named entities, do seem to help and attract visits. Google’s recent report on how to optimize for answer engines is likely because it is uncommon for websites to include this type of content.

But since optimization is the name of the game, don’t just listen to what Google says you should do; run your own experiments and see what the bots visit.

If you run a WordPress-powered website, grab my free open source plugin and see for yourself. It is called AEO Pugmill.

Fun fact: This blog was the first to run my AEO plugin.

This is a quick thought experiment, and also an example of writing content that helps humans but it also tailored for AI Crawlers.

Question: Why does AI keep referencing your competitor instead of you?

Backstory: I spotted this question on Reddit and answered it. The problem is emerging because human behavior is shifting from using search to find links to places that might provide answers, to using AI to get those answers. Afterall that is the core purpose of search in the first place, to find what (the answer) you’re looking for.

Answer: Try this little experiment asking an AI to explain which it prefers, your competitors content or your content.

1. Visit the competitor homepage and your homepage (or multiple corresponding pages on each site to create a matching set).
2. Grab the page source (e.g., View > Page Source).
3. Copy and paste the code into a simple text editor.
4. Drop those text files into an LLM and ask it to analyze without leading questions. AI crafts answers based on probability calculations and attempts to give you the answer it thinks you want. When you give it leading questions, just like a person, it can sway it’s answer toward the result you wanted to hear, so craft your prompt to make it clear your objective is to determine the truth and it’s true opinion without knowing your bias.
5. Ask it which website does a better job for answer engines and explain how and why.
6. Then ask how you could use the learnings to improve your site.
7. Implement the impriovements.

If your site runs on WordPress I made a little free plugin that helps you fine tune content and put it in a form AI Crawlers seem to like. Answer Engine Optimization is still evolving so the content types (endpoints) it creates just for the bots is experimentation.

Visit AEOPugmill.com to get the WordPress plugin and see the data I’m collecting on bot behavior.