Michael Janzen

Bridging Art, Design, and Technology with a Maker's Mindset

Tag: AEO

  • What AI Bots Actually See When They Crawl a WordPress Site

    What AI Bots Actually See When They Crawl a WordPress Site

    AEO Pugmill tracks how AI answer engines consume WordPress content and formats site data for those systems. AEO Pugmill operates as a network tracking how AI answer engines consume WordPress content, paired with a plugin that formats site data for these systems. AI answer engines extract and cite facts, requiring specific structuring for machine readability.

    Adding the plugin to a WordPress installation generates structured data and machine-readable endpoints. Serving specific outputs as distinct URLs allows bots to request resources independently. The trackable endpoints include a plain-text llms.txt index. This index functions as a table of contents, helping crawlers determine which pages to fetch. The system produces structured Markdown renderings of individual posts. This gives bots a clean version of the text, including publication dates, summaries, entity lists, and Q&A pairs, omitting HTML markup and theme elements.

    The plugin generates standalone JSON-LD files containing FAQPage schema, entity mentions, and citations. Updating the standard WordPress XML sitemap adds alternate links pointing to the Markdown endpoints. Additions to the robots.txt file signal the availability of the structured content index. Enriching the standard RSS feed incorporates AEO elements like structured summaries and named entities alongside the post content.

    Embedding outputs directly into the HTML places data where search engines and crawlers expect to find it. The plugin injects FAQPage JSON-LD derived from post metadata. Entities stored in the metadata become typed mentions with links to authoritative references, assisting AI systems in disambiguating subjects. Extracting external links populates the citation JSON-LD. The plugin injects structured data derived from the post summary, falling back to the WordPress excerpt. These embedded elements register as standard HTML page requests. Separating schema into standalone files reduces utility for traditional search while providing no added benefit for AI crawlers that already parse the full page. for traditional search, while providing no added benefit for AI crawlers that already parse the full page. The distinction matters for understanding the limits of bot analytics, as parsing a specific embedded element remains indistinguishable from a full page load.

    Evaluating bot activity occurs by checking incoming user-agent strings against a list of 25 recognized signatures, including GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider, DeepSeekBot, and traditional search crawlers. Identifying a match records the canonical bot name, the requested resource type, and the date in a local daily summary table. The system does not keep a per-request log. Analyzing HTML requests captures content signals like word count brackets, freshness, fact density, and URL depth. Sharing data with the wider aggregation network is an opt-in setting. Enabling this feature transmits daily count summaries using a one-way hashed identifier, ensuring no URLs, content, or user data leave the server. When a post goes live, participating search engines receive a notification through an automated ping system that respects a 30-minute burst limit between updates.

    Full architecture and technical implementation details are available at https://www.aeopugmill.com/about.

    The plugin is available for WordPress installation at https://www.aeopugmill.com/plugin.

  • Structuring Content So AI Answer Engines Cite It as a Source

    Structuring Content So AI Answer Engines Cite It as a Source

    Answer Engine Optimization (AEO) is the practice of structuring digital content so that AI answer engines — such as Claude, ChatGPT, and Gemini — select and cite it as a definitive source when responding to user queries. Unlike traditional SEO, which targets search engine rankings and clicks to a webpage, AEO targets the synthesis layer where AI generates direct answers, making citation by an AI model the primary success metric.

    The Structural Shift from Search to Synthesis

    For three decades, digital marketing centered on optimizing headers, keyword density, and backlinks to rank on search engine results pages. Users followed links to pages that might contain answers. AI answer engines collapse this process — users receive synthesized answers directly, without a required click.

    Citation frequency replaces click-through rate as the top-of-funnel objective.; becoming the source an AI cites when answering a question is the relevant goal.

    Why Users Trust AI-Generated Answers

    Erik Brynjolfsson’s concept of the Turing Trap describes a pattern where AI that closely mimics human interaction is more likely to replace human roles in a given process. Applied to marketing, this dynamic matters: because AI answer engines present a conversational, human-like interface, users tend to accept synthesized answers as authoritative without verifying the underlying source.

    When an answer engine recommends a specific product or service at the top of a results page, users treat that recommendation with a level of trust traditionally reserved for human referrals. AI models are becoming a channel for social proof and purchase influence.

    A Multi-Layered AEO Architecture

    One practical approach to AEO involves a layered technical architecture designed to make content legible to AI crawlers without degrading the human user experience. The following components form this approach:

    • Markdown system prompt file (e.g., llm.txt): A plain-text file formatted in Markdown that gives AI bots an executive summary and thesis immediately, bypassing typical website code. This file targets AI crawlers specifically and has no reported impact on standard Google Search rankings.
    • Static JSON corpus: Hosting full source material — such as a manuscript or knowledge base — as a static JSON file gives answer engines direct access to content in an AI-native format.
    • JSON-LD schema injection: Overriding generic SEO schema with specific JSON-LD markup that explicitly maps entity relationships — such as author, work, and core concepts — allows AI to process structured data efficiently.
    • Question-and-answer content structure: Formatting content directly as Q&A pairs targets high-probability queries and increases the likelihood that an AI selects the correct paragraph as a definitive answer.

    AEO and Standard Google Search

    Google has stated that it does not currently use Markdown files like llm.txt for crawling or indexing organic search results. Google Search guidance continues to emphasize optimizing for depth, clear headings, and well-structured data — content that offers a human experience an AI summary cannot replicate.

    Observed outcomes from at least one production implementation suggest AEO tactics may also influence standard SERP blue-link rankings, which conflicts with official Google messaging. This space is evolving, and ongoing testing and measurement can clarify which effects hold across implementations.

    Team Composition for the Agentic Web

    A team blending marketing, product management, and applied AI covers both campaign execution and technical implementation. Foundational marketing experience remains necessary, and supplementing existing teams with professionals who bring a blended background in marketing, product management, and applied AI covers campaign execution alongside the technical requirements of AI-indexed content.

Privacy Policy - Terms and Conditions