Crawlability and Indexing

AI bots for search engines like ChatGPT, Perplexity, or Grok don’t scan web pages like traditional search engines, instead, they read, understand and build connections of your website’s content.

The ratio of bot hits to human visitors has sharply increased. Cloudflare data shows PerplexityBot's activity climbed by over 250% in 2025, while GPTBot’s traffic saw a 305% surge compared to 2024.

• Googlebot/Gemini is the best for ongoing search referral and (still) sends the most users/citations

• PerplexityBot is best if you want to be cited in direct answers

• GPTBot/ClaudeBot are vital for training, but don’t send much search traffic

  
  

robots.txt Configuration

This is the first file that the AI crawlers check. Any misconfiguration here directly blocks AI from accessing the website content. Some misconfigurations that can reduce your AI visibility drastically:

• Forgetting to allow GPTBot, Bingbot (used by ChatGPT and others), PerplexityBot, etc., to crawl your website. It’s important to double check access.

• Accidentally disallowing key directories which includes important content

• Blocking XML sitemaps. This leads to bots missing critical pages

• Not updating for new AI bots. You should add them by name User-agent: * with allow statements as they may be blocked by default

Recommended configuration:

# OpenAI (ChatGPT, GPT-4)
User-agent: GPTBot
User-agent: ChatGPT-User
Allow: /

# Anthropic (Claude)
User-agent: ClaudeBot
User-agent: Claude-Web
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

# Google AI (Gemini, Bard)
User-agent: Google-Extended
Allow: /

# Common Crawl (feeds many AI models)
User-agent: CCBot
Allow: /

# Meta AI
User-agent: FacebookBot
Allow: /

# Cohere
User-agent: cohere-ai
Allow:

Sitemaps and site structure

Your sitemaps act as roadmaps for AI bots by telling them which pages matter the most (the ones you want them to cite in their answer) and how often they’re updated.

Recommended practices for an AI-optimized sitemap:

• Assign <priority> to your pages to indicate which pages are important (e.g. FAQs, core articles, product pages)

• Indicate last modified date using <lastmod> . This is super critical as AI prefers fresh content.

• A complimentary tag to include alongside <lastmod> is <changefreq> to tell the bot how frequently this content changes

The arrangement of your pages, menus and links should be easy to understand without any hidden pathways. For example, your navigation menu should reflect the main site topics and the sub-topics should be connected with parent topics. Users and bots should reach any key page within a few clicks (ideally ≤3).

Few of the critical errors you should be mindful about:

• Check your site for broken links which leads the bots to missing or deleted pages.

• Validate link routes to avoid redirect loops.

• Check server errors that hinder pages to load.

• Using a diagnostic tool like Screaming Frog could be helpful here.

Pre-rendering and renderability

Most AI crawlers have severe limitations when processing JavaScript. Key things you should be mindful about when working with JS heavy website:

• AI bots like GPTBot, PerplexityBot and ClaudeBot typically execute minimal or no JS.

• They have a strict timeout (5-10 seconds)

• Cannot process complex client-side routing

• Fail with lazy-loaded content.

Static site generation

Static site generation is one of the recommended methods for optimal SEO and AEO. In this method, all the HTML pages for your website are created ahead of time, during the build process. This means we don’t wait for the user to visit the website. When a user visits, it serves a pre-built, ready-to-go HTML file.

Pre-rendering for bots

Another effective method is to pre-render website content for bots. We can generate and cache static HTML snapshots for bots and serve interactive versions to users. The typical process of how it works is:

1. Bot requests page

2. Middleware detects bot user-agent

3. Server checks if cached snapshot exists

   1. If yes: serve cached HTML

   2. If no: render page, cache HTML, serve to bot

4. Human users receive normal JavaScript app

Quick crawlers tests

• View Page Source: Right-click → View Source. Is your content there? If not, it's JS-dependent.

• Disable JavaScript: Chrome DevTools → Settings → Disable JavaScript. Can you see content?

• Screaming Frog Crawl: Set rendering to "Text Only" mode. What does the crawler see

• Google URL Inspection: Compare "Raw HTML" vs "Rendered HTML" tabs

• AI Crawler Simulation: Use curl to fetch your page:

  
  

  curl -A "GPTBot/1.0" <https://yoursite.com/page>

Benchmark metrics (source)

• Time to First Byte (TTFB): <200ms ideal, <500ms acceptable

• First Contentful Paint (FCP): <1.5s

• Largest Contentful Paint (LCP): <2.5s

Schema and structured data

Schema markup is a piece of code that comprises of your website’s structured data. This is added as custom HTML in your website’s code. When LLMs crawl your website, it helps the bots to read website content as well as guide them to the right information.

JSON-LD implementation

The most common way to define the schema is using the JSON-LD (JavaScript Object Notation for Linked Data) format.

There are three mandatory elements

• @context ⇒ This defines the vocabulary we’re going to use for the entire schema. The most commonly used is Schema.org

• @type ⇒ specifies the type of entity (Article, Product, Organisation, Person, etc)

• Properties ⇒ these are the descriptive attributes that we use to define the entity. In the below example “headline” and “description” are properties

  
  

  
  

  Example:
  
  <script type="application/ld+json">
  {
    "@context": "<https://schema.org>",
    "@type": "Article",
    "headline": "Complete Guide to Technical SEO for AI",
    "description": "Learn advanced technical SEO strategies for AI answer engines"
  }
  </script>

Different types of schemas

Article schema for all blog/content pages

{
  "@context": "<https://schema.org>",
  "@type": "Article",
  "headline": "What Is Technical SEO?",
  "description": "A quick overview of technical SEO essentials.",
  "author": {
    "@type": "Person",
    "name": "Jane Smith"
  },
  "datePublished": "2025-10-16",
  "mainEntityOfPage": "<https://yoursite.com/technical-seo-overview>"
}

• headline: Title for snippet creation and knowledge card extraction.

• description: Used for summarizing and outlining answer context.

• author: Signals E-E-A-T (expertise, authority, trust); helps AI cite the right person.

• datePublished: Allows AI to reference the timeliness of content.

• mainEntityOfPage: Identifies the canonical source page for citation.

FAQPage schema

{
  "@context": "<https://schema.org>",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What is FAQ schema?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "FAQ schema is structured data format that lets AI search engines and bots extract questions and answers for direct display in results."
      }
    }
  ]
}

• mainEntity: Directly represents FAQs for extraction and citation.

• Question: AI matches user queries with these explicit Q&A entries.

• acceptedAnswer: Lets AI pull precise, authoritative answers from your own content.

HowTo schema on instructional content

HowTo schema is defined specifically for guide type content. This is one of the content types that get cited by the AI very often alongside FAQs. If you want to read more about what content gets cited, you can read The Content Writing Playbook That Helped Our Customers 3x Their Visibility in AI Search

{
  "@context": "<https://schema.org>",
  "@type": "HowTo",
  "name": "How to implement FAQ Schema",
  "description": "Step-by-step guide for adding FAQ structured data.",
  "step": [
    {
      "@type": "HowToStep",
      "name": "Add JSON-LD script",
      "text": "Insert a JSON-LD block into the page's HTML."
    },
    {
      "@type": "HowToStep",
      "name": "Validate schema",
      "text": "Use Google's Rich Results Test to confirm it's error-free."
    }
  ],
  "image": "<https://yoursite.com/images/faq-schema-example.png>"
}

• name/description: Define the task and context for AI to match step-by-step results.

• step: Gives AI actionable steps for answer extraction or voice synthesis.

• image: Enables visual answer cards or media citations in search/AI results.

Speakable schema on key content sections

Speakable schema tells voice assistants and AI readers which portions of your content should be read aloud. This is critical as voice search and AI audio responses grow.

{
  "@context": "<https://schema.org>",
  "@type": "Article",
  "headline": "Technical SEO Best Practices for Voice and AI",
  "speakable": {
    "@type": "SpeakableSpecification",
    "cssSelector": [
      ".summary",
      ".key-takeaway",
      "h2"
    ]
  }
}

• @type: Typically an Article, WebPage, or other content type.

• headline: Used for voice summary and AI answer card context.

• speakable: Tells voice assistants and AI which parts of your page should be read aloud or used for answer extraction.

  • cssSelector: Specifies which sections (by CSS selector → e.g., specific headings, paragraph classes, summaries) are optimized for voice output.

  
  

Entity Mapping and Knowledge Graph

For AI to build holistic understanding, the goal is to create a comprehensive knowledge graph for your entire domain.

Pre-requisites for building a knowledge graph

1. Defining site entities

   
   

   Audit your content to identify all major entities and assign canonical URLs that would later get linked with sub-topics. Some of the major entities could be:

   
   

   • Your brand/organization

   • Products and services

   • People (authors, executives, experts)

   • Locations (offices, service areas)

   • Concepts and topics (key subject matter areas)

     
     

2. Creating canonical URLs

   
   

   Having canonical URLs is critical to help AI understand the graph semantically and for you to understand how your content connects with each entity.

   
   

   Below is a example for setting up URLs for identified entities:

   
   

   • https://yoursite.com/about/company (Organization)

   • https://yoursite.com/products/seo-tool (Product)

   • https://yoursite.com/team/jane-doe (Person)

   • https://yoursite.com/topics/technical-seo (Concept/Topic)

     
     

Internal knowledge graph

• Implement @id references and use the same @id for each entity

  The @id property is where schema transcends basic markup and becomes a knowledge graph. Every entity should have a unique identifier that AI systems can reference across your entire site and across the web.

// On company about page
{
  "@type": "Organization",
  "@id": "<https://yoursite.com/#organization>",
  "name": "TechSEO Solutions"
}

// On product page
{
  "@type": "Product",
  "manufacturer": {
    "@id": "<https://yoursite.com/#organization>"
  }
}

// On article page
{
  "@type": "Article",
  "publisher": {
    "@id": "<https://yoursite.com/#organization>"
  }
}

Multi-entity mapping

• AI answer engines excel at understanding complex relationships.

• The @graph property allows you to include multiple, related schema objects in a single block.

• In the below example → we use @graph to connect different types of schemas together.

{
  "@context": "<https://schema.org>",
  "@graph": [
    {
      "@type": "Article",
      "@id": "<https://yoursite.com/guide#article>",
      "headline": "Technical SEO for AI",
      "about": {
        "@type": "Thing",
        "name": "Technical SEO",
        "sameAs": "<https://en.wikipedia.org/wiki/Search_engine_optimization>"
      },
      "mentions": [
        {
          "@type": "SoftwareApplication",
          "@id": "<https://chat.openai.com>",
          "name": "ChatGPT",
          "applicationCategory": "AI Assistant"
        },
        {
          "@type": "SoftwareApplication",
          "@id": "<https://perplexity.ai>",
          "name": "Perplexity AI"
        }
      ]
    }
  ]
}

Semantic triples

Semantic triples represent the atomic unit of knowledge in the semantic web: Subject → Predicate → Object. This structure is how AI systems internally represent all information. (source). Using semantic triples helps us establish a more effective relationship between different entities.

To explain in simpler terms for what each of these parts mean:

• Subject: The entity being described (e.g., "OpenAI")

• Predicate: The relationship or property (e.g., "develops")

• Object: The value or related entity (e.g., "ChatGPT")

See a product entity with relationships example. To understand this in practice, see how each property-value pair here creates a semantic triple:

• <AI SEO Optimizer Pro> <manufacturer> <TechSEO Solutions>

• <AI SEO Optimizer Pro> <price> <299.00>

• <AI SEO Optimizer Pro> <isRelatedTo> <Content Optimizer>

{
  "@context": "<https://schema.org>",
  "@type": "Product",
  "@id": "<https://yoursite.com/products/ai-seo-tool>",
  "name": "AI SEO Optimizer Pro",
  "manufacturer": {
    "@type": "Organization",
    "@id": "<https://yoursite.com/#organization>",
    "name": "TechSEO Solutions"
  },
  "offers": {
    "@type": "Offer",
    "price": "299.00",
    "priceCurrency": "USD",
    "seller": {
      "@id": "<https://yoursite.com/#organization>"
    }
  },
  "isRelatedTo": [
    {
      "@type": "Product",
      "@id": "<https://yoursite.com/products/content-optimizer>",
      "name": "Content Optimizer"
    }
  ]
}

Internal Linking

Traditional SEO focused on keyword based anchoring which doesn’t work for AI visibility. For AEO, we need entity-rich anchors that define relationships between different content pieces.

**Keyword-based approach**

<a href="/chatgpt-guide">click here for more</a>
<a href="/ai-tools">AI tools</a>

**Entity-based approach**

<a href="/chatgpt-guide" title="Comprehensive ChatGPT optimization guide">
  ChatGPT optimization strategies
</a>

<a href="/ai-tools" rel="related">
  Compare ChatGPT with other AI language models
</a>

Topic cluster

Think of topic cluster as a tree → you have the main trunk and the branches. In order for AI to build robust understanding of your website semantically, you need to think about your content in terms of a forest with multiple trees.

We have a pillar page (the trunk), which is a comprehensive guide covering a broad topic (3000-5000+ words). We then have cluster pages (branches) which are in-depth articles on specific subtopics (1500-2500+ words). To connect all of these together we implement bidirectional links - pillar links to all clusters and clusters link back to pillar and to each other.

Pillar: /technical-seo-complete-guide
  ├── /technical-seo/schema-markup
  ├── /technical-seo/site-speed
  ├── /technical-seo/crawlability
  ├── /technical-seo/renderability
  └── /technical-seo/internal-linking

Each cluster links:
- Back to pillar
- To 2-3 related clusters

URLs and Links

Having the right URL structure plays a huge role in helping AI make sense of your website content for semantic understanding.

Best Practices:

• Use descriptive, hierarchical URLs

• Keep URL depth to 3-4 levels maximum

• Include primary keyword in URL

• Use hyphens (not underscores) to separate words