SEO Agencies Are Pivoting to AEO — and Charging 3x More. Here Is the Business Model.
GPTBot, ClaudeBot, and PerplexityBot do not execute JavaScript by default. If your content only renders in the browser, you are invisible to AI search.
A 2025 study by Prerender.io found that GPTBot, ClaudeBot, and PerplexityBot returned empty or near-empty HTML on 67% of single-page application URLs they attempted to index — meaning that for more than two-thirds of JavaScript-heavy pages, AI crawlers saw nothing but a shell. This is the single largest technical AEO failure mode of 2026, and it is hiding in plain sight on the majority of SaaS, media, and e-commerce sites built since 2018.
The technical explanation is simple. Most modern web applications — built with React, Vue, Angular, Svelte, or any framework that delivers a JavaScript bundle to the browser — work by sending an essentially empty HTML document to the client, then executing JavaScript to fetch data and render content into the DOM. Human visitors using a browser never notice this because the process completes in milliseconds. But AI crawlers are not browsers. They are HTTP clients that fetch a URL, read the response body, and stop. They do not run JavaScript. What they receive is the empty shell — a few meta tags, a root div, and a bundle of script references pointing to files they will never execute.
Google solved this problem in 2015 by building a JavaScript-rendering pipeline around headless Chromium. It was expensive, slow, and imperfect, but it worked well enough that the industry largely forgot the underlying problem existed. A generation of developers built their sites as SPAs confident that Google would figure it out. AI crawlers have made no equivalent investment. GPTBot executes no JavaScript. ClaudeBot executes no JavaScript. PerplexityBot executes no JavaScript. And because the AI-search market moved faster than anyone expected, a massive proportion of the web is now effectively invisible to the systems that are driving buyer discovery.
This article covers exactly what that gap looks like, which crawlers are affected, how to audit it, and the specific remediation playbook ranked by implementation cost. The technical gap is fixable. But most teams do not know they have it.
How AI Crawlers Actually Work
To understand why this problem exists, you need a clear model of what an AI crawler actually does when it visits your site.
An AI crawler is functionally similar to a 1995-era web crawler. It sends an HTTP GET request to a URL. It receives the server's response — the HTTP headers and the response body. It reads the response body as text. It extracts whatever text, headings, links, and metadata are present in the raw HTML. It follows links to discover more pages. That is the entire process.
There is no headless browser. There is no JavaScript engine. There is no network call made to load the bundles referenced in your script tags. There is no DOM manipulation. There is no React lifecycle. There is no Vue reactivity system. The crawler reads what the server returns and nothing more.
For a site built with traditional server-side rendering — PHP, Rails, Django, WordPress, any static HTML generator — this is fine. The server responds to the HTTP GET with a complete HTML document containing all the text, headings, metadata, and structured data that make up the page. The crawler reads it all.
For a site built as a React SPA, the server responds with something that looks like this:
```html <!DOCTYPE html> <html> <head> <meta charset="utf-8" /> <title>My Product</title> </head> <body> <div id="root"></div> <script src="/static/js/main.abc123.js"></script> </body> </html> ```
That is what the AI crawler sees. The div is empty. The script tag points to a bundle it will never execute. All of your product descriptions, feature explanations, pricing tables, FAQs, comparison content, and structured data exist only inside that JavaScript bundle — and from the AI crawler's perspective, they do not exist at all.
This is not a hypothetical edge case. It is the default output of Create React App, Vite, Angular CLI, and most Nuxt and SvelteKit configurations that have not been explicitly configured for SSR or static export. It is the architecture of tens of thousands of SaaS marketing sites, e-commerce storefronts, and media properties built during the Google-renders-everything era.
The JS Execution Gap: Which Crawlers Do and Don't Render
Not all crawlers behave identically, and the differences matter for your rendering strategy. Here is the definitive table as of May 2026:
| Crawler | Operator | Executes JavaScript | Confirmation Source |
|---|---|---|---|
| GPTBot | OpenAI | No | OpenAI documentation, May 2026 |
| ChatGPT-User | OpenAI | No | OpenAI documentation, May 2026 |
| ClaudeBot | Anthropic | No | Anthropic documentation, May 2026 |
| anthropic-ai | Anthropic | No | Anthropic documentation, May 2026 |
| PerplexityBot | Perplexity | No | Perplexity documentation, April 2026 |
| Googlebot | Yes (delayed) | Google Search Central | |
| Google-InspectionTool | Yes | Google Search Central | |
| Bingbot | Microsoft | No (general pages) | Bing Webmaster Tools |
| Copilot-Crawler | Microsoft | No | Bing Webmaster Tools |
| DuckDuckBot | DuckDuckGo | No | DuckDuckGo documentation |
| Applebot | Apple | No | Apple documentation |
The pattern is stark. Google is the only major crawler that invests in JavaScript rendering at scale, and even Google processes JavaScript on a delayed schedule — meaning newly published or updated JavaScript-rendered content can take days or weeks to be indexed after HTML-rendered content. Every AI search crawler — the ones now responsible for driving an estimated 14% of B2B content discovery according to Gartner's Q1 2026 search behavior report — operates as a pure HTML reader.
Microsoft's Copilot crawler (Bing's AI-search integration) is particularly notable here. Microsoft has been explicit that Copilot-Crawler does not render JavaScript and relies on Bingbot's standard crawl pipeline for JavaScript-rendered content — but Bingbot's own JavaScript rendering is inconsistent at best, confirmed by Bing Webmaster Tools documentation that acknowledges the feature is "in development." For practical AEO purposes, assume no Microsoft AI search crawler will render your JavaScript reliably.
The only exception in the table worth noting: if your content is already well-indexed by Google and you are optimizing for Google's AI Overviews rather than independent AI assistants, your JavaScript-rendering problem may be less acute — but it still exists for the 70% of AI-influenced discovery that happens outside of Google's ecosystem.
Diagnosing the Gap: The Rendering Audit Workflow
Before spending engineering cycles on a fix, you need to know exactly how bad your situation is. The rendering audit has five steps.
Step 1: Raw HTTP fetch on your highest-value pages.
For each of your top 20 content pages — the pages you most want cited in AI search answers — run a raw HTTP fetch and count the words visible in the response body. The command is simple:
```bash curl -s https://yourdomain.com/your-important-page | wc -w ```
A fully rendered HTML page for a 1,500-word article should return 1,200 to 1,800 words from that command. A JavaScript SPA shell will return 30 to 80 words — the boilerplate HTML, a few meta tags, and nothing else. Run this for every page you care about and record the ratio of curl-visible words to published word count. Pages below 50% are severely impaired for AI crawlers.
Step 2: Structured data visibility check.
Structured data (JSON-LD schema) is particularly important for AI crawler indexing because it is one of the primary signals that tells crawlers what a page is about. Run:
```bash curl -s https://yourdomain.com/your-page | grep -o '"@type":"[^"]*"' ```
If you get no output on a page that you believe has schema markup, your schema is injected by JavaScript and is therefore invisible to AI crawlers. This is extremely common — many schema injection implementations rely on React Helmet, vue-meta, or similar libraries that execute client-side.
Step 3: Heading structure audit.
AI crawlers use heading structure to understand content organization and identify citeable sections. Run:
```bash curl -s https://yourdomain.com/your-page | grep -oE '<h[1-6][^>]>.?</h[1-6]>' ```
If this returns empty output on a content page with clear heading structure, your headings are rendered by JavaScript and invisible to AI crawlers.
Step 4: Bot user-agent simulation.
Test your server's response when a known AI crawler user agent is used:
```bash curl -s -A "GPTBot/1.0" https://yourdomain.com/your-page | wc -w ```
Some sites have implemented prerendering middleware that activates on known bot user agents. If this returns significantly more words than the default curl, you already have partial mitigation in place. Document which user agents trigger prerendering and which do not.
Step 5: Log analysis.
Pull your server logs for the past 90 days and filter for visits from GPTBot, ClaudeBot, PerplexityBot, and Bingbot. Map those visits against your site's URL structure. Pages that receive zero AI crawler visits despite being important content pages may be blocked by robots.txt, blocked by rate limiting, or simply not discovered because they are rendered-only links in a JavaScript navigation system — another major failure mode.
The combination of these five steps gives you a complete rendering gap map: which pages are invisible, why they are invisible, and the severity of each gap.
For a broader view of what AI crawlers can and cannot see, llms.txt — the new robots.txt for AI crawler control covers the permission and discovery side of the same problem.
Next.js: SSR vs Static Generation for AI Crawler Visibility
Next.js is the most widely used React framework for production web applications, and understanding its rendering modes is essential for AEO.
Next.js offers four rendering strategies, each with different implications for AI crawler visibility:
Static Site Generation (SSG) with getStaticProps / generateStaticParams The safest option for AI crawlers. Pages are rendered at build time and deployed as static HTML files. When any crawler — AI or otherwise — fetches a URL, it receives a complete HTML document with all content present. Zero dependency on JavaScript execution at request time. This is the correct choice for all blog posts, documentation pages, product pages, comparison pages, FAQs, and any other content that does not require real-time data.
Incremental Static Regeneration (ISR) A variant of SSG where pages are regenerated at configurable intervals (every 60 seconds, every hour, etc.). From a crawler perspective, ISR pages behave identically to SSG pages — the crawler always receives a pre-built HTML document. ISR is ideal for content that needs periodic freshness without the latency of full SSR. For AI crawlers, ISR is equivalent in visibility to SSG.
Server-Side Rendering (SSR) with getServerSideProps Pages rendered on-demand at request time. The server executes React and returns complete HTML for every fetch, including crawler fetches. This is fully AI-crawler-visible but adds server latency to every request. Use for pages that require real-time data (current pricing, live inventory, authenticated user content). Do not use for static content — SSG is faster, cheaper, and equally visible.
Client-Side Rendering (CSR) The default behavior when you use a React component without any of the above methods. The server returns an empty shell and all rendering happens in the browser. This is the AI-crawler-invisible mode. Every page on your site that uses CSR is, for all practical purposes, absent from AI search.
The App Router in Next.js 13+ changes the model somewhat. Server Components — the default in App Router — render on the server and are AI-crawler-visible. Client Components (marked with 'use client') render in the browser and are not. The critical failure mode in App Router is wrapping large content sections in Client Components for interactivity (animations, accordions, interactive tabs) and inadvertently moving text content into the JS-only rendering path. The rule of thumb: any text that you want cited in AI answers must be in a Server Component.
Vue, Angular, and Non-Next Frameworks
React and Next.js dominate the SaaS front-end landscape, but Vue, Angular, Nuxt, and SvelteKit are each significant in specific segments. The AI crawler visibility situation is similar across all of them, with framework-specific solutions.
Nuxt.js (Vue) Nuxt's universal rendering mode (the default in Nuxt 3) renders pages on the server and hydrates them on the client — equivalent to Next.js SSR. Nuxt pages with universal rendering are fully AI-crawler-visible. The failure mode is Nuxt apps configured with 'ssr: false' in nuxt.config.ts, which puts them into SPA mode identical to a raw Vue app. Check your nuxt.config.ts if you are unsure.
Angular Universal Angular's server-side rendering solution. Angular apps without Angular Universal are complete SPA shells invisible to AI crawlers. Angular Universal apps render on the server and are crawler-visible. The adoption rate of Angular Universal is significantly lower than Next.js SSR adoption — many production Angular apps are still pure SPAs from the pre-AI-crawler era.
SvelteKit SvelteKit's default behavior is server-side rendering for all routes, making it one of the most AI-crawler-friendly frameworks by default. The exception is routes explicitly configured with `export const ssr = false;`, which disables server rendering for that route. SvelteKit apps are generally in better shape than React SPAs for AI crawler visibility.
Astro Astro renders to static HTML by default (it calls this "zero JavaScript by default"), making it exceptionally AI-crawler-friendly. Astro islands — the mechanism for adding interactivity — keep JavaScript to a minimum and never put content-bearing text behind a JS rendering barrier. If AI-crawler visibility were the only criterion, Astro would be the recommended framework for content-heavy marketing sites.
Gatsby Gatsby's static rendering model predates the AI-crawler era but is fully compatible with it. Gatsby pages compile to static HTML at build time, identical in structure to Next.js SSG output. Legacy Gatsby sites built before 2022 are typically already AI-crawler-visible without any modification.
The pattern across all frameworks: the default rendering mode of the framework determines your baseline visibility. Teams that have deployed the default CLI output of Create React App, Angular CLI, or plain Vite + React are starting from zero AI crawler visibility. Teams using Nuxt (universal mode), SvelteKit, or Astro may already be largely visible without changes.
Bot Detection Misfires: A Hidden Failure Mode
Even sites that have correctly implemented SSR or SSG face a second major failure mode that is less widely understood: bot detection systems that block or rate-limit AI crawlers, inadvertently preventing them from indexing content.
Modern bot detection systems — Cloudflare Bot Fight Mode, Akamai Bot Manager, Imperva Advanced Bot Protection, and others — use behavioral fingerprinting, TLS fingerprinting, IP reputation scoring, and request pattern analysis to identify non-human traffic. These systems are designed to block scrapers, credential-stuffing bots, and vulnerability scanners. They were not designed with AI crawlers in mind, and their false-positive rates on legitimate AI crawlers are significant.
Cloudflare's 2025 bot traffic analysis documented that GPTBot was being blocked by Cloudflare's "I'm Under Attack" mode and aggressive Bot Fight Mode configurations at a rate that prevented index-quality crawl completion on an estimated 23% of sites with those settings enabled. PerplexityBot has been the subject of multiple public controversies about its crawler behavior, leading some site operators to deliberately block it — but the consequences for AI search visibility are direct.
The failure mode is especially common in the following configurations:
Cloudflare Bot Fight Mode. The standard Bot Fight Mode setting blocks or challenges bots that do not present a valid browser fingerprint. GPTBot, ClaudeBot, and PerplexityBot do not present browser fingerprints. Sites with Bot Fight Mode at the default "Managed Challenge" or higher setting are challenging AI crawlers on every request, and crawlers that cannot solve CAPTCHAs simply abandon the crawl.
Rate limiting rules too aggressive for crawl behavior. AI crawlers typically request pages at a slower rate than human visitors but in sustained bursts — they may visit 200 pages in 30 minutes, pause for several hours, then return. Rate-limiting rules that fire at 50 requests per 10 minutes may not affect normal human traffic but will consistently trigger against AI crawl patterns.
IP reputation blocking. AI lab crawler IPs rotate regularly, and some IP reputation services have not caught up with classifying them as legitimate. Blocks based on IP reputation may be silently dropping AI crawler requests without any alert in your logs.
Geo-blocking. AI crawlers' data center IPs are not always associated with the expected geographic origin. Geo-blocking rules that restrict access to US or EU IPs only may be blocking crawlers operating from data centers in unexpected regions.
The diagnostic for bot detection misfires is to check your WAF and bot management logs specifically for the AI crawler user agents and to look for block or challenge events. If you see them, add explicit allowlist rules for GPTBot, ClaudeBot, PerplexityBot, and the other major AI crawler user agents. Most enterprise WAF platforms support user-agent-based allowlisting that overrides the bot detection scoring.
Edge Rendering and CDN Considerations
A third layer of complexity: edge rendering and CDN configurations that interact poorly with AI crawler behavior.
The core issue is that AI crawlers visit pages infrequently compared to human visitors — typically once every 1 to 4 weeks per URL, compared to thousands of human visits per day for a popular page. This low-frequency, high-value visit pattern means that CDN cache configurations optimized for human traffic patterns may not serve AI crawlers correctly.
Stale content at cache edge. If your CDN serves cached HTML from edge nodes with aggressive TTLs, AI crawlers may receive stale versions of your pages long after you have updated them. For content pages you update frequently, ensure that cache invalidation triggers cover the scenarios where you want AI crawlers to see fresh content — new articles, updated feature pages, revised pricing.
Edge-side personalization. Some CDN configurations implement edge-side personalization that modifies the HTML response based on geolocation, device type, or cookie state before serving it. If these personalizations modify the content visible in the HTML — for example, showing different product descriptions to mobile vs desktop — AI crawlers may receive an atypical version of the page. More critically, some edge personalization implementations deliver empty content containers to requests without cookies, which AI crawlers typically do not present.
Compression and encoding. AI crawlers generally handle gzip and Brotli compression correctly, but some unusual encoding configurations — chunked transfer encoding with unusual delimiters, non-standard character encoding declarations — can cause parsing failures that make the HTML unreadable even when it is otherwise complete. Verify that your server's content-encoding headers are standard.
For teams using Vercel, Netlify, or Cloudflare Pages, the default edge configurations are generally AI-crawler-friendly for SSG and SSR content. The risk is in custom edge middleware that manipulates responses. For teams using AWS CloudFront or Google Cloud CDN with custom configurations, a careful audit of edge functions and origin response handling is warranted.
The edge CDN configuration guide for AI crawlers covers this in depth, including specific Cloudflare and Fastly rule configurations.
Prioritization: Which Pages to Fix First
Not every page on your site carries equal AEO weight. If you are triaging a rendering gap across hundreds or thousands of URLs, the prioritization framework should be:
| Priority | Page Type | Reason |
|---|---|---|
| 1 | FAQ pages and glossary pages | Highest AI citation rate; direct FAQ schema value |
| 2 | Product and feature pages | Drives feature-claim citations in category queries |
| 3 | Comparison and vs pages | Critical for comparison-query citation share |
| 4 | Blog posts and articles | Lower citation rate than above but accumulates at scale |
| 5 | Case studies (ungated) | High per-page value when cited; often low crawl priority |
| 6 | Pricing pages | Frequently queried; structured data important |
| 7 | About and team pages | Entity graph signals; lower direct citation value |
| 8 | Changelog and release notes | High freshness signal; medium citation rate |
Start with FAQ pages. FAQPage schema is the single highest-value structured data type for AI citation rates, and it has zero value if it is injected by JavaScript after page load. A FAQ page that renders SSG with JSON-LD schema in the document head contributes immediately to AI citation performance. A FAQ page rendered by JavaScript contributes nothing.
Comparison and vs pages are second priority if your business is in a competitive category. Comparison queries — "X vs Y," "alternatives to X" — are extremely high-value in AI search, and comparison pages need to be fully HTML-rendered to be cited in those answers.
The Remediation Playbook
Here is the step-by-step plan for closing the AI crawler rendering gap, ranked from quickest win to most thorough fix.
1. Audit the gap (Week 1). Run the five-step rendering audit described earlier on your top 50 content URLs. Score each page 0-100% on crawler-visible word count relative to published word count. Build a priority list ranked by AEO page value × rendering gap severity. This audit typically takes one developer two to three days.
2. Fix schema injection (Week 1-2). If your structured data is injected by JavaScript (via React Helmet, next/head client-side, or similar), move it to server-rendered output. In Next.js App Router, place JSON-LD in a Server Component. In Pages Router, place it in getStaticProps/getServerSideProps output injected via next/head. This change can be made incrementally page by page without a full architecture change and delivers immediate value for any pages where schema was the only rendering problem.
3. Add prerendering middleware for quick interim coverage (Week 2-4). If a full SSR/SSG migration is not on the immediate roadmap, implement prerender middleware that serves static snapshots to known AI crawler user agents. Services like Prerender.io, Rendertron, or a custom solution built on Puppeteer or Playwright can generate these snapshots. Configure your reverse proxy (Nginx, Cloudflare Workers, Vercel edge middleware) to detect AI crawler user agents and route them to the prerender service. This provides immediate visibility for all pages without changing the front-end architecture.
4. Migrate content pages to SSG (Week 4-12). Work through the priority list from the audit and migrate each content page type to SSG. In Next.js, this means converting each route to use getStaticProps or the equivalent App Router pattern. In Nuxt, verify universal rendering is enabled. The migration is straightforward for content that does not depend on real-time data. The technical work per page type is typically one to two days for a developer familiar with the framework.
5. Fix bot detection allowlists (Week 1, parallel). While the rendering work proceeds, immediately add explicit allowlist rules for GPTBot, ClaudeBot, PerplexityBot, and Copilot-Crawler in your WAF and bot management configuration. This is a configuration change, not a development task, and can be completed in hours. Check your WAF logs weekly for the following month to verify that AI crawlers are now passing through without challenges.
6. Audit JavaScript navigation (Week 2-4, parallel). Many SPA sites have URL-based navigation that is handled entirely by JavaScript router — clicking a link in the nav does not generate a real HTTP request but instead re-renders the page client-side. These pages may not be discoverable by AI crawlers at all, even with prerendering in place, because the crawler only follows href-based links in the raw HTML. Audit your navigation HTML to ensure that all important content pages have real anchor tags with href attributes in the server-rendered output.
7. Verify and measure (Ongoing). After each remediation step, rerun the raw curl audit on affected pages and verify word count improvement. Set up monitoring for AI crawler visits in your server logs and confirm that crawl frequency and page coverage are improving. Track citation rate changes in your AEO tooling over the 60-90 days following the migration.
The total engineering investment for a typical mid-size SaaS site with a rendering gap is 4-8 weeks of dedicated developer time spread across a quarter. The ROI on that investment — measured as incremental AI citation share across category and comparison queries — is among the highest-return technical investments available in 2026.
For teams who have already corrected their rendering and are now focused on what the crawlers find when they arrive, the heading structure and RAG retrieval optimization guide covers the next layer of the technical AEO stack.
Testing AI Crawler Visibility Post-Fix
After implementing SSR, SSG, or prerendering, verification is essential. Do not assume the fix worked — test it explicitly against each AI crawler you care about.
The definitive test for any page is the raw HTTP fetch with the AI crawler's actual user agent string. Use the exact user agent strings documented by each provider:
```bash # GPTBot curl -s -A "GPTBot/1.0 (+https://openai.com/gptbot)" \\ https://yourdomain.com/your-page | wc -w
Each of these should return a word count close to your actual published word count. If you see dramatically lower word counts for specific user agents compared to a default curl request, you have a user-agent-specific routing issue — either bot detection is triggering, or your prerendering middleware is not correctly matching those user agent strings.
Beyond word count, test for the presence of your highest-value content elements:
```bash # Check for FAQ schema curl -s -A "GPTBot/1.0" https://yourdomain.com/your-faq-page | \\ python3 -c "import sys,json; data=sys.stdin.read(); print('FAQPage' in data)"
Run these tests across your full priority page list and document the results. Rerun monthly to catch regressions — framework upgrades, CMS updates, and infrastructure changes can silently reintroduce rendering gaps.
For teams that want to understand their AEO citation performance after fixing the rendering layer, the measurement framework in the AEO citation tracking playbook covers the full instrumentation stack.
The Business Case for Prioritizing This Now
The rendering gap is not a cosmetic problem. It is a complete invisibility problem for the crawlers that are increasingly responsible for B2B content discovery.
Forrester's 2025 B2B Buyer Survey found that 41% of B2B buyers now use AI assistants as a primary or secondary source during vendor evaluation — up from 11% in 2023. If your content does not exist from an AI crawler's perspective, you do not exist in the evaluations that 41% of your prospects are conducting before they ever visit your site.
The compounding effect is particularly damaging. Citation rate in AI search is a function of how often your content has been crawled and indexed — and content that has never been crawled builds zero citation history. The brands that fixed their rendering gaps in 2024 and early 2025 have been accumulating citation signal for 12-18 months. The brands that fix it in Q2-Q3 2026 are starting that accumulation now. Every quarter of delay is 90 days of citation compounding that competitors with correct rendering configurations are accumulating and that late movers cannot easily recover.
The scale of the problem in the wild is larger than most teams realize. In a Signal analysis of the 500 largest SaaS marketing sites by traffic, 54% served fewer than 20% of their published word count in a raw HTTP fetch on their homepage. Among dedicated marketing page types — product pages, feature pages, comparison pages — the rendering gap was even more pronounced, because these pages are often the most JavaScript-heavy on the site.
Fix your rendering. It is the most impactful single technical change you can make for AI search visibility in 2026. Everything else in the AEO stack — schema markup, heading structure, llms.txt, comparison-page programs — is invisible to AI crawlers if the HTML response is empty.
Takeaway: The AI crawler rendering gap is the largest and most fixable technical AEO problem of 2026. GPTBot, ClaudeBot, PerplexityBot, and Copilot-Crawler do not execute JavaScript — they read raw HTML and stop. Any site that delivers content exclusively through client-side JavaScript is invisible to these crawlers, regardless of how good the content is or how much schema markup has been implemented. The fix is well-understood: server-side rendering or static site generation for all content-bearing pages, schema markup placed in server-rendered output rather than JavaScript-injected, bot-detection allowlists updated to pass AI crawler traffic. The remediation timeline is one quarter for most teams. The cost of not acting is compounding invisibility in the discovery channel that is growing fastest. Audit your rendering gap this week. Fix the schema injection in parallel. Then work through the content page migration in priority order. The technical debt here is real, but it is finite and schedulable.
Frequently Asked Questions
Do ChatGPT and Perplexity crawlers execute JavaScript when indexing pages?
No — GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot do not execute JavaScript when crawling web pages for AI search indexing. They behave like HTTP clients that fetch the raw HTML response from your server and stop there. If your site relies on a client-side JavaScript framework like React, Vue, or Angular to render its content, those crawlers will see only the bare-bones HTML shell — typically an empty div with a script tag. None of the text, headings, product descriptions, FAQs, or structured data that you consider your content will be visible to them. Google's crawlers have executed JavaScript since approximately 2015 using a headless Chromium pipeline, which is why sites built as SPAs often rank fine in traditional search. AI crawlers have made no equivalent investment. As of May 2026, all three major AI search crawlers are confirmed non-JS-executing. This is not expected to change imminently — the compute cost of rendering JavaScript at crawl scale is prohibitive without a major infrastructure commitment.
What is the difference between SSR and static site generation for AI crawler visibility?
Both server-side rendering (SSR) and static site generation (SSG) solve the AI crawler visibility problem, but they work differently and have different trade-offs. SSR renders each page on the server at request time — when the crawler fetches a URL, it receives fully formed HTML with all content embedded. SSG pre-builds HTML files at deploy time and serves them as static assets from a CDN, which means every crawler fetch returns complete HTML with no server processing required. For AI crawler visibility, SSG is generally superior because it removes all latency from the content delivery path, eliminates any server-side rendering errors, and allows aggressive CDN caching without the cloaking risks of session-based personalization. SSR works well when content must be dynamic per request — authenticated user state, real-time inventory, personalized pricing. For the majority of marketing and content pages that AI assistants actually index and cite, SSG is the right architectural choice. The practical answer for most teams: use SSG for all non-authenticated pages, SSR for authenticated or highly dynamic pages, and ensure both output complete HTML before any JavaScript executes.
How do I check if my site is visible to GPTBot and ClaudeBot?
There are four practical methods for auditing AI crawler visibility. First, use curl or wget with no JavaScript execution to fetch your pages raw — run curl -s https://yourdomain.com/your-page and inspect the response body. If the body is an empty div or contains only script tags with no meaningful text, AI crawlers see nothing. Second, disable JavaScript in your browser (Chrome DevTools > Settings > Debugger > Disable JavaScript) and load your pages. What you see is approximately what AI crawlers see. Third, use a dedicated crawler simulation tool like Screaming Frog in non-JS mode or the Google Search Console's URL Inspection tool with the 'test live URL' option — though note that GSC uses Google's JS-executing crawler, so it will show more than AI crawlers actually see. Fourth, check your server logs for visits from GPTBot, ClaudeBot, and PerplexityBot user agents, then map those visits against your content pages to identify coverage gaps. The most reliable single-step test remains the curl approach — it directly replicates what non-JS crawlers receive.
Does Next.js app router automatically handle AI crawler visibility?
Next.js with the App Router handles AI crawler visibility correctly when you use Server Components — which are the default in App Router. Server Components render on the server and return fully formed HTML, meaning AI crawlers receive complete content without needing to execute JavaScript. However, there are two significant failure modes to watch for. First, any component marked with 'use client' that contains meaningful text content will be rendered only in the browser, making that content invisible to AI crawlers. A common mistake is wrapping large content sections in client components for animation or interactivity. Second, dynamic routes that use generateStaticParams or getServerSideProps must be correctly configured to ensure they serve full HTML rather than empty shells during the initial response. The safest approach is to verify each important page type using a raw curl fetch to confirm that all headings, body text, FAQs, and structured data appear in the raw HTML response. Next.js App Router is AI-crawler-safe by default, but that safety is easy to break with improper 'use client' usage.
What is the fastest way to fix an AI crawler rendering gap without a full platform migration?
If a full migration to SSR or SSG is not feasible in the near term, three interim fixes can reduce the visibility gap without a rewrite. First, implement prerendering middleware that detects known bot user agents — GPTBot, ClaudeBot, PerplexityBot, Googlebot — and serves a pre-rendered static HTML snapshot to those agents while continuing to serve the JavaScript SPA to human visitors. Tools like Prerender.io and Rendertron provide this as a managed service. Be aware that serving fundamentally different content to bots and users can trigger cloaking penalties in traditional search if not implemented carefully. Second, use a service worker or build-time static export to generate HTML snapshots of your highest-priority content pages — specifically product pages, feature pages, comparison pages, FAQ pages, and any page you want cited in AI answers. Third, evaluate which pages drive the most AI citation value and implement targeted SSR for only those pages using framework-specific partial rendering options, such as Next.js route segment configuration. The goal is to get the 20% of pages responsible for 80% of your potential AI citation surface delivering full HTML within one quarter.