Do AI search engines respect robots.txt?

The major ones do. OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, Google-Extended and PerplexityBot all honour robots.txt directives. Blocking them removes you from AI search visibility — most brands should explicitly allow them, and only block specific paths (admin, staging) where appropriate.

Does Google render JavaScript for SEO?

Yes, but with a delay and a budget. Googlebot renders most JavaScript via the Web Rendering Service, but indexation can lag days or weeks behind crawl. AI crawlers (GPTBot, ClaudeBot) typically do NOT execute JavaScript — they only see the raw HTML response. SPAs without SSR/SSG/prerender are effectively invisible to most AI engines.

What are the Core Web Vitals targets?

Largest Contentful Paint (LCP) under 2.5 seconds, Interaction to Next Paint (INP) under 200 milliseconds, and Cumulative Layout Shift (CLS) under 0.1 — all measured on mobile at the 75th percentile of real users.

Why does schema matter for AI search?

JSON-LD schema turns prose into machine-parseable facts and Q&A pairs. AI search engines and LLMs preferentially cite sources that are unambiguously structured because the answers can be extracted with high confidence. Schema also reinforces entity relationships, which directly improves brand recognition across every major LLM.

Technical SEO

Technical SEO — the crawlable, render-safe foundation AI search needs.

No amount of GEO, AEO or content investment compensates for a site Google can't render or AI crawlers can't reach. Technical SEO is the load-bearing layer underneath every modern search strategy.

Book a free 20-min call See client outcomes

Definition

What is Technical SEO?

Technical SEO is the engineering discipline of making a website fully crawlable, renderable, indexable and citation-ready for both classic search engines (Google, Bing) and AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot). It covers Core Web Vitals, JavaScript rendering, JSON-LD schema, internal linking architecture, log-file analysis, indexation control, hreflang for international sites, and crawler access management in robots.txt.

What's included

Outcomes you walk away with.

Full technical audit: crawlability, renderability, indexation, schema, Core Web Vitals.

AI-crawler access policy: GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot.

Render-safe architecture for SPAs (SSR/SSG/prerender) so AI crawlers see real HTML.

Sitewide JSON-LD: Organization, WebSite, BreadcrumbList, Article, FAQPage, Service.

Core Web Vitals work: LCP < 2.5s, INP < 200ms, CLS < 0.1 on mobile.

Log-file analysis to confirm Googlebot and AI crawlers are reaching priority URLs.

Process

How the engagement runs.

Audit & prioritize

Crawl with Screaming Frog, render with Puppeteer, pull GSC + Bing Webmaster + log files. Score every issue by indexation, ranking and AI-citation impact, not just severity.

Render & schema

Confirm Googlebot and GPTBot see fully rendered HTML. Fix client-side rendering gaps with SSR/SSG/prerender. Deploy sitewide JSON-LD with stable @id graph references.

Performance & crawl budget

Hit Core Web Vitals targets, prune low-value URLs, fix internal linking, and tune robots.txt + sitemap.xml so crawl budget concentrates on revenue pages.

AI crawler readiness

Verify and document access for GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended and PerplexityBot. Monitor log files monthly to confirm AI crawlers are actually retrieving priority pages.

FAQ

Technical SEO — frequently asked.

What is technical SEO?: Technical SEO is the engineering discipline of making a website fully crawlable, renderable, indexable and citation-ready for search engines and AI crawlers. It covers Core Web Vitals, JavaScript rendering, JSON-LD schema, internal linking, indexation control, hreflang and crawler access — the load-bearing layer underneath content and link strategy.
Do AI search engines respect robots.txt?: The major ones do. OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, Google-Extended and PerplexityBot all honour robots.txt directives. Blocking them removes you from AI search visibility — most brands should explicitly allow them, and only block specific paths (admin, staging) where appropriate.
Does Google render JavaScript for SEO?: Yes, but with a delay and a budget. Googlebot renders most JavaScript via the Web Rendering Service, but indexation can lag days or weeks behind crawl. AI crawlers (GPTBot, ClaudeBot) typically do NOT execute JavaScript — they only see the raw HTML response. SPAs without SSR/SSG/prerender are effectively invisible to most AI engines.
What are the Core Web Vitals targets?: Largest Contentful Paint (LCP) under 2.5 seconds, Interaction to Next Paint (INP) under 200 milliseconds, and Cumulative Layout Shift (CLS) under 0.1 — all measured on mobile at the 75th percentile of real users.
Why does schema matter for AI search?: JSON-LD schema turns prose into machine-parseable facts and Q&A pairs. AI search engines and LLMs preferentially cite sources that are unambiguously structured because the answers can be extracted with high confidence. Schema also reinforces entity relationships, which directly improves brand recognition across every major LLM.