What SEO Bots See When They Crawl Your Page

An SEO bot sees your page in three separate moments, and the version it ends up indexing is rarely the file you last saved. When Googlebot crawls a URL, it first reads your robots.txt, then requests the HTML document, then, later, runs the JavaScript and builds a rendered page. Those three views can disagree with each other and with what you see in a browser. This guide walks through each view, shows you how to inspect all three yourself with view-source and Google Search Console’s URL Inspection tool, explains why JavaScript-heavy pages are the risky case, and ends with a checklist for pages a bot can read correctly. Our earlier post, SEO Bots Explained, covers what SEO bots are. This is the operational follow-up: what they see, and how to look for yourself.

What Googlebot actually requests when it crawls a URL

Crawling a page is not one request. It is a small sequence, and each stage costs a round trip:

Google’s own description of how it processes JavaScript names three phases, crawling, rendering and indexing, and it starts with the permission check: “When Googlebot fetches a URL from the crawling queue by making an HTTP request, it first checks if you allow crawling. Googlebot reads the robots.txt file.” The rules it parses follow the Robots Exclusion Protocol, and Google documents exactly how it interprets the file, including how it treats 3xx, 4xx and 5xx responses.

The request itself is worth understanding too. Google publishes the technical properties of its crawlers: Googlebot crawls over HTTP/1.1 or HTTP/2, offers gzip, deflate and Brotli compression, and honors ETag and Last-Modified caching. That last point matters operationally: if your server or CDN sends the right caching headers, a recrawl can be answered with a cheap 304 Not Modified instead of a full document. Requests come from a fleet of machines spread across many data centers, mostly with US IP addresses, so your logs will show several IPs rather than one. And size is capped: by default crawlers read only the first 15 MB of a file, with Google noting that Googlebot can apply smaller per-format limits for some file types and larger ones for others.

Because the raw HTML parse happens first, everything the bot learns about your site’s structure it learns from the href attributes in that first response. Links that appear only after JavaScript runs are discovered later, in the rendering stage, which is exactly the gap the next section is about.

Raw HTML versus the rendered DOM

The raw HTML is the document your server returns. The rendered DOM is the tree a browser builds after running that document’s JavaScript and applying its CSS. For a plain server-side page the two are nearly identical. For a JavaScript app, they can be almost unrelated.

Consider a page built with the app-shell model: the HTML response contains a loader and a script tag, and the actual title, headings and product text appear only after that script runs. That is fine in a browser, and invisible to anything that stops at the first stage. Google is explicit about what happens next: “Googlebot queues all pages with a 200 HTTP status code for rendering… The page may stay on this queue for a few seconds, but it can take longer than that. Once Google’s resources allow, a headless Chromium renders the page and executes the JavaScript.” Two consequences follow. The rendered version is what gets indexed, because Google “also uses the rendered HTML to index the page.” And waiting is normal, so anything injected by JavaScript is indexed later than server-rendered content, if nothing breaks in the meantime.

Compare that image with what the same page looks like rendered, and you are looking at the whole difference between crawling and rendering. Googlebot is also the generous case. “Not all bots can run JavaScript,” Google notes on the same page: many smaller crawlers and content scrapers never render at all, so a page whose text exists only in JavaScript is empty to them. Whatever you want a bot to read reliably should already be in the raw HTML.

Why a JavaScript-only page is a risk

If your content, headings and links only exist after JavaScript runs, every ranking signal they carry depends on the render queue clearing without error. That creates four practical risks, all documented in Google’s JavaScript SEO basics:

1. Delay. Rendered content is indexed when the render queue gets to your page, not when the crawl happens. Server-rendered pages skip this step entirely.
2. App-shell blindness. A client-side app whose initial HTML holds only a loader offers Googlebot nothing to parse at stage one. If rendering fails, is slow, or the script needs a browser API Google’s Chromium does not support, the page can end up indexed as an empty shell.
3. Canonical and status-code problems. Google recommends setting the canonical in the original HTML rather than by JavaScript, and it warns that single-page apps often cannot return a meaningful 404, producing a “soft 404” for missing content.
4. Other bots never render at all. Not every crawler has Googlebot’s rendering stage. As Google’s guide puts it, “not all bots can run JavaScript”: for a bot that reads only raw HTML, a JavaScript-only page is an empty shell and there is nothing to record.

Google’s own recommendation follows directly: “server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers.” If you must ship a client-rendered app, pre-render the critical content, or rewrite it at the edge as described below. Google Search Central’s short official explainer, How Google Search indexes JavaScript sites, covers this crawl, render and index pipeline:

How to see what Google saw

Three ways, from simplest to most complete.

View source. Right-click and choose View page source (or Ctrl-U). This shows the raw HTML your server returns right now, before any JavaScript runs, which is the stage-one crawler view. If your title, description and body text are not in this document, they exist only for renderers.

The cache: trick is gone. Google retired its cached-page service on February 2, 2024, pointed users to the Wayback Machine, and since September 2024 search results have been able to link to archived copies directly. Bing removed its cache links in December 2024 as well. A cache:yoursite.com query now returns nothing. So there is no shortcut to “the copy Google stored”; the reliable way to see the indexed version is Search Console.

Google Search Console, URL Inspection tool. Paste a URL from your own property and you get the full crawler’s eye view, documented in Google’s help center:

• Index status. “URL is on Google” means the URL is eligible to appear in results, not guaranteed to appear there.
• Indexed data, not live data. The results reflect the most recently indexed version of the page, which may be older than what is live now. To test the current version, use Live Test.
• View crawled page. Shows the HTTP request and response, and the returned HTML Google actually received. If this button is disabled, there was a problem fetching the page, which is itself a diagnosis.
• View rendered page. A screenshot of how Google’s inspection tool sees the page, with the list of loaded resources and the JavaScript output under “More info”. This is your window into the render queue’s result.
• Test live URL. Re-runs the same inspection, including rendering, against the current live page. Use it after a fix to ask “does Google still see the problem.”
• Request indexing. Asks Google to crawl the URL again, subject to a daily limit per property.

The one-minute habit: view-source the page, then open URL Inspection, then compare the two. If the raw HTML you see and the rendered page Google screenshots disagree, your page depends on rendering, and it is time to fix that.

How edge SEO serves one page two ways

Edge SEO runs between the bot and your origin. A reverse proxy, often a Cloudflare Worker, inspects the request and can answer depending on who asked. The light version injects only the parts crawlers read and visitors never see: the title, the meta description, robots directives, the canonical and JSON-LD structured data, all rewritten per URL at the edge without touching your CMS. The heavy version pre-renders a JavaScript app on the fly for Googlebot while visitors get the interactive experience.

The line between that and cloaking is intent, and Google defines cloaking precisely in its spam policies: “presenting different content to users and search engines with the intent to manipulate search rankings and mislead users.” The same page explicitly tells sites that rely on JavaScript to make the content accessible “without cloaking”. So the safe rule is: same content, better packaging. Rewrite the head, pre-render the same body, but never give a bot text a visitor would not find.

That is exactly the model SEOEdgeAI runs for Cloudflare sites: a Worker goes up in front of your domain, a proxy fetches your real page, and it injects the title, meta description, response headers and JSON-LD the agent has decided on, for each URL, without ever touching your origin code. The proxy fails open, so if it ever errors the request goes straight to your origin and visitors are unaffected. For a site learning this article’s lesson, edge SEO is the fastest way to make the raw HTML say what the rendered page shows.

The bot-readable page checklist

Run a page through this list before you publish, and again through URL Inspection after:

• [ ] Title tag and meta description are present in the raw HTML, not injected by JavaScript. Run them through our free meta title and description checker to see how Google renders them.
• [ ] Main content is in the initial HTML, or server-side or pre-rendered; JavaScript only enhances it.
• [ ] Canonical tag is in the HTML and always points to the same URL.
• [ ] Meaningful HTTP status codes: 404 for missing pages, 301 or 308 for moved ones.
• [ ] robots.txt is reachable, allows the paths you want indexed, and references your sitemap.
• [ ] Structured data (JSON-LD) sits in the raw HTML head.
• [ ] Compression and ETag or Last-Modified headers are set, so recrawls are cheap.
• [ ] The page stays comfortably under crawler fetch size limits.
• [ ] Googlebot and visitors get the same content; only the packaging differs.
• [ ] URL Inspection shows “URL is on Google”, and View crawled page plus Test live URL agree with what view-source shows.

Each of those is something you can verify in five minutes with the tools above. The payoff is that Googlebot and the AI crawlers stop guessing on your page’s behalf, and the page they index is the page you wrote.