How to tell real Googlebot from fake Googlebot in logs

Anyone can put Googlebot in a user agent string, and plenty of scrapers do, because some sites let Googlebot through where they would block anyone else. That makes the user agent useless on its own as proof. SEO by Zaweba checks the address instead: when you analyse an access log, it verifies each crawler IP by reverse DNS and then confirms the result with a forward lookup. Visits that pass are real Googlebot (or Bingbot), and the rest are separated out as fakes.

You do it on the Logs and Search Console page, with a log from your own site.

How the check works

Reverse DNS asks what name belongs to an IP address. A genuine Google crawler's address resolves to a name in Google's own domains, and a genuine Bing crawler's to Microsoft's. That alone could be faked by whoever controls the address's reverse record, so the second step looks the name up again in the normal direction and checks it leads back to the same IP. Only when both directions agree is the crawler counted as verified. This is the method the search engines themselves recommend.

SEO by Zaweba does this once per crawler IP rather than once per line, which keeps it quick on large logs, and it checks up to 300 crawler addresses in one analysis.

Running it

Download the access log from your host. Step 5 of the guided audit lists access logs among the extra tasks for your own sites and says where to find them. Drop the file onto the Logs and Search Console page (plain text, .log, .txt or gzip in the Apache or Nginx combined format all work) or paste some lines, and click Analyse.

The results then show crawl per address, daily hits by status, never crawled pages, junk crawling and status codes, all based on verified crawlers, with the impostors kept apart.

Why it matters for SEO

Without verification, crawl figures are inflated by bots that only pretend to be Google. You might think a page is crawled every day when the real Googlebot hardly visits it, or blame Google for a burst of 404s that was really a scraper guessing addresses. Separating them gives you the true picture of how Google spends its time on your site, which is the point of reading logs in the first place.

It also helps with the opposite worry. If a firewall or security plugin is blocking anything that claims to be a crawler, the verified list tells you whether real Googlebot is getting through.

What to do about fakes

Usually nothing dramatic. Fake crawlers are common and mostly harmless beyond the bandwidth they use. If one is hammering the server, block its addresses at the firewall or with your host, but be careful never to block on the user agent alone, or you may shut out the real crawler too. That is the main reason to verify before you act.

SEO by Zaweba never contacts Google search itself. The log and the results stay on your PC with the rest of the app's data.