Analysing server logs for SEO with SEO by Zaweba

Server logs show what search engine crawlers actually do on your site, rather than what you hope they do. On the Logs and Search Console page of SEO by Zaweba, drop in an Apache or Nginx combined access log (plain, .log, .txt or gzip) or paste some lines, then press Analyse. You get crawl per page, daily hits by status as a chart, pages never crawled, junk crawling and how quickly new pages are found.

It's for your own sites, since you need access to the server's logs. For a small business site it's often the first time anyone has looked at what Googlebot is spending its visits on.

Getting a log

Most hosting control panels let you download raw access logs, often as gzip files. SEO by Zaweba reads the combined format used by Apache and Nginx, and accepts plain files, .log, .txt or gzip. In Step 5 of the guided audit, access logs appear in the extra tasks list for your own sites, with where each export lives.

A few days of logs give a snapshot. A few weeks show patterns, such as which sections get crawled daily and which hardly ever.

Real crawlers and fake ones

Anyone can call their crawler Googlebot. To separate the real ones, SEO by Zaweba verifies Googlebot and Bingbot by reverse DNS followed by forward confirmation, once per crawler IP address and up to 300 addresses. Visits that claim to be a search engine but fail the check are separated out as fake bots, so they don't distort the picture.

That matters because scrapers often disguise themselves as Googlebot. Counting them as real would make it look as though Google was crawling far more than it is.

What the analysis shows

Crawl per page lists how often each address was requested by verified crawlers. Daily hits by status are drawn as a chart, so a sudden run of errors stands out. Never-crawled pages are ones that exist but haven't been visited in the log period. Junk crawling picks out visits wasted on addresses with parameters, 404 pages and redirects. Status codes are summarised, and you can see how new pages were crawled after you published them.

Acting on it

Junk crawling is often the quickest win. If crawlers keep visiting addresses with tracking parameters or old pages that 404, they spend visits there instead of on pages you care about. Fixing internal links that point at redirects or broken pages, which the site crawl also flags, cuts that waste.

Never-crawled pages are worth checking against the site problems list. A page that's never crawled is often an orphan page with no internal links, or missing from the sitemap. Adding links from relevant pages, which the app suggests, gives crawlers a way in.

If new pages take a long time to be crawled, look at how they're linked from the rest of the site. Pages buried deep in the click depth tend to be found later.

Privacy and good habits

Logs stay on your PC. SEO by Zaweba stores its data locally, and the only network use for log analysis is the DNS lookups that verify crawlers. Keep the logs you analyse with the project, and run the analysis again after making changes; comparing the two shows whether crawlers have shifted their attention to the pages that matter.