How to check your robots.txt and XML sitemap

Your robots.txt file and your XML sitemap are both read by SEO by Zaweba when it crawls your site. It reads robots.txt using Google's matching rules and flags important paths that are blocked, including CSS and JavaScript files. It reads your sitemaps, including sitemap indexes and gzip files, and flags pages listed in them that carry a noindex, along with pages your site links to that the sitemap leaves out.

All of it comes from the site crawl in step 4 of the guided audit.

How it reads robots.txt

Robots.txt rules are easy to get subtly wrong, because the way they match is not obvious from reading the file. SEO by Zaweba follows Google's approach: the most specific group for the user agent applies, the longest matching rule wins, and wildcards are understood. That means the answer it gives about a path is the answer Google would give, not a rough guess.

The app also obeys robots.txt itself while crawling. If the file returns a server error, it is treated as closed, and a Crawl-delay of up to 10 seconds is honoured. So if your crawl stops sooner than expected, check whether robots.txt is telling it to.

What it flags

Blocked important paths are the serious finding. A rule meant to hide an admin folder can accidentally cover a whole section of the site, or block the CSS and JavaScript a page needs to render properly. Either can hurt how pages are understood.

On the sitemap side, the checks look for noindexed pages in the sitemap, which send mixed signals, and for pages linked from your site but missing from the sitemap. Orphan pages, which are in the sitemap but have no internal links, come from the same comparison. Canonical conflicts and redirect chains are also flagged, and a sitemap should list neither redirected addresses nor pages that point their canonical elsewhere.

Running the check

Set up a project with your root domain in step 1, give step 2 a page and a term, and in step 4 choose to crawl your own site. The crawler finds robots.txt and your sitemaps, then crawls breadth first to the page limit, 500 pages by default. You can change the limit in Settings under Limits, and the pause between requests under Crawling manners.

After the scan, the Site pages tab lists the site problems. The All checks tab holds every individual check with pass, warn or fail, a severity, an effort estimate and a plain reason, and you can sort it to bring the robots and sitemap findings together.

Fixing the common problems

For a blocked path, narrow the rule so it covers only what you meant to hide, and leave CSS and JavaScript open. For a noindexed page in the sitemap, decide which signal is right: if the page should be indexed, remove the noindex; if not, take it out of the sitemap. For pages missing from the sitemap, add the ones you want found. Many sites generate the sitemap automatically, in which case the fix belongs in the settings of whatever builds it.

Robots.txt and the sitemap are short files with a big effect, and they are worth checking whenever a site is redesigned or moved. Run the audit again after any change; SEO by Zaweba keeps every scan on the History page, so you can compare before and after.