Setting crawl limits in SEO by Zaweba
Settings, Limits in SEO by Zaweba controls how many pages and competitors are crawled and how many links and images are checked, for speed or depth.
Settings, then Limits, in SEO by Zaweba decides how much work an audit does. It sets the number of pages to crawl on your own site, the pages per competitor site, how many competitors are crawled (the top N), how many links are checked on each page, how many images are sized per page, the percentage of results kept in a modifier split, and a switch for competitor-strength lookups.
Bigger numbers mean a fuller picture and a longer wait. Smaller ones mean a quicker audit that may miss things. The right setting depends on the job in front of you.
Your site and competitor sites
Pages to crawl on your site sets how far the site crawler goes. By default the site crawl reaches up to 500 pages, crawling breadth-first so the pages nearest the home page come first. For a small business site, that covers everything. For a large shop, it covers the main structure, which is often enough to find orphan pages, duplicate titles and redirect chains.
Pages per competitor site sets how deeply the top competitors' sites are crawled to look at their internal linking; by default it's up to 40 pages each for the top three. How many competitors decides how many of the top organic results get the full set of page checks, 10 by default.
Links and images per page
Links checked per page sets how many links on each page are tested for being broken, using polite HEAD or GET requests. Images sized per page sets how many images are measured for file size and natural versus displayed size. On a page with hundreds of links or images, a lower limit keeps the audit moving. On a page where you suspect broken links, a higher one is worth the wait.
Modifier splits and competitor strength
The percentage of results kept in a modifier split is used when the app weighs modifier searches (near me, wholesale, cost and so on) as part of suggesting pages that should exist. The switch for competitor-strength lookups turns those lookups on or off, which is handy if you'd rather keep network use to a minimum.
Choosing sensible limits
For a first audit of your own site, leave the defaults alone. They're chosen to give a useful picture in a reasonable time. If the audit takes too long, reduce links checked per page and images sized per page before you cut the number of competitors, because the comparison with the pages above you is the heart of the audit.
If you're auditing a large site and want a fuller site crawl, raise the page limit and expect to wait longer. Long jobs show progress and can be cancelled, so it's safe to try.
Limits and politeness
Higher limits mean more requests to other people's sites. SEO by Zaweba crawls politely whatever the limits: it always obeys robots.txt, pauses between requests to one site (at least one second by default, under Settings, Crawling manners) and honours a robots Crawl-delay up to 10 seconds. So a bigger limit makes the audit slower rather than ruder.
Keeping limits steady
If you're tracking a project over time, try to keep the limits the same between scans. History compares every check result, and changing how much was crawled can make a scan look better or worse for reasons that have nothing to do with your site.
