Ecommerce Technical SEO: 7 Crawlers That Index 2026

Ecommerce technical SEO is the work of making category, filter, and product URLs crawlable, indexable, and internally linked so Google can discover the catalog — including paginated sets — without treating every sort order as a new site. It is not a content score, and it is not a paid-ads feed.

Google Search Central documents how pagination and incremental page loading on ecommerce sites affect Google Search discovery of category and filtered URLs, according to Google Search Central (docs dated 2025-12-10). Googlebot follows href links; it does not click "Load more." The seven tools below are crawlers and monitors. US Tech Automations is not a crawler.

Crawl decision, not a theme setting

If Google cannot fetch /c/shoes?page=2 as its own URL, the second page of shoes does not exist in Search. Faceted filters that only exist after a JavaScript click have the same problem. That is the category decision: pick a crawler that can render JS if your theme requires it, then fix the templates the crawl names.

TL;DR: Screaming Frog paid at £199/year is the default desktop audit. Sitebulb Pro is the visual-hint alternative (500,000 URL default). Botify, Oncrawl, Lumar, and JetOctopus are log+JS cloud platforms (contact vendor). ContentKing is the always-on monitor. Do not buy a cloud crawler to replace a 400-URL boutique you can finish in the free Frog.

Seven technical SEO platforms

1. Screaming Frog SEO Spider — desktop crawl

Best fit: SEO leads who want 300+ issue types, JS rendering, and GSC/GA/PSI APIs on a laptop.

Limitations: free edition caps 500 URLs. Paid licence is £199 per year. Max crawl size depends on RAM.

Primary evidence: screamingfrog.co.uk/seo-spider.

2. Sitebulb — audits with hints

Best fit: consultants who want prioritized "hints," crawl maps, and PDF client reports.

Limitations: Lite prints a 10,000 URL cap. Pro prints a 500,000 URL default (raisable toward 2 million). Extra desktop users from £7 per month. Cloud from £95/month. Lite vs Pro monthly desk price did not extract as a dollar on the 2026-09-06 fetch — contact vendor for the desk SKU.

Primary evidence: sitebulb.com/subscriptions/pricing.

3. Botify — log-aware cloud crawl

Best fit: large catalogs that need crawl budget vs log-file reality.

Limitations: contact vendor. Not a £199 laptop tool.

Primary evidence: botify.com.

4. Oncrawl — crawler plus logs

Best fit: teams that want crawl + log + data-lake style SEO datasets.

Limitations: contact vendor.

Primary evidence: oncrawl.com.

5. Lumar (Deepcrawl) — enterprise crawl

Best fit: multi-site enterprises with scheduled cloud crawls and issue workflows.

Limitations: contact vendor.

Primary evidence: lumar.io.

6. JetOctopus — cloud crawl + logs

Best fit: mid-market catalogs that want cloud JS crawls without Botify's enterprise sales motion.

Limitations: contact vendor.

Primary evidence: jetoctopus.com.

7. ContentKing — real-time change monitor

Best fit: teams that need alerts when robots, canonicals, or titles change after a deploy — not a one-shot audit.

Limitations: contact vendor. It is not a replacement for a full-site inventory crawl.

Primary evidence: contentkingapp.com.

Evaluation weights

Licence dollars we can print

Screaming Frog paid licence: £199/year according to Screaming Frog (page dated in schema 2026-06-05).

Sitebulb Cloud: from £95/month according to Sitebulb (2026-09-06 fetch).

Capability matrix

Pagination recipe

Google's pagination guide says to link pages sequentially with , give each page a unique URL such as ?page=n, avoid fragment identifiers for page numbers, and avoid indexing filter/sort variants with noindex or robots.txt, according to Google Search Central (2025-12-10). rel="next"/rel="prev" is no longer used by Google.

Who this is for: technical SEO and engineering pairs on Shopify, Magento, WooCommerce, or custom catalogs who ship category templates.

Red flags: Skip Botify if Screaming Frog paid already finishes the catalog overnight. Skip a second crawler if ContentKing already catches the only failure mode (accidental noindex). Skip orchestration if Jira already blocks deploys on 5xx spikes.

Key Takeaways

• Desktop Frog at £199/year is the default ecommerce technical SEO audit; cloud crawlers are for logs and scale.

• BEST_OF pages earned 15.2% according to US Tech Automations (12,514 pages, 2026-08-24).

• 7 Best titles earned 25.5% according to US Tech Automations (vs 14.0% for 5 Best, same count).

• Sitebulb extra desktop user: £7/mo according to Sitebulb (2026-09-06).

• Screaming Frog free cap: 500 URLs according to Screaming Frog (2026) — a boutique, not a 12,000-SKU catalog.

• Googlebot does not click "Load more"; paginate with crawlable href URLs.

Worked crawl: 12,000 SKUs, £199 vs 500-URL free

A 12,000-SKU store plus 400 category URLs is about 12,400 HTML targets before images. Screaming Frog free stops at 500 URLs, so the operator buys the £199/year licence, crawls in JS mode, and fails every row where status_code is 404 or 301-into-a-loop. Sitebulb Pro's printed default is 500,000 URLs, which clears 12,400 with room; Cloud starts at £95/month if the laptop cannot hold the crawl. US Tech Automations would open a ticket per template family (PDP, PLP, facet) rather than per SKU, and would not recrawl overnight without a cost check on cloud minutes — 12,400 URLs, 500-cap miss, £199 seat.

Botify's public home positions log-aware site crawls, according to Botify (vendor, 2026). Get the contract dollar from them.

When a crawler is enough

Zapier, Make, or n8n can watch a folder of Screaming Frog exports, retry a failed BigQuery load, and page Slack when 5xx share exceeds a threshold you set. They can keep run history and error branches. You design idempotency (same crawl filename must not double-create tickets), who may mark a 404 as "won't fix," retention of full HTML, and a hold on the next deploy.

When NOT to use US Tech Automations: if Screaming Frog plus a spreadsheet already gates the release, stop. If ContentKing already alerts on robots.txt edits, do not add a second pager. If the catalog is 80 URLs, use Frog free.

A proposed US Tech Automations design would not replace the Spider. It would refuse a "deploy storefront" ticket while status_code 5xx share is above the threshold a human set, and would require a reviewer before closing the ticket.

Cost context: ecommerce stores SEO cost. ROI: SEO worth it for ecommerce stores. Graders vs crawlers: Semrush vs Surfer SEO for ecommerce stores.

Week-one crawl plan for a real catalog

Day 1: robots.txt, XML sitemap, and a 50-URL sample in Screaming Frog free. If the sample is already full of 404s, do not buy Botify to admire them in a prettier dashboard. Fix the template. The free Spider's 500-URL cap is enough to prove whether PDPs return 200 and whether category page 2 exists as ?page=2 with an href.

Day 2: if the catalog is larger than 500 HTML URLs, buy the £199/year licence. Turn on JavaScript rendering only after a raw-HTML crawl, because rendering is slower and you need the cheap inventory first. Export status_code, indexability, canonical, and whether the URL is in the sitemap. Sitebulb Pro's 500,000 URL default is the alternative if your team wants hint-style PDFs; Cloud from £95/month is the alternative if the laptop cannot hold the crawl. Extra Sitebulb users from £7/month are cheaper than a second Botify seat you have not quoted yet.

Day 3: pagination. Google's 2025-12-10 guide wants sequential links, unique URLs, no #page= fragments, and noindex or robots on sort/filter duplicates. If your theme uses "Load more," add crawlable page links or you are hiding inventory from Googlebot on purpose. Screaming Frog will list the "Load more" button as a button, not as a URL. That is the finding.

Day 4: facets. Most ecommerce technical SEO damage is ?color=red&sort=price exploding into millions of indexable URLs. Do not raise Sitebulb's crawl ceiling toward 2 million to "cover" that. Block the parameters. Lumar, Oncrawl, Botify, and JetOctopus earn their quotes when you need log files to see whether Googlebot already wastes budget on those facets. ContentKing earns its quote when a deploy silently adds noindex to /c/.

Day 5: tickets, not a new crawler. Group issues by template (PDP, PLP, blog, brand). One 404 on a deleted SKU is a redirect. One thousand 404s with the same path pattern is engineering. n8n can load the Frog CSV nightly, retry a failed sheet write, and page Slack when 5xx share exceeds your threshold. You set the threshold. You set who may mark "won't fix." You set how long HTML snapshots live.

Do not crawl staging with production credentials. Do not paste customer emails from GA into a crawl tool. Do not pay Cloud £95/month and Frog £199/year for the same 800-URL boutique. Re-fetch Frog and Sitebulb before renewal; this pack printed £199/year and £95/month cloud plus £7 users, not a dollar-converted fantasy.

If Jira already blocks deploys on 5xx, you own the loop. A second crawler will not attend the standup.

Frequently Asked Questions

What is the best ecommerce technical SEO crawler?

Screaming Frog paid at £199/year for most catalogs; Botify/Oncrawl/Lumar/JetOctopus when logs and cloud JS at scale are the job.

Does Google still use rel next and prev?

No. Google's pagination document says it no longer uses those link relations. Sequential href URLs still matter.

Can I infinite-scroll a category and rank?

Googlebot generally will not trigger the scroll handler. Expose paginated URLs with crawlable links.

Is Sitebulb cheaper than Screaming Frog?

Frog prints £199/year. Sitebulb Cloud prints from £95/month (about £1,140/year before extras). Desk Pro's monthly dollar did not extract here — compare quotes.

Does ContentKing replace a crawl?

No. It watches changes. You still need an inventory crawl to find the first 404.

When do we add a workflow layer?

When crawl fails have no owner and the next deploy still ships.

Decision checklist

1. Count HTML URLs (PDP + PLP + blog + paging).

2. If ≤500, try Frog free.

3. If >500, buy Frog £199/year or Sitebulb Pro.

4. If you need CDN logs vs Googlebot hits, quote Botify/Oncrawl/JetOctopus.

5. If deploys break robots weekly, add ContentKing.

6. Paginate with ?page=n, not #page=.

7. noindex sort orders.

Sitebulb also states most work computers handle sites up to 500,000 URLs on Pro, according to Sitebulb (2026-09-06). That is a hardware note, not a guarantee for a 2-million-URL facet explosion.

If crawl fails already have no owner, use pricing after the crawler exists. Corpus: ustechautomations.com.

Seven platforms, two of which printed a full licence (Frog £199, Sitebulb cloud/user). The other five remain "contact vendor" on purpose. Do not invent Botify dollars to pretty the table. Pretty is how a 12,000-SKU site stays on the 500-URL free Spider and never sees page 8 of the category.

What "unlimited" crawl actually means on a laptop

Screaming Frog's paid row says unlimited URLs with an asterisk: RAM and disk bound the crawl. A 12,000-SKU catalog with images, faceted URLs, and JS templates can still knock over a 16 GB laptop if you render every page. Sitebulb Pro's documentation on the pricing page says most work computers handle up to 500,000 URLs and that you can raise toward 2 million if you accept hardware risk. That is not a promise that your MacBook will finish tonight. Cloud from £95/month exists because of that sentence.

JS rendering is a second crawl, not a checkbox you leave on forever. First pass: raw HTML, status codes, canonicals, robots. Second pass: render the templates that hid links. Third pass: only the templates engineering changed. ContentKing's job is the third pass as a watch, not as an inventory. Scheduling Frog or Sitebulb nightly against a 12,000-SKU store without a change budget is how you pay electricity to rediscover the same 404.

Logs are the reason Botify, Oncrawl, Lumar, and JetOctopus get a sales call. A desktop crawl shows what your spider can fetch. A log file shows what Googlebot actually fetched. If Googlebot is burning crawl budget on ?sort=price, no amount of Frog exports will show it. Quote those four only after you have a week of logs and a facet policy. Until then, £199/year plus a robots.txt change is the cheaper experiment.

Internal links still have to be . Google's pagination document is blunt: crawlers do not click buttons. "Load more" without a crawlable URL is a choice to hide inventory. Infinite scroll without paginated URLs is the same choice. Merchant Center feeds and sitemaps help discovery; they do not repeal the href rule.

If your next deploy will change canonicals on every PDP, freeze crawls until the deploy is done, then run one paid Frog job, then ticket by template. Three crawlers on one deploy is how the same 301 is filed three times. One owner. One export. One ticket family. Re-fetch Frog (£199/year, 500 free) and Sitebulb (£7 users, £95 cloud, 10,000 Lite / 500,000 Pro) on the day you PO.