Close inventory gaps
Find orphaned, forgotten, or leftover URLs that still respond and compete for crawl attention.
Website crawler
Crawl every reachable page on the domains you own.
Crawlox starts from your sitemap and homepage, follows internal links, and builds a live page inventory so SEO, content, and engineering teams can see what search engines can actually reach.
Authorized root domains
Add a domain you own, confirm rights, and keep crawls scoped to that root and the pages beneath it.
Sitemap and link following
Seed from sitemap.xml, then discover more URLs through HTML internal links with a breadth-first crawl.
Page inventory
Review status codes, titles, inbound and outbound links, and last-seen timestamps in one table.
Scheduled and on-demand crawls
Set a recurring cadence or launch a manual crawl whenever content, redirects, or releases change.
What is a website crawler?
A website crawler inventories the URLs search engines can follow.
A website crawler fetches public pages, follows internal links, and records what it finds: URLs, status codes, titles, headings, and link relationships. Crawlox does this for authorized root domains so teams do not have to run Screaming Frog locally, stitch CSV exports, or guess which pages still exist after a redesign.
Why it matters
You cannot improve pages you cannot see.
Sites grow, redirects pile up, and campaign URLs linger. Without a recurring crawl, SEO work happens on a partial map. Crawlox keeps the inventory current so missing titles, thin pages, and broken URLs show up before they waste crawl budget.
Catch crawl errors early
See 4xx and 5xx pages in the same workspace as healthy URLs, instead of discovering them in Search Console weeks later.
Feed structure and linking views
The same crawl powers site silos, the internal link graph, keyword inventory, and page-level SEO checks.
Keep the map fresh
Schedule crawls so the inventory tracks the live site instead of a one-off audit snapshot.
How it works
From authorized domain to a reviewable page list.
01
Authorize the root domain
Add a clean root domain and confirm you own it or have permission to crawl it.
02
Seed the crawl
Crawlox checks sitemap.xml and starts from the homepage so important URLs enter the queue early.
03
Follow internal links
The crawler expands through HTML links, staying focused on the authorized root.
04
Store page signals
Each URL keeps status, title, meta, headings, content depth, links, and an on-page SEO score.
05
Review and recrawl
Open Pages, Structure, Linking, and Keywords, then recrawl when content changes.
What the inventory includes
Every crawled URL carries the context teams need to act.
The page list is not just a URL dump. Crawlox attaches crawl and on-page fields so SEO and content work can start from evidence.
Canonical URL and path
See the crawled address, path depth, and whether the URL looks SEO-friendly.
HTTP status
Review 2xx, redirects, and error codes recorded on the latest crawl.
Title, H1, and meta description
Inspect the tags search engines use to understand and display the page.
Inbound and outbound links
Count internal links in and out so isolated pages are easy to spot.
On-page SEO score
Get a 0–100 score from title, meta, headings, content, images, HTTPS, and related checks.
Last seen in crawl
Know when the crawler last fetched the URL so stale pages are obvious.
What teams do next
Turn crawl output into a weekly SEO queue.
Filter pages that need work
Sort by SEO score, status code, or missing metadata and send the list to content owners.
Check structure after a redesign
Recrawl after information architecture changes and compare silos and path depth.
Export the inventory
Download crawled pages with SEO fields when you need a spreadsheet for a wider team.
FAQ
Website crawler questions.
Short answers for teams evaluating Crawlox as a recurring website crawler and page inventory.
What does the Crawlox website crawler do?
Crawlox crawls authorized root domains, discovers reachable pages from sitemaps and internal links, and stores on-page SEO signals for each URL in one workspace.
Does Crawlox crawl subdomains?
Discovered pages under a paid root domain stay in that slot. You authorize a root such as example.com, then review the inventory Crawlox collects beneath it.
How often can I crawl a domain?
You can schedule recurring crawls and launch an on-demand crawl when content, redirects, or releases change.
Is Crawlox a replacement for a desktop crawler?
Crawlox is a hosted crawler with a persistent workspace, scheduled runs, and SEO views for pages, structure, linking, keywords, sitemap, and health. It is built for ongoing monitoring, not a one-off local crawl.
Do I need to own the domain?
Yes. Every domain requires an explicit confirmation that you own it or have permission to crawl and analyze it.
What happens after the crawl finishes?
Pages appear in the inventory with SEO scores and checks. Structure, linking, keywords, sitemap health, and dashboard insights update from the same crawl.
Start crawling
Put your site map in one workspace.
Add an authorized domain, run the first crawl, and review every reachable page without operating crawler infrastructure.