Building DIY SEO Hub: An SEO Crawl You Can Keep

A broken link is useful information when the report tells me which page contains it, where it points, and what response came back. A title warning becomes actionable when I can see the actual title and its length. That is the level of detail I wanted from DIY SEO Hub.

The centre of the site is the Site Spider. Give it a website address, let it collect the pages and their resources, and use that stored crawl to investigate the site. The same data powers the health summary, individual checks, page inspector, and downloadable SQLite file.

The five screenshots below were captured from the live site on September 11, 2026. They show real 8Examples crawls from September 6 and September 10. Their dates and scores describe those snapshots; they are not a fresh scan of today’s website.

Start with the website, without an account

The homepage has a domain field and a scan button. An account is optional for starting a scan. The result is an unlisted workspace link you can return to while the crawler runs. If you supply an email address, the form offers a completion report; it does not sign you up for a newsletter.

An account becomes useful when you want to keep scans together or schedule another run. This makes the first interaction small: enter the site you own or are authorised to check, then look at what the crawler finds.

The published limits at the time of writing are 5,000 pages and 1,000 assets per scan. Files over 10 MB are skipped and counted. The crawler follows internal links and sitemap entries, reads robots.txt, and identifies itself as DIYSEOHubBot. Those boundaries matter when you interpret a report: “not fetched” can mean a limit or a crawl rule, rather than a missing page.

Crawl once, ask several questions

The web application and the background processor have different jobs. The application accepts the request and serves the results. The processor works through the crawl queue, fetches URLs, extracts information, and records what happened. The browser can show progress without holding the original request open for the whole crawl.

Once that information is stored, running the title check does not visit every page again. Neither does checking headings or internal links. Those checks query the same snapshot. That makes it possible to move between questions without changing the evidence underneath each one.

The score is the beginning of the investigation

The health panel groups findings into things a site owner can act on: titles and descriptions, headings, broken links, redirects, response codes, indexability, image alt text, large files, internal links, and sitemap coverage. Each card opens the corresponding results.

A summary can also explain why something was excluded. A page deliberately marked noindex should not create the same editorial to-do list as a public landing page. An image with an explicitly empty alt attribute may be decorative; that is different from an image with no alt attribute at all.

The score comes from the application’s rules and weighted findings. It is a way to organise this crawl’s results. It does not measure Google rankings, traffic, or whether a particular page is in Google’s index. The site’s “SEO rank” directory likewise orders scanned sites by this health score.

Go from a warning to the actual page

Opening the title check gives me the URL, title, length, and reason for the warning. In this snapshot, four titles exceeded the tool’s 60-character threshold. The report puts those rows first and provides an inspect link beside each page.

A threshold gives me a review queue. I still decide whether a shorter title communicates the page well. The same principle applies to a short page or a large download: understand its purpose before changing it just to make a badge turn green.

The inspector brings a single URL’s evidence together. It shows the fetch result, content type, size, crawl depth, sitemap membership, and a link to the stored response. Further down are the extracted page fields, response headers, inbound and outbound links, and resources.

That distinction between the saved page and the current page is essential. After a fix, rerunning a check on the old database still examines the old crawl. A new crawl is how I check the deployed change.

The download is the crawl itself

The download button produces a single SQLite database containing the scan’s events, stored response bodies, and extracted data. It includes tables for URLs, pages, links, resources, images, and sitemap entries. You can open it with a SQLite browser or query it yourself.

For example, this query lists internal links whose fetched target returned an error:

SELECT links.from_url,
       links.to_url,
       urls.status_code
FROM links
JOIN urls ON urls.url = links.to_url
WHERE links.internal = 1
  AND urls.status_code >= 400
ORDER BY urls.status_code, links.from_url;

This is a starting query, not an exact copy of the product’s broken-link check. It includes throttled responses such as 429 and does not include requests that failed without an HTTP status. The stored data lets you make those distinctions explicitly.

Underneath the interface, each scan has its own database. Events and stored bodies are the source from which the query tables can be rebuilt. The download uses SQLite’s VACUUM INTO to create a consistent standalone copy. That gives a scan a useful life outside the running application.

Fix, test, deploy, then rescan

I have used these reports on 8Examples itself. One earlier finding was particularly concrete: two pages linked to /cdn-cgi/l/email-protection, which returned 404. That kind of report gives me a bounded problem to investigate instead of a vague instruction to “improve SEO.”

My working loop is to inspect the finding, decide whether it is appropriate to fix, make one reviewable change, test it, deploy it, and run another crawl. Separate commits help keep the reason for each change visible.

The 8Examples scan history keeps dated reports at their own addresses. The two illustrated reports are the September 6 snapshot and the September 10 snapshot. Comparing individual findings is more informative than treating the small difference between their overall scores as proof of an outcome.

For scans associated with the same browser or account, the workspace can compare changes in pages, links, status codes, redirects, titles, descriptions, and indexability. Accounts can schedule scans at one-, seven-, or thirty-day intervals. Change alerts are designed to send when the comparison finds changes, rather than emailing an identical report every time.

Keep the automation tied to the evidence

The background processor can also write an AI fix plan from check summaries and sampled findings. That helps translate a table into a short set of priorities. The plan is advice to verify. A suggestion to add a CDN, for example, is not evidence that the site lacks one; I still check the actual setup.

The crawler also needs to interpret responses carefully. A rate limit is a request to slow down. The implementation backs off and retries, then lists persistent throttling separately from ordinary errors. Treating those responses as a distinct condition makes the report easier to act on.

This version fetches and parses HTTP responses; it does not render every page in a full browser. Content that appears only after client-side JavaScript runs may not be present in the saved HTML. Together with crawl limits and robots rules, that defines what this particular report can tell me. Search Console and a real browser remain useful for questions outside that view.

Try it on a site you can improve

The part I most wanted to build is the path from a finding to inspectable evidence. I can start at the health panel, open a specific check, inspect the affected page, and keep the database. After making a change, I can come back with a new crawl and see what changed.

If you have a site that has accumulated old links, uneven metadata, or pages that are hard to reach, run it through the Site Spider. Pick one finding you understand, make a careful fix, and use the next scan to check your work.