How to Find and Fix Broken Internal Links at Scale

A broken link report with 9,000 rows looks like a disaster. Most of the time it’s six bugs repeated across a few templates. Somebody renamed a category, the footer still points at the old one, and now every page on the site carries the same bad link.

Fixing these one URL at a time is how teams burn a quarter and end up right where they started. The goal is to find the handful of sources producing most of the damage and fix those first.

Crawl with the source page attached

A list of broken URLs alone is useless. You need the source page, the destination, and the status code, or you’re just staring at addresses. Anchor text helps too, because “Shop Winter Boots” tells you more than a row number does. A broken links report that shows source page, anchor text, and status code in one row is the difference between a repair queue and a wall of addresses.

Run the crawl with rendering on if your navigation is built with JavaScript. Otherwise you’ll miss the links your users actually click. If you’re not sure whether your crawler is actually executing JavaScript before following links, the gap between what renders in a browser and what a crawler sees is larger than most teams expect.

Group by source template

Skip this and you’re fixing the wrong things. Take the source URLs and collapse them into templates. /blog/2021/03/some-post and /blog/2023/11/another-post are both /blog/*. If 4,000 broken links come from /products/*, you have one bug in the product template, not 4,000 problems.

A few lines of Python will get you there:

import pandas as pd

df = pd.read_csv("broken-links.csv")
df["template"] = df["source"].str.extract(r"^https?://[^/]+(/[^/]+)/")
summary = (
    df.groupby(["template", "destination"])
      .size()
      .sort_values(ascending=False)
)
print(summary.head(20))

The top 20 rows of that output are usually your whole week.

Sort stale URLs from redirectable ones

When you sort the destinations, two types show up: dead ends and pages that just moved.

Stale URLs point at things that are gone for good. A discontinued product, a deleted author page, or a promo from a sale that ended. Remove the link or point it somewhere useful. Don’t redirect everything to the homepage. That’s the SEO equivalent of answering every question with “I don’t know,” and Google catches it and calls it a soft 404.

Redirectable URLs have a clear new home. The page moved, the slug changed, or the category got restructured. Add the 301, but also update the link at the source. A redirect is a patch. The link in the nav template is the actual fix, and leaving it alone means every new page inherits the extra hop.

Also check the links that already redirect. Internal links pointing at 301 chains aren’t technically broken, but each one adds a request for every user and every crawler. They pile up fast after a migration. The same grouping approach that works for broken links applies to crawl errors more broadly — a large export is almost always a small number of real problems in disguise.

Prioritize by link count and user path

The order comes down to two things.

How many internal links point at this broken destination? A 404 linked from the global footer is on every page of the site. A 404 linked from one blog post in 2019 can wait.

Does it block something people are trying to do? I once shipped a fix that looked clean in Chrome, skipped QA in the browser the actual customers were using, and took down an e-commerce checkout on launch weekend. The broken link was the buy button. A broken link in the checkout flow, the pricing page, or the signup path matters more than 300 broken links in an archive nobody visits. Test like a user, in the browser your users have. For ecommerce sites specifically, auditing by revenue impact rather than raw link count changes which template you fix first.

Turn the export into a queue

Your final list should be short and ordered. One row per template and destination pair, with a count, a bucket (stale or redirect), and an owner. A programmer should be able to pick up the first row and go straight to the template file, with no context meeting and no ticket archaeology.

bseoa’s filtered issue lists get you most of the way there. Filter to internal broken links, sort by inlink count, and you have a repair queue instead of a wall of warnings.

After each fix ships, re-crawl the affected URLs. Don’t wait for Search Console. That data lags by days, and you’ll have already moved on to the next template by the time it catches up.

bseoa writes source page, destination, status code, and redirect path in a single row so you’re not joining three exports in a spreadsheet. See how it works.

-Sethers