How to Find All Links on a Website, Page by Page
By Seth Black · Published
To find all links on a website, crawl it: a crawler opens every page it can reach, records each <a href> with its anchor text, and gives you one row per link with the page it was found on. For a single page, your browser’s console lists every link in a second. For a whole site, you need a crawler that keeps the source page next to each link, because “which page is this link on” is the question you’ll ask next.
This post is about the links themselves. If what you want is the list of pages, start with how to see all pages on a website, which covers sitemaps, Search Console and the rest.
How do you see all the links on one page?
Open the page, press F12, go to the Console tab and paste this:
console.table([...document.links].map(a => ({
href: a.href,
text: a.innerText.trim().slice(0, 80),
internal: a.hostname === location.hostname
})))
That gives you a table of every link after JavaScript has run, with anchor text and an internal flag. Right-click the table to copy it. It’s the quickest answer for one page and the wrong tool for a hundred.
From the command line, without a browser:
curl -s https://example.com/pricing \
| grep -oE 'href="[^"]+"' \
| sed 's/href="//; s/"$//' \
| sort -u
This reads the raw HTML, so it misses links injected by JavaScript and it doesn’t resolve relative URLs. Good enough for a quick look at a server-rendered page.
How to find all links on a website
A site-wide link list needs three things: every page, every link on each page, and the source page attached to each link. Here are the options, roughly in order of how much they give you.
| Method | Whole site | Source page per link | Anchor text | JavaScript links |
|---|---|---|---|---|
| Browser console | No | One page | Yes | Yes |
curl and grep |
No | One page | No | No |
wget --spider |
Yes | No | No | No |
| Desktop crawler | Yes | Yes | Yes | With rendering on |
| Search Console Links report | Yes | Samples | Samples | Yes |
wget, if it’s all you have
wget --spider -r -l inf -nd -nv -o crawl.log https://example.com
grep -oE 'https?://[^ ]+' crawl.log | sort -u > urls.txt
wget follows links and logs every URL it requests. What you get is the set of URLs it reached, not a list of links with their source pages, and it never runs JavaScript. It tells you what exists, not what links to what.
A crawler that keeps the link graph
This is what I use. bseoa stores every link it finds on every page, with the anchor text and whether the link is internal. Crawl once:
black-seo-analyzer \
--url-to-begin-crawl https://example.com \
--output-type json \
--output-file crawl.json
Each page in crawl.json has a complete_analysis.link_analysis array. Each entry has url, anchor_text, is_internal and is_file, with relative URLs already resolved to absolute ones. Flatten it into a spreadsheet with jq:
jq -r '
["source","link","anchor_text","internal"],
(.pages[] | .url as $src
| .complete_analysis.link_analysis[]?
| [$src, .url, .anchor_text, (.is_internal | tostring)])
| @csv' crawl.json > all-links.csv
That’s every link on the site, one row each, with the page it lives on. Add --spa to the crawl if the site builds its navigation with JavaScript; without rendering, a React or Vue site often shows a crawler nothing but the shell. I covered that failure in why your JavaScript crawl is broken.
Add --disable-external-links if you only care about the site’s own pages and don’t want every outbound link checked. The links are still recorded; they just aren’t fetched.
Search Console, for sites you own
Links in Search Console shows top linked pages and top linking text, internal and external. It’s Google’s view, which is useful, but it reports samples and top lists rather than every link with its source. Use it to check that your most important pages are the most linked, not as a full inventory.
What should you do with the link list?
A raw list of 40,000 links isn’t an answer to anything. These are the questions it answers well:
Which links are broken? Filter to internal links and check their status. bseoa does this for you with --output-type broken-links, which writes the page, the broken URL, the anchor text and the status code. The broken links report walks through reading it.
Which links point at redirects? Every internal link to a 301 costs users and crawlers an extra request, and they pile up after every migration. Join the link list against the crawl’s status_code and redirected_to fields and update the links at the source.
Which links have useless anchor text? “Click here”, “read more” and empty anchors tell search engines nothing about the target. Sort the CSV by anchor_text and the generic ones cluster at the top. The crawler also flags them per page as generic anchor text and empty anchor text.
What links to a specific page? That’s a filter on the link column: keep the rows where link is the target URL (and its trailing-slash variant), and the source column is your list of linking pages, with the anchor text each one uses.
Which pages have no internal links at all? Those are orphans, and a crawler can’t find them by following links. You find them by comparing the crawl with other URL sources; see how to find orphan pages.
Common mistakes when listing links
Counting the same link twice. Navigation and footer links appear on every page. A site with 300 pages and 60 template links has 18,000 template link rows. Before you draw conclusions about link counts, group by link and count distinct source values instead of raw rows.
Mixing up URL variants. https://example.com/page, https://example.com/page/ and https://www.example.com/page are different URLs to a crawler. If your link list shows all three, the site is linking inconsistently, and that’s a finding, not noise.
Forgetting links outside <a> tags. Canonical tags, hreflang annotations, pagination links and sitemaps also point at URLs. They aren’t navigational links, but search engines use them to discover pages. A full audit looks at all of them.
Trusting raw HTML on a JavaScript site. If the menu is built client-side, curl and non-rendering crawlers return almost no links. Compare a rendered crawl with a raw one; the difference is the set of links that depend on JavaScript.
The short version
For one page, use the browser console. For a whole site, crawl it with a tool that records the source page and anchor text for every link, then export to CSV and filter. bseoa’s website crawler records every link on every page with its anchor text and internal flag, and runs on your own machine, so a 50,000-link site is a few minutes and a jq command, not an export limit.
-Sethers