--- title: "Blog - bseoa" description: "Read the latest articles and insights on SEO, AI, and command-line tools from bseoa." image: "https://www.blackseoanalyzer.com/static/images/black-seo-analyzer-og-image.png" canonical: "https://www.blackseoanalyzer.com/en/blog" language: "en" --- # Our Blog ## What Is a Non-Canonical URL? Published on September 17, 2026 A non-canonical URL is a duplicate of a page that search engines index under another URL. When that's fine, when it hurts, and how to fix canonical problems. [Read More](https://www.blackseoanalyzer.com/en/blog/what-is-a-non-canonical-url) ## X-Robots-Tag: How to Use the HTTP Header Published on September 17, 2026 How the X-Robots-Tag HTTP header works: supported directives, syntax, Apache and Nginx examples, noindexing PDFs and staging sites, and how to check it. [Read More](https://www.blackseoanalyzer.com/en/blog/x-robots-tag) ## Technical SEO Audit Checklist Published on September 17, 2026 A technical SEO audit checklist in the order to run it: crawlability, indexability, structure, metadata, Core Web Vitals, mobile, security and international. [Read More](https://www.blackseoanalyzer.com/en/blog/technical-seo-audit-checklist) ## AhrefsBot and SemrushBot: What They Are and How to Control Them Published on September 17, 2026 What AhrefsBot, SemrushBot and their site audit bots do, whether blocking them affects SEO, and robots.txt rules to slow down or block each one. [Read More](https://www.blackseoanalyzer.com/en/blog/ahrefsbot-and-semrushbot) ## How to Find Orphan Pages Published on September 17, 2026 Find orphan pages by comparing a crawl with your sitemap, Search Console, analytics, logs and CMS, then decide whether to link, redirect, noindex or remove each. [Read More](https://www.blackseoanalyzer.com/en/blog/how-to-find-orphan-pages) ## How to See All Pages on a Website Published on September 17, 2026 See every page on a website using the sitemap, a site: search, Search Console, a crawler, your CMS, analytics or the Wayback Machine, and find orphan pages. [Read More](https://www.blackseoanalyzer.com/en/blog/how-to-see-all-pages-on-a-website) ## How to Find Links to Your Website Published on September 17, 2026 Find sites that link to yours with Search Console, Bing Webmaster Tools and free backlink checkers, then find every internal link to a page by crawling your site. [Read More](https://www.blackseoanalyzer.com/en/blog/how-to-find-links-to-your-website) ![Ecommerce Core Web Vitals: Fix LCP, CLS, and INP Across Your Catalog](https://cdn.blackseoanalyzer.com/blog-images/ecommerce-seo-fix-core-web-vitals-now.jpg) ## Ecommerce Core Web Vitals: Fix LCP, CLS, and INP Across Your Catalog Published on September 16, 2026 Ecommerce Core Web Vitals fail in predictable ways: product images, unsized tiles, variant pickers, and tag bloat. How I fix them template by template. [Read More](https://www.blackseoanalyzer.com/en/blog/ecommerce-seo-fix-core-web-vitals-now) ![How to Audit an Ecommerce Site for Crawl Waste and Index Bloat](https://cdn.blackseoanalyzer.com/blog-images/how-to-audit-an-ecommerce-site-for-crawl-waste-and-index-bloat.jpg) ## How to Audit an Ecommerce Site for Crawl Waste and Index Bloat Published on September 16, 2026 Count URLs three ways, find indexed pages that never earned a click, and pick one signal per pattern. The index bloat audit I run on ecommerce sites. [Read More](https://www.blackseoanalyzer.com/en/blog/how-to-audit-an-ecommerce-site-for-crawl-waste-and-index-bloat) ![A Practical Technical SEO Checklist for Agencies That Need to Ship](https://cdn.blackseoanalyzer.com/blog-images/a-practical-technical-seo-checklist-for-agencies-that-need-to-ship.jpg) ## A Practical Technical SEO Checklist for Agencies That Need to Ship Published on September 16, 2026 The 200-bullet technical SEO checklist is why audits die in a Drive folder. A time-boxed, three-pass audit checklist you can actually ship in a week. [Read More](https://www.blackseoanalyzer.com/en/blog/a-practical-technical-seo-checklist-for-agencies-that-need-to-ship) ![Buying a Technical SEO Audit for React and Angular Apps](https://cdn.blackseoanalyzer.com/blog-images/buying-technical-seo-audits-for-react-angular-apps.jpg) ## Buying a Technical SEO Audit for React and Angular Apps Published on September 16, 2026 Most SPA audits crawl your index.html shell and miss what Google sees. What a real React or Angular audit covers, and the questions to ask before you sign. [Read More](https://www.blackseoanalyzer.com/en/blog/buying-technical-seo-audits-for-react-angular-apps) ![Auditing Faceted Navigation: Find and Fix Your Crawl Budget Sinkhole](https://cdn.blackseoanalyzer.com/blog-images/auditing-faceted-navigation-find-and-fix-your-crawl-budget-sinkhole.jpg) ## Auditing Faceted Navigation: Find and Fix Your Crawl Budget Sinkhole Published on September 16, 2026 Faceted navigation can bury products under millions of filter URLs. Find the waste in your logs, sort parameters, and pick canonical, robots, or noindex. [Read More](https://www.blackseoanalyzer.com/en/blog/auditing-faceted-navigation-find-and-fix-your-crawl-budget-sinkhole) ![The Pre-Deployment SEO Checklist for Developers](https://cdn.blackseoanalyzer.com/blog-images/the-pre-deployment-seo-checklist-for-developers.jpg) ## The Pre-Deployment SEO Checklist for Developers Published on September 16, 2026 Most SEO disasters ship in a routine PR. The pre-merge SEO checklist I use, plus a CI regression check that fails the build before traffic drops. [Read More](https://www.blackseoanalyzer.com/en/blog/the-pre-deployment-seo-checklist-for-developers) ![Auditing at Scale: Crawling 10 Million Pages Without Crashing](https://cdn.blackseoanalyzer.com/blog-images/auditing-at-scale-taming-10-million-pages-without-crashing.jpg) ## Auditing at Scale: Crawling 10 Million Pages Without Crashing Published on September 16, 2026 How to crawl 10M URLs without your tool giving up at 4 million: the URL frontier, storage, chained crawls, orphan diffs, and the log data that matters. [Read More](https://www.blackseoanalyzer.com/en/blog/auditing-at-scale-taming-10-million-pages-without-crashing) ![Fixing Duplicate Meta Descriptions at Scale](https://cdn.blackseoanalyzer.com/blog-images/fixing-duplicate-metas-beyond-the-obvious-copy-paste-errors.jpg) ## Fixing Duplicate Meta Descriptions at Scale Published on September 16, 2026 Exact-match duplicate meta descriptions are easy. The damage is in templates producing thousands of near-identical strings. How to find and fix them. [Read More](https://www.blackseoanalyzer.com/en/blog/fixing-duplicate-meta-descriptions-at-scale) ![The AI Crawler robots.txt Guide: GPTBot, ClaudeBot, PerplexityBot, and the Rest](https://cdn.blackseoanalyzer.com/blog-images/ai-crawler-robots-txt-guide.jpg) ## The AI Crawler robots.txt Guide: GPTBot, ClaudeBot, PerplexityBot, and the Rest Published on September 16, 2026 Which AI bots train, which index for search, and which fetch for users. Exact user-agent tokens, copy-paste robots.txt recipes, and how to verify bots. [Read More](https://www.blackseoanalyzer.com/en/blog/ai-crawler-robots-txt-guide) ![Can AI Crawlers See Your JavaScript Content?](https://cdn.blackseoanalyzer.com/blog-images/can-ai-crawlers-see-javascript-content.jpg) ## Can AI Crawlers See Your JavaScript Content? Published on September 16, 2026 GPTBot, ClaudeBot, and PerplexityBot read raw HTML and don't run JavaScript. How to test what they see and fix it with SSR, SSG, or prerendering. [Read More](https://www.blackseoanalyzer.com/en/blog/can-ai-crawlers-see-javascript-content) ![llms.txt: Does It Matter?](https://cdn.blackseoanalyzer.com/blog-images/llms-txt-does-it-matter.jpg) ## llms.txt: Does It Matter? Published on September 16, 2026 What the llms.txt spec is, what llms-full.txt really is, who actually reads the file, and a minimal example. Add it, but don't expect rankings. [Read More](https://www.blackseoanalyzer.com/en/blog/llms-txt-does-it-matter) ![Black SEO Analyzer v26.4.16 Release](https://cdn.blackseoanalyzer.com/blog-images/v26.4.16-update.png) ## Black SEO Analyzer v26.4.16 Release Published on April 17, 2026 v26.4.16 is a UX polish release for the Windows GUI: right-click context menus, scroll position memory, and a first-run experience that actually makes sense. [Read More](https://www.blackseoanalyzer.com/en/blog/black-seo-analyzer-v26416-release) ![Structured Data Errors That Kill Rich Results](https://cdn.blackseoanalyzer.com/blog-images/structured-data-errors-that-kill-rich-results.jpg) ## Structured Data Errors That Kill Rich Results Published on January 28, 2026 Perfect JSON-LD that passes every validator can still produce zero rich results, and slightly broken schema can work for months until it doesn't. Here's how to find the errors that actually cost you SERP real estate. [Read More](https://www.blackseoanalyzer.com/en/blog/structured-data-errors-that-kill-rich-results) ![Technical SEO for Developers](https://cdn.blackseoanalyzer.com/blog-images/technical-seo-for-developers.jpg) ## Technical SEO for Developers Published on January 22, 2026 Technical SEO breaks between quarterly audits and nobody notices until rankings drop. Here's how to wire it into your CI pipeline so a failing canonical tag blocks the merge, same as any other test. [Read More](https://www.blackseoanalyzer.com/en/blog/technical-seo-for-developers) ![How to Diagnose Core Web Vitals Failures at the Page Level](https://cdn.blackseoanalyzer.com/blog-images/how-to-diagnose-core-web-vitals-failures-at-the-page-level.jpg) ## How to Diagnose Core Web Vitals Failures at the Page Level Published on January 20, 2026 Google Search Console hands you a bucket of 1,247 "Poor LCP" URLs and calls it a day. Here's how to actually diagnose Core Web Vitals failures at the page level using field data, DevTools, and the stuff Search Console won't tell you. [Read More](https://www.blackseoanalyzer.com/en/blog/how-to-diagnose-core-web-vitals-failures-at-the-page-level) ![Why Your Site Audit Tool Is Missing Half Your Pages](https://cdn.blackseoanalyzer.com/blog-images/why-your-site-audit-tool-is-missing-half-your-pages.jpg) ## Why Your Site Audit Tool Is Missing Half Your Pages Published on January 19, 2026 Your SEO crawler is probably missing half your site and telling you everything looks great. Here's where URLs actually hide and how to test whether your current tool is finding them. [Read More](https://www.blackseoanalyzer.com/en/blog/why-your-site-audit-tool-is-missing-half-your-pages) ![Hreflang: Why It Breaks, How to Audit It, and How to Fix It](https://cdn.blackseoanalyzer.com/blog-images/hreflang-why-it-breaks-how-to-audit-it-and-how-to-fix-it.jpg) ## Hreflang: Why It Breaks, How to Audit It, and How to Fix It Published on January 18, 2026 The seven hreflang errors I find on almost every international site, with the exact markup, and how to audit the whole hreflang graph at once. [Read More](https://www.blackseoanalyzer.com/en/blog/hreflang-why-it-breaks-how-to-audit-it-and-how-to-fix-it) ![Why Your JavaScript Crawl Is Broken (And How to Fix It)](https://cdn.blackseoanalyzer.com/blog-images/why-your-javascript-crawl-is-broken-and-how-to-fix-it.jpg) ## Why Your JavaScript Crawl Is Broken (And How to Fix It) Published on January 16, 2026 What Googlebot actually renders, why most crawlers audit your React app's empty shell, and the JavaScript rendering failures I check first. [Read More](https://www.blackseoanalyzer.com/en/blog/why-your-javascript-crawl-is-broken-and-how-to-fix-it) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://www.blackseoanalyzer.com/#organization","name":"Fiscus Technology, LLC","url":"https://www.blackseoanalyzer.com/","logo":"https://www.blackseoanalyzer.com/static/images/black-seo-analyzer.png","founder":{"@type":"Person","name":"Seth Black"}},{"@type":"WebSite","@id":"https://www.blackseoanalyzer.com/#website","name":"bseoa","url":"https://www.blackseoanalyzer.com/","publisher":{"@id":"https://www.blackseoanalyzer.com/#organization"}},{"@type":"CollectionPage","@id":"https://www.blackseoanalyzer.com/en/blog","name":"Blog - bseoa","url":"https://www.blackseoanalyzer.com/en/blog","description":"The official blog of bseoa, featuring articles and insights on SEO, AI, and command-line tools.","isPartOf":{"@id":"https://www.blackseoanalyzer.com/#website"}},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://www.blackseoanalyzer.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://www.blackseoanalyzer.com/en/blog"}]}]} ```