feat(docs): add sitemap, llms.txt, robots and per-page share metadata - #787
Open
Decipher wants to merge 4 commits into
Open
feat(docs): add sitemap, llms.txt, robots and per-page share metadata#787Decipher wants to merge 4 commits into
Decipher wants to merge 4 commits into
Conversation
All 130 routes shared one set of Open Graph tags, set site-wide in nuxt.config.js, so every page presented the homepage's title, description and share image. None declared a canonical URL, leaving the trailing-slash and query-tagged variants of a URL as separate indexable pages. seoHead builds the full set per page and is now the only place any of it is decided. Every tag is keyed by hid so a page or layout can override one by name without emitting a second copy. Descriptions are derived from the first line of prose, because almost nothing in content/ sets one in frontmatter and @nuxt/content only surfaces frontmatter fields. Generated API pages are detected by structure and skipped: they are jsdoc output, and excerpting them yields fragments torn out of the middle of a tag. Those fall back to a description of their section, which beats repeating the site-wide blurb on 98 pages. Two modules were competing for the same tags and are no longer: - nuxt-social-meta is removed. Its tags were hid-keyed and so were being replaced rather than duplicated, but it stayed a second source of truth for the share title and image. What it uniquely supplied has moved into seoHead: og:image:width/height, which several scrapers want before they will render a card, and the Twitter handle. - @nuxtjs/pwa also emitted og:* and twitter:* built from package.json, so og:title read "druxtjs-org", the npm package name. It was invisible underneath nuxt-social-meta. The same leak reached apple-mobile-web-app-title and the web manifest, where a share-tag audit would not look. Its share tags are off and its name and description come from lib/site.js. Measured: 137 pages carried the package name, now none. Adds an error layout, whose purpose is its noindex. Seven files are generated for intermediate API directory paths that have no document, and the breadcrumbs on their own child pages link to them, so a crawler reaches a "Document not found" page by following ordinary links. Verified against a full generate: 130 content pages, each of the 12 share tags appearing exactly once on every one, and no page missing a canonical.
The site had none of the three. Every URL below the homepage was discoverable only by crawling links, and the generated API reference is the part of the site least likely to be linked from anywhere external. Written at generate:done, from the content tree rather than through @nuxt/content: its v1 package exports only Database, and instantiating one starts a file watcher that never releases the event loop, so a generate using it never exits. Frontmatter is parsed directly. content/ uses three single-line scalars, which is not worth a dependency, and the docs site is not a yarn workspace of the root, so anything it declares is absent from the root tooling that also reads these files. llms.txt lists the guide and module pages in full and the ten package API indexes under `## Optional`. Listing all 109 generated API pages buries what the project is under near-identical entries. Links carry UTM params, which is the only way to find out whether the file earns its place rather than guessing later; per-page canonicals mean the extra params cost nothing in search. Package entries are labelled from the route, because docgen titles each package index after the first symbol it documents. robots.txt allows everything except the build artefacts and points at the sitemap. Assistants are deliberately not blocked. Also fixes 59 API pages that were never generated. Nuxt discovers dynamic routes by crawling links, and the API index fetches its entries client side, so those links do not exist in the HTML for the crawler to follow. They existed only as the SPA fallback: unreachable by any crawler and impossible to put in a sitemap honestly. generate.routes now names every content route. Measured 50 of 109 API pages generated before, 116 after, and sitemap coverage is 130 of 130 with no URL lacking a page.
Every one of these modules documented a test seam it did not use, and the contracts were enforced only by running a full generate: a change to routeFor, excerpt or the description clamp would have corrupted sitemap.xml, llms.txt and every page's meta at once with nothing failing. 79 tests over the cases most worth pinning: README and index collapsing, the generated-HTML short circuit in excerpt, the TODO skip, frontmatter parsing including colons in values and quoted forms, XML escaping, clampDescription either side of the limit including the single-long-word branch, and the exact hid set seoHead returns. Fixes a jest config bug found while writing them. testPathIgnorePatterns and modulePathIgnorePatterns matched the bare string '/docs' against the whole absolute path, so any checkout whose own directory contained "docs" excluded the entire repository: in a worktree at .worktrees/docs-seo, jest found 0 of 72 test files and exited 1. Both are anchored to <rootDir> now and name the directories actually worth skipping. Suite goes from 72 files and 257 tests to 76 and 336.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Decipher
force-pushed
the
feature/docs-seo
branch
2 times, most recently
from
August 29, 2026 21:14
187c408 to
97d1b26
Compare
Decipher
force-pushed
the
feature/docs-seo
branch
from
August 29, 2026 21:34
97d1b26 to
68781d0
Compare
Decipher
force-pushed
the
feature/docs-seo
branch
from
August 29, 2026 21:58
68781d0 to
36a8afe
Compare
Decipher
force-pushed
the
feature/docs-seo
branch
from
August 29, 2026 22:44
36a8afe to
150e8d7
Compare
5 tasks
Decipher
force-pushed
the
feature/docs-seo
branch
from
August 29, 2026 23:14
150e8d7 to
29b97ac
Compare
7 tasks
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## feature/docs-site-redesign #787 +/- ##
===========================================================
Coverage 81.49% 81.49%
===========================================================
Files 112 112
Lines 2853 2853
Branches 616 576 -40
===========================================================
Hits 2325 2325
Misses 436 436
Partials 92 92 🚀 New features to boost your workflow:
|
Decipher
force-pushed
the
feature/docs-seo
branch
from
August 29, 2026 23:22
29b97ac to
857f5f5
Compare
Decipher
force-pushed
the
feature/docs-seo
branch
from
August 29, 2026 23:33
857f5f5 to
4d18c71
Compare
Adds an unlighthouse config and a Lighthouse audit to both pipelines. GitLab needs two jobs, because the halves cannot share a Node version: the site is pinned to Node 16 and unlighthouse needs 18+. GitHub Actions needs only one, because setup-node can switch versions mid-job. Either way the audit only speaks HTTP to the built site and never imports project code, so it runs on its own Node and the toolchain freeze is untouched. The GitHub job reuses the build job's artifact rather than rebuilding, since build:docs runs packages/docgen/bin/druxt-docgen.js, which is itself a build output. Nothing had downloaded that artifact before, so its layout was never exercised, and the job checks content/api arrived rather than trusting it. Without it the site still generates, just with 21 routes instead of 131. Routes come from dist/sitemap.xml with the origin stripped, so audit coverage and sitemap coverage are the same set by construction. Pointing the scanner at that file directly does not work: unlighthouse discards sitemap entries whose origin differs from the site being audited, which it always will, since the audit serves the build on localhost. It says so and falls back to crawling links, which quietly audits only what happens to be linked. Getting that right took three attempts, each of which reported success while auditing less than it claimed: 47 routes crawled, then 4 when the route list went unread and the crawler was disabled alongside it, then 69 in CI against 129 locally when the list was read from content/, which that job does not receive. The crawler is deliberately left enabled beneath the supplied list as a floor, and both jobs assert Playwright's Chromium was found, dist/sitemap.xml exists, and the sitemap advertises at least 100 routes, so a degraded run fails loudly rather than passing. Versions are pinned rather than floating on @latest. This toolchain is version-sensitive in ways that bite silently, and a config key that a future release stops reading would change an audit on a re-run. ci.budget is 0 for now. The first green run gives the numbers to set it from: 131 routes, worst page 0.93 overall, SEO 100 across the board.
Decipher
force-pushed
the
feature/docs-seo
branch
from
August 29, 2026 23:55
4d18c71 to
b121787
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
Merge from the bottom up. Each PR is based on the one below it, so the diff of
each shows only its own work.
develop#786Based on
feature/docs-site-redesign, so #786 must merge first. GitHub willretarget this PR to
developautomatically when it does.Stacked on #786. The diff shown here is only the SEO work; merge that one first.
The docs site had no sitemap, no robots.txt and no canonical URLs, and a single set of Open Graph tags shared by all 130 routes. Every page presented the homepage's title, description and share image.
What changed
sitemap.xmllastmod,priority,changefreqllms.txtrobots.txtFour commits, in dependency order: per-page metadata, then the generated indexes, then tests, then the CI audit.
The part worth reviewing closely
59 of 109 API pages were never being generated.
Nuxt discovers dynamic routes by crawling links out of pages it has already generated.
AppApiIndexfetches its entries client side, so most of those links do not exist in the generated HTML for the crawler to follow. Those 59 pages existed only as the SPA fallback: served by200.html, invisible to a crawler, and impossible to list in a sitemap honestly.generate.routesnow names every content route.distThis is a pre-existing bug. The sitemap only made it visible, by listing URLs that turned out to 404.
Decisions worth flagging
Content is read from disk, not through
@nuxt/content. Its v1 package exports onlyDatabase, and instantiating one starts a file watcher that never releases the event loop, so agenerateusing it never exits.Frontmatter is parsed directly rather than with a library.
content/uses three single-line scalars. The docs site is not a yarn workspace of the root, so a dependency it declares is missing from the root tooling that also reads these files: adding one broketest:unitandlint:knip, and relocating it to the root invalidated the CI cache and timed out six jobs. The parser is verified byte-identical togray-matteracross all 129 content files, and ignores anything it does not recognise rather than guessing.llms.txtdoes not list all 109 API pages. The guide and module pages are listed in full; the ten package API indexes go under## Optional. Listing every generated page buries what the project is under near-identical entries.Links in
llms.txtcarry UTM params. The convention is not settled, and tagging is the only way to learn whether the file earns its place. Per-pagerel=canonicalmeans the extra params cost nothing in search.Two modules were competing for the same tags.
nuxt-social-metais removed: its tags were hid-keyed and so were being replaced rather than duplicated, but it was a second source of truth for the share title and image. Removing it exposed a third source.@nuxtjs/pwaalso emitsog:*andtwitter:*built frompackage.json, soog:titlereaddruxtjs-org, the npm package name, and the same leak reachedapple-mobile-web-app-titleand the web manifest. Measured: 137 pages carried the package name, now none.The error layout exists for its
noindex. Seven files are generated for intermediate API directory paths that have no document, and the breadcrumbs on their own child pages link to them, so a crawler reaches a "Document not found" page by following ordinary links.Tests
79 tests over the four pure modules, covering README and index collapsing, the generated-HTML short circuit when excerpting, frontmatter edge cases, XML escaping, the description clamp either side of its limit, and the exact tag set
seoHeadreturns. The suite goes from 72 files and 257 tests to 76 and 336.This also fixes a jest config bug it turned up, unrelated to the SEO work and latent on
develop:testPathIgnorePatternsandmodulePathIgnorePatternsmatched the bare string/docsagainst the whole absolute path, so any checkout whose own directory contained "docs" excluded the entire repository. Both are anchored to<rootDir>now.Lighthouse audit
Adds an unlighthouse config and two CI jobs. They are split because the halves cannot share a Node version: the site is pinned to Node 16 and unlighthouse needs 18+. The audit only speaks HTTP to the built site, so it runs on its own image and never imports project code, leaving the toolchain freeze intact. Note this is wired into the GitLab pipeline; the equivalent GitHub Actions job is not included here.
Baseline over all 131 routes:
ci.budgetis0for now. The worst page scores 0.93 overall, so 85 is a sensible value to set once someone is happy for this to gate a merge.Routes come from
dist/sitemap.xmlwith the origin stripped, so audit coverage and sitemap coverage are the same set by construction. Pointing the scanner at that file directly does not work: unlighthouse discards sitemap entries from a different origin than the site it is auditing, which it always will here, and falls back to crawling links. That took three attempts, each reporting success while auditing less than it claimed, so the job now asserts the sitemap exists and advertises at least 100 routes.Known failures this surfaced, not fixed here
Both are colour decisions rather than mechanical fixes, so they are left for a separate change:
color-contraston 120 pages:badge-primary badge-outlinerenders#53b1ebon white at 2.37:1, against the 4.5:1 needed for small text.color-contraston 22 of those: Prism'stoken commentat 2.43:1, from the upstreammaterial-oceanictheme.Follow-ups filed separately
Two content problems this work depends on but does not fix, both tracked on the internal tracker rather than here:
modules/entity/deprecations.mdhas no frontmatter, so its title falls back to the package name and its navigation link renders with no accessible text. The generators fall back to a route-derived title, but the page itself still needs the fix.docgentitles each package's API index after the first symbol it documents, so/api/packages/siteis titledDruxtSiteMixin. Worked around here by labellingllms.txtentries from the route.Verification
Against a real
nuxt generate, not the config:noindex, 0 pages leaking the package namesitemap.xmlparses as XML, correct origins, no stray trailing slashes, every URL resolving to a generated file/llms.txt,/sitemap.xml,/robots.txtall return 200 from the served build