fix(website): SEO, noindex, canonical, nosnippet & more

[sam-goodwin]

SEO pass. Google Search Console has been surfacing non-content (terminal chrome, the copy-for-agent prompt, raw .md mirrors, mirror hosts) as results and snippets.

One indexable origin

https://alchemy.run is canonical. Every other host the worker answers on — main.alchemy.run, PR previews, future aliases — is a mirror:

// src/worker.ts — on any non-canonical host
X-Robots-Tag: noindex            // every response
<link rel="canonical">           // left pointing at https://alchemy.run (no longer rewritten)
robots.txt                       // Allow: /, no Sitemap: line
og:url / og:image / twitter:image // still rewritten so previews unfurl themselves

Before this, main.alchemy.run/getting-started/ declared itself canonical and was fully indexable.

robots.txt + markdown mirrors

alchemy.run/robots.txt was serving only Cloudflare's content-signals block — no Sitemap:. Now website/public/robots.txt advertises the sitemap.

.md mirrors (/getting-started.md, Accept: text/markdown) and llms*.txt get X-Robots-Tag: noindex — crawlable so the directive is seen, out of the index. DuckDuckGo/Google were listing alchemy.run/guides/cloudflare.md with import { Tabs, TabItem } from '@astrojs/starlight/components'; as the snippet.

v1-era URLs

/docs/getting-started, /docs/providers/aws-control/…, /guides/cloudflare are still indexed and 404 on prod. On a prod 404 under /docs/, /guides/, /providers/, /concepts/ the worker HEADs v1.alchemy.run and 301s there when it has the page.

data-nosnippet

On every simulated terminal / browser / diagram island on the homepage, the docs Terminal, DAG, LoopDiagram figures, and both SSR'd copies of the 2.8 KB copy-for-agent prompt. Live snippet before: Learn more →. ‹›↻. ○localhost:1337/ HMR. ~/my-appDEV.

Generated reference pages are noindex, follow

/providers/** is 4,070 of the site's 4,412 pages (AWS alone 3,584), each a one-paragraph summary plus one snippet. Indexed individually they read as programmatic thin content and are what surfaces as "random" results instead of the hub/guide pages. The generator now emits:

head:
  - tag: meta
    attrs: { name: robots, content: "noindex, follow" }

Still navigable, still crawled (follow), still in Pagefind. follow keeps link equity flowing to the hub pages. Lifting it later is one line in scripts/generate-api-reference.ts.

The sitemap no longer needs a hand-maintained exclusion list: a collectNoindexPages integration scans the build output for a noindex robots meta and the sitemap filter drops those URLs — covers the reference pages and /auth/* alike.

Structured data

Organization (sameAs → GitHub, X, Discord, npm) + WebSite JSON-LD and twitter:site on every page (src/components/StructuredData.astro).

Also

• /auth/* pages noindex, follow and out of the sitemap
• blog pagination pages (/blog/2/…) out of the sitemap
• homepage description 254 → 154 chars

🤖 Generated with Claude Code

[alchemy-version-bot]

Website Preview Deployed

URL: https://alchemyeffectwebsite-website-pr-1269-6v4i3nu2hxiftalm.testing-2b2.workers.dev

Built from commit b8512c5.

---

This comment updates automatically with each push.

[sam-goodwin]

@BlankParticle did you have a chance to look at this? We need to fix the SEO issue on google

[BlankParticle]

i forgot, will take a look

[alchemy-version-bot]

Website Preview Deployed

URL: https://pr-1269-alchemyeffectwebsite-website-pjklbtzujol4cpasllfbla6qj.testing-2b2.workers.dev

Built from commit 403c635.

---

This comment updates automatically with each push.