Skip to content

SEO And Discovery

This document describes the LMS discovery surfaces that expose only public, default-locale content for search engines and answer engines.

Implemented Endpoints

  • /sitemap.xml (src/app/sitemap.ts)
  • /robots.txt (src/app/robots.ts)
  • /.well-known/change-password redirect to /learn/forgot (next.config.ts)
  • /llms.txt (src/app/llms.txt/route.ts)
  • /llms-full.txt (src/app/llms-full.txt/route.ts)
  • /*.md (scoped rewrites to src/app/api/aeo/markdown/[...path]/route.ts)

Tenant And Locale Policy

  • Discovery routes resolve tenant context directly from request host via src/lib/seo/tenant-context.ts because dot-paths and /api/* bypass proxy.ts.
  • unknown hosts and archived tenants return 404 for text discovery routes.
  • Suspended tenants return disallow-all robots.txt; inventory-backed routes return empty content.
  • Discovery output is default-locale only. Locale-prefixed paths canonicalize to unprefixed routes through src/lib/seo/canonical.ts.
  • robots.txt explicitly disallows locale-prefixed variants of non-indexable LMS routes for non-default locales, while static platform paths (/api, /ingest, /_next) remain unprefixed.

Published URL Inventory

src/lib/seo/url-inventory.ts is the shared allowlist for discovery routes:

  • CMS pages come from getProjectPageItems and must have status === "published".
  • Structural markers (home, blog, learn) resolve like findStructuralNode: top-level or Header children only — footer acronym aliases are ignored.
  • Generic pages exclude structural and reserved slugs.
  • Blog posts are direct children of the preferred blog marker page.
  • Course URLs require both CMS publication and tenant-scoped catalog.listed=true.
  • Protected-route leakage is guarded by tests against PROTECTED_PATTERNS exported from src/proxy.ts.

Structured Data And Metadata

  • Public pages emit canonical default-locale metadata and markdown alternates (alternates.types["text/markdown"]) where applicable.
  • Auth-gated and token-bearing pages are noindex, nofollow.
  • JSON-LD builders live in src/lib/seo/json-ld.ts; scripts are rendered via src/components/seo/json-ld.tsx on public pages only.

Rate Limits

  • llms-full.txt: aeoCorpus policy (10 requests / minute / IP)
  • Markdown API: aeoMarkdown policy (60 requests / minute / IP)

Policies and keys are defined in:

  • src/lib/rate-limit/index.ts
  • src/lib/rate-limit/keys.ts

Testing

  • Unit/route tests:
    • src/app/sitemap.test.ts
    • src/app/robots.test.ts
    • src/app/llms.txt/route.test.ts
    • src/app/llms-full.txt/route.test.ts
    • src/app/api/aeo/markdown/[...path]/route.test.ts
    • tests/unit/lib/seo/discovery-boundary.test.ts
    • tests/unit/lib/aeo/widget-to-markdown.test.ts
    • tests/unit/next-config.discovery.test.ts
  • E2E coverage: tests/e2e/discovery.spec.ts