robots.txt + canonical tags

What TestSprite reads

The testing agent reads this JSON, opens the deployed URL in headless Chromium, executes each action step, evaluates each assertion. Verdict: passed / failed / blocked / inconclusive.

{
  "projectId": "1ad26753-ee03-4689-8f0f-6fa5d67c5c72",
  "type": "frontend",
  "name": "Surfaces — robots.txt allows crawling and references the sitemap",
  "description": "GET /robots.txt must exist, not 404, allow crawling (no global Disallow: /) and include a Sitemap: reference. Catches the common gap where bots can crawl but social-share preview cards never warm up because the sitemap isn't discoverable.",
  "priority": "p2",
  "metadata": {
    "category": "surfaces",
    "stage": "all"
  },
  "planSteps": [
    {
      "type": "action",
      "description": "Navigate to /robots.txt"
    },
    {
      "type": "assertion",
      "description": "Verify the response status is 200 and the content-type is text/plain"
    },
    {
      "type": "assertion",
      "description": "Verify the body does NOT contain a global 'Disallow: /' rule (which would block all crawlers from the entire site)"
    },
    {
      "type": "assertion",
      "description": "Verify the body contains a 'Sitemap: ' line pointing to /sitemap.xml on the same origin"
    }
  ]
}