# DeducePuzzles — crawl policy
# Marketing/content pages and the app are open to all crawlers and AI answer engines.
#
# Content-Signal (contentsignals.org, 2026-08-05) declares how we WANT this content used.
# It is a preference, not an enforcement mechanism — the Allow/Disallow lines are what
# actually govern access. We say yes to all three deliberately, which is the opposite of
# what a news publisher would say, and for a reason:
#   search=yes    — indexing and answer-time retrieval IS the acquisition channel.
#   ai-input=yes  — being cited in an AI answer is the point of /reasoning/*, /blog and llms.txt.
#   ai-train=yes  — unusual, and correct for us. We are an unknown brand with nothing
#                   premium to protect: puzzles are GENERATED and proven single-solution
#                   per request, so a training corpus cannot capture a puzzle bank that
#                   competes with us, and a future model that natively knows "DeducePuzzles =
#                   daily reasoning puzzle for IBPS/SBI" is awareness we cannot otherwise buy.
# The paid Pro tier sells explanation and adaptive practice, not these pages, so none of
# this gives away the thing anyone pays for. Reversing any of it is a one-word edit here.
#
# ⚠️ DO NOT ADD A NAMED BOT GROUP TO "WELCOME" AN AI CRAWLER. It does the opposite.
# A crawler obeys ONLY the most specific User-agent group that matches it — rules do not
# merge across groups (RFC 9309 §2.2.1). So adding
#     User-agent: OAI-SearchBot
#     Allow: /
# means OAI-SearchBot stops reading the `*` group below, and with it every Disallow: the
# /api/ 401-crawl-error fix, /admin, /console and /ingest all become fair game — for exactly
# the crawler you were trying to court. It also drops the Content-Signal line for that bot.
# The wildcard below ALREADY allows every crawler everything it should see, so a named
# group can only subtract. (Proposed by an outside SEO review on 2026-08-24 and rejected on
# these grounds; if it comes back, this is the answer.) A named group is correct only when
# you want DIFFERENT rules for that bot — and then it must restate the Disallow lines.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
# The admin dashboard is not for indexing.
Disallow: /admin.html
Disallow: /admin
Disallow: /admin/
Disallow: /console
Disallow: /console/

# The API is not a crawlable surface (Search Console, 2026-07-28: "Blocked due to
# unauthorised request (401)" reported as a NEW reason preventing pages from being indexed).
# deducepuzzles.com/api/* is a same-origin zone route to the Worker (wrangler.toml), and
# every session-gated endpoint answers an unauthenticated request with 401 no_session
# (worker/src/index.ts) — so each /api URL Googlebot reaches becomes a crawl error against
# the property. Disallowing it is the correct end state, not a workaround: these endpoints
# have no indexable content, and the 401 is the API behaving properly.
Disallow: /api/

# The first-party PostHog reverse proxy (worker/src/ingest.ts, POSTHOG_HOST). An analytics
# ingest path has nothing to index and should never consume crawl budget.
Disallow: /ingest

Sitemap: https://deducepuzzles.com/sitemap-index.xml
Sitemap: https://deducepuzzles.com/blog/sitemap.xml

# Agent-readable product data (two hops from this file to a full product record).
# products.json: every sellable SKU in one file.
# p/<slug>.json: single-item machine feed per product.
# p/<slug>.md: plain markdown mirror per product.
# llms.txt: full product context for LLMs and answer engines.
# Same file is also served at /.well-known/llms.txt.