# https://www.robotstxt.org/robotstxt.html
#
# AI-crawler policy (S7c / GEO; ratified as deliberate policy 2026-07-11). A crawler obeys only
# its single most-specific User-agent group. Three tiers:
#
#   1. AI search & citation fetchers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User,
#      Claude-SearchBot, Applebot) are intentionally NOT given their own block — they inherit the
#      "User-agent: *" rules below: public marketing/SEO surfaces allowed (we WANT to be cited in
#      AI answers), logged-in app disallowed.
#   2. Training-control tokens are blocked ONLY where blocking cannot cost us citations: GPTBot
#      (OpenAI cites via OAI-SearchBot/ChatGPT-User regardless), Google-Extended (Search + AI
#      Overviews use regular Googlebot regardless), CCBot (bulk corpus, no answer surface of its
#      own).
#   3. Training crawlers that are ALSO the ingestion path into a vendor's assistant answers
#      (ClaudeBot, meta-externalagent, Amazonbot, Bytespider) are deliberately ALLOWED on public
#      surfaces: presence in those assistants' model knowledge is a GEO channel we value more than
#      blanket training-consent enforcement (evidence: Doubao referrals + Meta AI fetch volume,
#      GEO baseline 2026-07-11). Revisit if the consent stance changes.
#
# Token spellings verified against operator docs (2026): GPTBot / OAI-SearchBot / ChatGPT-User
# (developers.openai.com/api/docs/bots), Google-Extended (Google's robots training-control token),
# CCBot (Common Crawl), PerplexityBot (Perplexity).

# --- Training-control tokens with no citation cost: full-site disallow ---
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# --- Default group: human crawlers + AI search/citation crawlers (OAI-SearchBot, PerplexityBot,
#     ChatGPT-User) inherit these rules ---
User-agent: *
# Protected app surfaces
Disallow: /home
Disallow: /insights
Disallow: /insights/
Disallow: /insights/loading
Disallow: /collections
Disallow: /portfolio
Disallow: /settings
Disallow: /plans
Disallow: /manage-subscription
Disallow: /billing-outcome

# Public utility (not useful to index)
Disallow: /reset-password
Disallow: /verify-email

# S5A exact-preview capability page — never indexable (page also emits
# noindex,nofollow,noarchive via meta + X-Robots-Tag and is excluded from sitemaps/prerender)
Disallow: /article-preview

# Default allow everything else (marketing + /stock + /sectors + /articles)
Allow: /

Sitemap: https://stocklens.com/sitemap.xml
# Google News discovery: list the news sub-sitemap directly (it is also referenced by the index,
# but a direct listing removes the transitive-discovery hop for the freshness-critical ≤48h window).
Sitemap: https://stocklens.com/sitemap_news.xml