Remove the robots.txt rules that block the crawlers AI engines use to read your pages when answering. Crawlers used for model training are a separate choice and can stay as they are.
High impactQuick (< 15 min)1 page
Why
Blocked AI crawlers: GPTBot, ClaudeBot, CCBot
User-agent: * Allow: / # AI crawlers — allow citation User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: CCBot Allow: / User-agent: anthropic-ai Allow: / Sitemap: https://www.streetpress.com/sitemap.xml