User-agent: * Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin Sitemap: https://scriptgrain.com/sitemap.xml # ── AI crawlers — explicitly welcome ────────────────────────────────────────── # Added 2026-08-18. Modelled on hilyt.it. Absent stanzas are ambiguous, not # permissive: several AI crawlers treat "no explicit rule" conservatively, so # saying nothing is not the same as saying yes. # # A crawler that matches a named group obeys ONLY that group and ignores the # "*" group above, so each group repeats the app-route Disallow lines # (2026-09-25; before that GPTBot and the rest were allowed into /auth, # /dashboard and every other private route). src/test/robots.test.ts keeps the # lists identical. User-agent: GPTBot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin User-agent: ChatGPT-User Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin User-agent: OAI-SearchBot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin User-agent: ClaudeBot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin User-agent: Claude-Web Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin User-agent: anthropic-ai Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin User-agent: PerplexityBot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin User-agent: Google-Extended Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin User-agent: CCBot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin User-agent: Applebot-Extended Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin User-agent: cohere-ai Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin User-agent: DuckAssistBot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /upload Disallow: /processing Disallow: /generate Disallow: /history Disallow: /brain Disallow: /profile Disallow: /profiles Disallow: /compare Disallow: /voice-comparison Disallow: /transfer Disallow: /rewrite Disallow: /library Disallow: /mimic Disallow: /campaign Disallow: /settings Disallow: /payment-success Disallow: /reset-password Disallow: /invite Disallow: /claim Disallow: /unsubscribe Disallow: /admin