# RoofVista - Roofing Marketplace # https://roofvista.com # # Strategy: allow search engines + user-triggered AI assistants so RoofVista # can be cited accurately at answer time. Block training-only crawlers, # SEO spy bots, and scraper tools. # # Reference: https://platform.openai.com/docs/bots # OAI-SearchBot = ChatGPT search (drives traffic) → allowed # GPTBot = model training (consumes content) → blocked # ============================================ # Default — applies to any bot not named below # ============================================ User-agent: * Allow: / Allow: /api/sitemap Allow: /api/openai/openapi.json Allow: /api/openapi.json Allow: /api/pricing Allow: /api/agent/v1/ Allow: /api/mcp Allow: /api/public/ Disallow: /admin/ Disallow: /portal/ Disallow: /support/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /settings/ Disallow: /access Disallow: /api/ Disallow: /auth-debug/ Disallow: /debug-images/ Disallow: /image-test-minimal/ Disallow: /demo-language/ Disallow: /language-demo/ Disallow: /checkout/helcim-demo/ Disallow: /check/ Disallow: /diagnostics/ Disallow: /coming-soon/ Disallow: /.well-known/scraper-trap Crawl-delay: 2 # ============================================ # Traditional search engines — ALLOWED # ============================================ User-agent: Googlebot Allow: / Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: Bingbot Allow: / Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: DuckDuckBot Allow: / Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: Slurp Allow: / Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ # Kagi + Mojeek are independent indexes for privacy-conscious searchers. User-agent: KagiBot Allow: / Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: MojeekBot Allow: / Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ # ============================================ # AI Search Assistants — ALLOWED (they cite us at answer time) # Whitelist the public agent surface so they can fetch structured data. # Note: these are citation/search agents, NOT model-training crawlers. # Training crawlers stay blocked below. Google-Extended has a three-page, # founder-approved grounding canary; every other URL remains blocked. # ============================================ User-agent: Claude-User Allow: / Allow: /api/openai/openapi.json Allow: /api/openapi.json Allow: /api/pricing Allow: /api/agent/v1/ Allow: /api/mcp Allow: /api/public/ Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: Claude-SearchBot Allow: / Allow: /api/openai/openapi.json Allow: /api/openapi.json Allow: /api/pricing Allow: /api/agent/v1/ Allow: /api/mcp Allow: /api/public/ Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: ChatGPT-User Allow: / Allow: /api/openai/openapi.json Allow: /api/openapi.json Allow: /api/pricing Allow: /api/agent/v1/ Allow: /api/mcp Allow: /api/public/ Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: OAI-SearchBot Allow: / Allow: /api/openai/openapi.json Allow: /api/openapi.json Allow: /api/pricing Allow: /api/agent/v1/ Allow: /api/mcp Allow: /api/public/ Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: PerplexityBot Allow: / Allow: /api/openai/openapi.json Allow: /api/openapi.json Allow: /api/pricing Allow: /api/agent/v1/ Allow: /api/mcp Allow: /api/public/ Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: YouBot Allow: / Allow: /api/openai/openapi.json Allow: /api/openapi.json Allow: /api/pricing Allow: /api/agent/v1/ Allow: /api/mcp Allow: /api/public/ Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: Applebot Allow: / Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: Applebot-Extended Allow: / Allow: /api/openai/openapi.json Allow: /api/openapi.json Allow: /api/pricing Allow: /api/agent/v1/ Allow: /api/mcp Allow: /api/public/ Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: Meta-ExternalAgent Allow: / Allow: /api/openai/openapi.json Allow: /api/openapi.json Allow: /api/pricing Allow: /api/agent/v1/ Allow: /api/mcp Allow: /api/public/ Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ User-agent: Meta-ExternalFetcher Allow: / Allow: /api/openai/openapi.json Allow: /api/openapi.json Allow: /api/pricing Allow: /api/agent/v1/ Allow: /api/mcp Allow: /api/public/ Disallow: /admin/ Disallow: /contractor/ Disallow: /dashboard/ Disallow: /my-quotes/ Disallow: /api/ # ============================================ # AI Training Crawlers — BLOCKED BY DEFAULT # Scrape for model training, not search answers. They consume content # without sending users back. Google-Extended couples Gemini grounding with # future-model training, so only the approved public guide canary is open. # ============================================ User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Anthropic-AI Disallow: / User-agent: Google-Extended Allow: /resources/guides/roof-financing-bad-credit$ Allow: /resources/guides/architectural-shingles-cost$ Allow: /resources/guides/roof-replacement-cost-new-york$ Disallow: / User-agent: CCBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Bytespider Disallow: / User-agent: ByteSpider Disallow: / User-agent: Diffbot Disallow: / User-agent: Amazonbot Disallow: / User-agent: AmazonBot Disallow: / User-agent: FacebookBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / # ============================================ # SEO Spy Bots — BLOCKED # Don't give competitors free keyword/backlink intelligence. # ============================================ User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: PetalBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: SerendeputyBot Disallow: / User-agent: serpstatbot Disallow: / User-agent: BacklinksextendedBot Disallow: / User-agent: LinkpadBot Disallow: / # ============================================ # Non-US-market search engines — BLOCKED # RoofVista serves the US only. # ============================================ User-agent: YandexBot Disallow: / User-agent: Yandex Disallow: / User-agent: Baiduspider Disallow: / User-agent: Sogou Disallow: / User-agent: SogouSpider Disallow: / # ============================================ # Scraper Tools — BLOCKED # ============================================ User-agent: HTTrack Disallow: / User-agent: SiteSucker Disallow: / User-agent: WebCopier Disallow: / User-agent: WebReaper Disallow: / User-agent: WebStripper Disallow: / Sitemap: https://roofvista.com/sitemap.xml # AI Assistant Discovery # See /llms.txt for structured info about RoofVista for AI assistants. # Public action schema: https://roofvista.com/api/openai/openapi.json # MCP endpoint: https://roofvista.com/api/mcp # RSS feed: https://roofvista.com/feed.xml Feed: https://roofvista.com/feed.xml