User-Agent: * Allow: / Disallow: /api/ Disallow: /cart Disallow: /checkout Disallow: /(auth)/ Disallow: /crm Disallow: /crm/ User-Agent: GPTBot User-Agent: ChatGPT-User User-Agent: ClaudeBot User-Agent: PerplexityBot Allow: / Sitemap: https://marlows-diamonds.co.uk/sitemap.xml