foxxmd.dev / blog
[READ-ONLY] Mirror of https://github.com/FoxxMD/blog. blog.foxxmd.dev
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

Add robots.txt to block ai crawlers

FoxxMD (Apr 11, 2025, 8:22 PM UTC) 88c2ebc9 939115f8

+51
+51
robots.txt
··· 1 + User-agent: AI2Bot 2 + User-agent: Ai2Bot-Dolma 3 + User-agent: Amazonbot 4 + User-agent: anthropic-ai 5 + User-agent: Applebot 6 + User-agent: Applebot-Extended 7 + User-agent: Brightbot 1.0 8 + User-agent: Bytespider 9 + User-agent: CCBot 10 + User-agent: ChatGPT-User 11 + User-agent: Claude-Web 12 + User-agent: ClaudeBot 13 + User-agent: cohere-ai 14 + User-agent: cohere-training-data-crawler 15 + User-agent: Crawlspace 16 + User-agent: Diffbot 17 + User-agent: DuckAssistBot 18 + User-agent: FacebookBot 19 + User-agent: FriendlyCrawler 20 + User-agent: Google-Extended 21 + User-agent: GoogleOther 22 + User-agent: GoogleOther-Image 23 + User-agent: GoogleOther-Video 24 + User-agent: GPTBot 25 + User-agent: iaskspider/2.0 26 + User-agent: ICC-Crawler 27 + User-agent: ImagesiftBot 28 + User-agent: img2dataset 29 + User-agent: imgproxy 30 + User-agent: ISSCyberRiskCrawler 31 + User-agent: Kangaroo Bot 32 + User-agent: Meta-ExternalAgent 33 + User-agent: Meta-ExternalFetcher 34 + User-agent: OAI-SearchBot 35 + User-agent: omgili 36 + User-agent: omgilibot 37 + User-agent: PanguBot 38 + User-agent: Perplexity-User 39 + User-agent: PerplexityBot 40 + User-agent: PetalBot 41 + User-agent: Scrapy 42 + User-agent: SemrushBot-OCOB 43 + User-agent: SemrushBot-SWA 44 + User-agent: Sidetrade indexer bot 45 + User-agent: Timpibot 46 + User-agent: VelenPublicWebCrawler 47 + User-agent: Webzio-Extended 48 + User-agent: YouBot 49 + Disallow: / 50 + 51 + Sitemap: https://blog.foxxmd.dev/sitemap.xml