thehale.dev / thehale-github-io
Joseph Hale's Technical Blog
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

feat: Add settings to exclude content AI bots

Google's Bard and OpenAI's ChatGPT have promised to respect `robots.txt`
instructions with regards to preferences for including website content
in their training data.

This commit adds settings to `_config.yml` to help users choose which
content, if any, they want to request be excluded from the datasets for
these two bots.

The default settings exclude all website content from Bard and ChatGPT,
but keep website content available for search engine crawlers like
Googlebot and Bingbot.

Note that bots are free to ignore the contents of `robots.txt`, so
enabling these settings does not prevent website content from being
included in training datasets.

Learn more: https://www.eff.org/deeplinks/2023/12/no-robotstxt-how-ask-chatgpt-and-google-bard-not-use-your-website-training

Joseph Hale (Feb 6, 2024, 7:00 PM -0700) 35040e4f 0a1f049c

+41
+16
_config.yml
··· 120 120 # The base URL of your site 121 121 baseurl: "" 122 122 123 + # By default, AI bots scrape website content to train/improve their models. 124 + # 125 + # These settings configure the `robots.txt` file to request exclusion of 126 + # website content from certain AI training datasets. 127 + # 128 + # Learn more: https://www.eff.org/deeplinks/2023/12/no-robotstxt-how-ask-chatgpt-and-google-bard-not-use-your-website-training 129 + ai_bot_scraping: 130 + chatgpt: # by OpenAI 131 + # Request exclusion of pages starting with the following path. 132 + # A blank value allows inclusion in AI training sets. 133 + disallow: / 134 + bard: # by Google 135 + # Request exclusion of pages starting with the following path. 136 + # A blank value allows inclusion in AI training sets. 137 + disallow: / 138 + 123 139 footer: 124 140 show_chirpy_attribution: true # The attribution in the site footer. 125 141
+25
assets/robots.txt
··· 2 2 permalink: /robots.txt 3 3 # The robots rules 4 4 --- 5 + # 6 + # robots.txt 7 + # 8 + # This file is to prevent the crawling and indexing of certain parts 9 + # of your site by web crawlers and spiders run by sites like Yahoo! 10 + # and Google. By telling these "robots" where not to go on your site, 11 + # you save bandwidth and server resources. 12 + # 13 + # This file will be ignored unless it is at the root of your host: 14 + # Used: http://example.com/robots.txt 15 + # Ignored: http://example.com/site/robots.txt 16 + # 17 + # For more information about the robots.txt standard, see: 18 + # http://www.robotstxt.org/robotstxt.html 5 19 6 20 User-agent: * 7 21 8 22 Disallow: /norobots/ 9 23 10 24 Sitemap: {{ '/sitemap.xml' | absolute_url }} 25 + 26 + {% if site.ai_bot_scraping.chatgpt.disallow %} 27 + # OpenAI ChatGPT 28 + User-agent: GPTBot 29 + Disallow: {{ site.ai_bot_scraping.chatgpt.disallow }} 30 + {% endif %} 31 + {% if site.ai_bot_scraping.bard.disallow %} 32 + # Google Bard 33 + User-agent: Google-Extended 34 + Disallow: {{ site.ai_bot_scraping.bard.disallow }} 35 + {% endif %}