Pricefield | Lemmy
  • Communities
  • Create Post
  • Create Community
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
@[email protected] to lemmy.ml [email protected] • 2 years ago

Should lemmy.ml block chatgpt scraping in robots.txt?

message-square
15
fedilink
41
message-square

Should lemmy.ml block chatgpt scraping in robots.txt?

@[email protected] to lemmy.ml [email protected] • 2 years ago
message-square
15
fedilink

Some context about this here: https://arstechnica.com/information-technology/2023/08/openai-details-how-to-keep-chatgpt-from-gobbling-up-website-data/

the robots.txt would be updated with this entry

User-agent: GPTBot
Disallow: /

Obviously this is meaningless against non-openai scrapers or anyone who just doesn’t give a shit.

  • harshvoldy
    link
    fedilink
    27•2 years ago

    YES! Duh-uh.

lemmy.ml [email protected]

[email protected]
Create a post
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: [email protected]

Anything about the lemmy.ml instance and its moderation.

For discussion about the Lemmy software project, go to [email protected].

  • 1 user / day
  • 1 user / week
  • 1 user / month
  • 35 users / 6 months
  • 2 subscribers
  • 87 Posts
  • 853 Comments
  • Modlog
  • mods:
  • Nutomic
  • UI: 0.18.4
  • BE: 0.18.2
  • Modlog
  • Instances
  • Docs
  • Code
  • join-lemmy.org