Pricefield | Lemmy
  • Communities
  • Create Post
  • Create Community
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
@[email protected] to lemmy.ml [email protected] • 2 years ago

Should lemmy.ml block chatgpt scraping in robots.txt?

message-square
15
fedilink
41
message-square

Should lemmy.ml block chatgpt scraping in robots.txt?

@[email protected] to lemmy.ml [email protected] • 2 years ago
message-square
15
fedilink

Some context about this here: https://arstechnica.com/information-technology/2023/08/openai-details-how-to-keep-chatgpt-from-gobbling-up-website-data/

the robots.txt would be updated with this entry

User-agent: GPTBot
Disallow: /

Obviously this is meaningless against non-openai scrapers or anyone who just doesn’t give a shit.

  • @[email protected]
    link
    fedilink
    2•2 years ago

    If they’ll pay us when they scrape our content, sure.

    • @[email protected]
      link
      fedilink
      1•2 years ago

      … Is that like a non-argument? How do you suppose they would pay sites, let alone site users to scrape their content?

      • @[email protected]
        link
        fedilink
        1•2 years ago

        Yes that’s the point

lemmy.ml [email protected]

[email protected]
Create a post
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: [email protected]

Anything about the lemmy.ml instance and its moderation.

For discussion about the Lemmy software project, go to [email protected].

  • 1 user / day
  • 1 user / week
  • 1 user / month
  • 35 users / 6 months
  • 2 subscribers
  • 87 Posts
  • 853 Comments
  • Modlog
  • mods:
  • Nutomic
  • UI: 0.18.4
  • BE: 0.18.2
  • Modlog
  • Instances
  • Docs
  • Code
  • join-lemmy.org