@[email protected] to [email protected]English • 9 months agoReddit is now blocking major search engines and AI bots — except the ones that paywww.theverge.comexternal-linkmessage-square72fedilinkarrow-up1378cross-posted to: [email protected][email protected][email protected][email protected]
arrow-up1378external-linkReddit is now blocking major search engines and AI bots — except the ones that paywww.theverge.com@[email protected] to [email protected]English • 9 months agomessage-square72fedilinkcross-posted to: [email protected][email protected][email protected][email protected]
minus-square@[email protected]linkfedilinkEnglish9•9 months agoSome Lemmy instances disallow indexing in robots.txt, however indexers can choose to ignore that and actually blocking them takes a lot more effort.
minus-square@[email protected]linkfedilink8•9 months agoSome places on a “budget” like Ao3 just rate limit hard. I don’t like that solution at all though.
Some Lemmy instances disallow indexing in robots.txt, however indexers can choose to ignore that and actually blocking them takes a lot more effort.
Some places on a “budget” like Ao3 just rate limit hard.
I don’t like that solution at all though.