Tea to [email protected]English • 4 months agoCloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.blog.cloudflare.comexternal-linkmessage-square163fedilinkarrow-up1919cross-posted to: [email protected][email protected]
arrow-up1919external-linkCloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.blog.cloudflare.comTea to [email protected]English • 4 months agomessage-square163fedilinkcross-posted to: [email protected][email protected]
minus-square@[email protected]linkfedilinkEnglish11•4 months ago while allowing legitimate users and verified crawlers to browse normally. What is a “verified crawler” though? What I worry about is, is it only big companies like Google that are allowed to have them now?
minus-square@[email protected]linkfedilinkEnglish18•4 months agoI assume a crawler which adheres to robots.txt
minus-square@[email protected]linkfedilinkEnglish4•4 months agoI would love to think so. But the word “verified” suggests more.
minus-square@[email protected]linkfedilinkEnglish2•4 months agoIP verification is a not uncommon method for commercial crawlers
minus-square@[email protected]linkfedilinkEnglish1•4 months agoCloudflare isn’t the best at blocking things. As long as your crawler isn’t horribly misconfigured you shouldn’t have much issues.
What is a “verified crawler” though? What I worry about is, is it only big companies like Google that are allowed to have them now?
I assume a crawler which adheres to robots.txt
I would love to think so. But the word “verified” suggests more.
IP verification is a not uncommon method for commercial crawlers
Cloudflare isn’t the best at blocking things. As long as your crawler isn’t horribly misconfigured you shouldn’t have much issues.