@[email protected] to Lemmy [email protected] • 2 years agoChad scrapersh.itjust.worksimagemessage-square99fedilinkarrow-up11.05K
arrow-up11.05KimageChad scrapersh.itjust.works@[email protected] to Lemmy [email protected] • 2 years agomessage-square99fedilink
minus-square@[email protected]linkfedilink75•2 years agoJust a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.
Just a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.
But, but, norobots.txt!
Chad doesn’t care!
uses curl manually instead