DuckDuckGo, Bing, Mojeek, and other search engines are not returning full Reddit results any more.
I’m not understanding what stops a search engine from scraping a publicly accessible website. ?
robots.txt, I guess? Yes, you can just ignore it, but you shouldn’t, if you develop a responsible web scraper.
Also, rate limiting. A publicly accessible website doesn’t mean that it will allow scrapers to read millions of pages each week. They can easily identify and block scrapers because of the pattern of their activity. I don’t know if Reddit has rate-limiting, but I wouldn’t be surprised if they implement one.
Doesn’t seem legal that a robots.txt could pick and choose who scrapes. Seems like legally it would have to be all or nothing. Here’s hoping one of the search engines ignores it and makes it a legal case.
You’d probably feel differently if it were your service. Should you be able to control who scrapes your sites or should that be all or nothing?
For the record, I fucking hate what the internet is becoming. I naively believed that even if shit got cordoned off into the walled gardens that are mobile phone apps, the web would remain as open as it was. This is a terrible sign of things to come.
No, I wouldn’t feel differently. In fact letting search engines scrape and point to your content is what leads people to your site. It’s free advertising. If you’re going to let one search engine in, you should let them all in. If you want to be public, be public. Otherwise put up a login firewall and go private.
It’s not just search engines. Lots of people on Mastodon were using robots.txt to block ChatGPT (and any other LLM company they knew of) from scraping their sites/blogs.
I disagree, to a point. I want to be able to control my services to the greatest extent possible, including picking who scrapes me.
On the other hand, orgs as large as Google doing this poses a real threat to how the internet works right now which I hate.
Actually currently it contains this:
User-agent: * Disallow: /Well, that actually is a blanket ban for everyone, so something else must be at play here.
https://merj.com/blog/investigating-reddits-robots-txt-cloaking-strategy
Reddit is serving different file to google
We believe in the open internet, but we do not believe in the misuse of public content.
That’s real rich, coming from Reddit.
thats fucked
Best search meta is “searchterm -reddit +forum” anyway
deleted by creator
paywalled link :(
I got you https://archive.ph/GS2I0
We are indebted to Archive.ph which allows us to jump over paywalls.
yar har
Don’t forget, searx works too.
That’s google
Searx just aggregates
I wonder where searx gets the results from then.
That was sarcasm.
Oh good, I can’t view most reddit threads without an account anymore, so it’ll be nice to see those results go away.
I wasn’t aware of this, when did it start? So far, it has never happened to me not to be able to view reddit threads
They’re so scared from ai scrapers stealing “their” content that they blocked wide ranges of IP address
I can’t see it from my VPN for example
You actually can if you replace “www” with “old.”
Actually you can’t anymore
Yes you can. I literally just did it now to check, and it works fine. I get the “download the app” nonsense, edit the URL, then I can see the content just fine. It works every time.
I use reddit without being logged in a bunch. Worked fine earlier today! I’m on Firefox if that matters.
Whenever google isn’t giving me proper results (so almost every time) I add “Lemmy” to the search and it gives me… Reddit links.
Okay, hear me out, we should make a different service, like a network of computers where people can freely post information on some sort of page that other people can freely access without needing to log in to a million services. Maybe we could also add like little conversation boards to those to allow people to ask and answer questions, too.
Wild, I know, but I think there is some opportunity there.
That would be perfect to train LLMs.
Darn it
The important part is not building it, but convincing the world, alas.
Yeah, not gonna happen. Fact is, Johnny Everyday doesn’t give a shit he’s flooded with ads, tracked on every click and is now forced into exclusivity deals because “somebody has to pay for this stuff and it ain’t me”. Likewise, the people giving the answers don’t care that they literally work for free for some megacorporations. It’s a rotten system and society is to blame.
We are not making a difference here either, not fooling myself into thinking that. I mainly use Lemmy because I can access it with any app I want while sitting on the loo.
They should include reddit in the list of search engines that don’t work well with reddit
removed by mod
Every time I click a Reddit link now it’s just “download the app to verify your age” regardless of what it is
removed by mod
btw i think fatbird still is free and works on mobile (if you need an app)
I feel your pain.
I edit the URL to remove the first part of the URL and replace it with “http://old.reddit.com”. That still seems to work, last I checked, but I fully expect it to be killed any day now.
There’s a firefox extension “old reddit redirect” that’ll do this for you. Been using it for years. But yeah any day now I expect old reddit to be offline.
I don’t, I’m convinced it’s a “if you raise the price of the hotdog, I will kill you” kind of deal (as in, some of the devs still use the old UI for themselves)
I mean, I would’ve said the same about the mobile apps, but here we are.
That’s different, they didn’t remove the apis that enable mobile apps, they just made them unreasonably expensive. But you can theoretically still use them
deleted by creator
i like how the only reason i’d use reddit is to search “site:reddit.com/r/thesimpsons random simpsons quote” to see what the simpsons reddit says about the episode.
Weird, most of the results I get from Google’s search are from Quora (and they fucking suck). Google as a search engine has been going downhill for a while now. Reddit has becomes an increasingly spammy shithole full of corporate and political astroturfing too.
deleted by creator
