Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.
Most of these ‘attacks’ are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.
For the other attacks, we are using them to investigate and implement measures like rate limiting etc.
Also when will CloudFlare drop lemmy as a ‘Nazi’ site?
I don’t understand why people want to take down websites. Especially sites like Lemmy, which isn’t exactly sticking it to anyone because no one owns it!
Are they just Reddit groupies?
deleted by creator
Or paid for by Reddit…
For most hackers or wanna-bes (often called Script Kiddies, that is, people (generally young, even children thus the “Kiddies”) who are not technologically inclined enough to be real hackers and see a tutorial online on how to run pre-written scripts that repeatedly perform various functions), the answer to “Why do you do it?” is often:
-
“Because I was bored.”
-
“Because I can.”
Very rarely are other reasons given.
The ones seen on
masterhackerreddit.More like “I get zero action, so I take my anger out on other people”
-
Some people enjoy causing suffering to others. On the internet they are termed trolls. Irl people usually just call them assholes. Most people have encountered them before.
I think they are far more common and likely than anyone giving two shits about reddit.
Some people just want to watch the world burn.
You don’t think just being bored is enough reason for some?
If I’m bored I find something productive and/or fun to do.
Launching a DDoS attack is neither.
You, sure. It’s not difficult to imagine a teen who’s not you
Nah, it’s not the 00s anymore. Hacker gangs are a real thing today.
I’m not actually in the security field so take this with a grain of salt. But I believe that these attacks play a similar role to random attacks in low level gangs. It proves that your criminal group has power and the ability to deface a website.
So if you publish that Lemmy.world will go down next week because your hackers are on it… It’s advertising. Its just business. It proves that your hackers have an ability and that you are up for sale.
Cyberspace. A consensual hallucination experienced daily by billions of legitimate operators, in every nation, by children being taught mathematical concepts… A graphic representation of data abstracted from banks of every computer in the human system. Unthinkable complexity. Lines of light ranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding…
I was using voip.ms last year when they were DDoS’d for over a week, by a group demanding payment via anonymous crypto. The DDoS ended when they switched to CloudFlare (which was probably pretty difficult because they’re a SIP provider.)
Almost any website with a small number of servers is vulnerable to this attack, which happens to be great business for CloudFlare. I wonder which companies are most effectively competing with CloudFlare?
There are others, but I think the craziest thing about Cloudflare is its basic level of protection is free. Free, unmetered, DDOS protection. It’s so popular because so many hobbyists use it for free, and are familiar with it. Then they convince their workplaces to adopt it when the need arises because they are already familiar with it.
They make money by selling support to companies, and selling access to some more advanced features (that often have a free tier as well). It’s honestly so impressive, it made me wonder how much they actually make because it seems unnecessary for most to pay at all. Turns out they cleared almost a billion dollars in revenue in 2022.
They’re just trolls. Lemmy is popular enough that it’s fun target for them, but still small and infantile enough that you don’t have to be hackerman to ddos it. Reddit, twitter, etc… would be constantly getting ddos’d just for the lulz by people if they didn’t have the infrastructure to make it a challenge.
With my tinfoil hat on, I’d say one concern is that Cloudfare is basically a monopoly and nothing is stopping them from DDoSing sites to force them to use their product.
While it’s good to be suspicious, I don’t think we can call CloudFlare a monopoly quite yet.
Akamai is a big, giant competitor. You also have the big cloud providers like AWS that have their own CDN systems, like CloudFront. (I don’t recall GCP’s or Azure’s product names.) Then you have specialized CDNs like Google’s AMP system.
Now, is it possible that there could be a horizontal trust between these companies? Certainly. There’s few enough players for that to happen, but so far, I haven’t seen signs of it happening.
deleted by creator
Upvoting because this has to be satire
You have more faith in people than I do…
It’s coming from someone over on Kbin. Wonder if that’s the motivation.
Delete your account and go back to reddit
Genius
Thank you for the amazing job, as always! Cloudflare is a solid solution :)
Sure but maybe something less centralized/proprietary would be preferable
What a
Agreed
This isn’t a helpful reply. There’s no reason to just call someone a name without even explaining why you think what he said is moronic.
There are a couple elements that a DDOS mitigation system needs to have.
It needs to be able to absorb the raw network traffic of the attack. A purely volumetric attack seeks to just overload the network pipes that lead to the servers. This can be with junk packets that don’t even make sense to an OS kernel, but have a valid destination IP address so they get through the routers. If the DDOS mitigation system acts as a filter in front of the servers, it has to not get overloaded in the same way the routers do.
It needs to allow good traffic through to the servers. If the attack causes the pipes to just shut down and reject all traffic, then the attack has succeeded. So the mitigation system has to distinguish attack traffic from good traffic, and keep the pipes open enough to let the good traffic through.
For attacks trying to do expensive stuff on the database, or create spam posts, one useful reflex the system can have is to notice when an endpoint is doing those attacks, and then block it at the network layer.
That is not necessarily easy, and it requires control of the network ingress, which arbitrary hosting providers may not be able to provide.
Thank you for the clear explanation. It seems a lot of folks here don’t understand the tech, but this explains things clearly and accurately
Is “decentralised” the new “blockchain”?
Well, no. Unlike the blockchain, decentralized platforms aren’t snake oil.
Why are the Lemmy devs asking for snake oil on their Donate page then?
Sitting comfy in a country where the financial system works for you elites is the real snake oil.
…what are you even talking about? Your sentence makes zero sense.
100% of the crypto hate I see is from citizens of neocolonial states. You lord your control of the financial system over us and when something threatens it, it’s always delegitimised for any number of reasons.
Take your pick: scam, destroying the environment, eroding state power etc.
A decentralised system/society will need a value layer to transact. You think Visa should be in control of that?
Just because you don’t like it, doesn’t make it snake oil. I hope you never find yourself at the mercy of a government that persecutes you and imposes capital control so you can’t even run away with your money. If crypto existed when my people were literally being genocided, my parents would not have to end up in a new country with nothing to their name.
Just because you’re smart at writing code doesn’t mean you’re smart at other things :) Or more likely, maybe they’re ideology-driven rather than by practicality.
Lemmy is an unusual but fortunate example of where ideology and practicality line up.
If you can find an entire nation state that runs on crypto currency with a functional, stable economy, I’ll eat my words.
Why would I want a nation state to run on anything? The end of the nation state is the communist utopia!
I am for whatever erodes the illegitimate violence exerted by nation states to safeguard their parasitic domain. If it’s crypto, it’s crypto. If people not eating apples brought about their end, I would be out there burning orchards.
This explains nothing. It’s literally saying “one thing is bad, the other thing isn’t”. Try to explain why instead, if you do happen to have an explanation.
One of the things that makes Lemmy unique is the underlying decentralized infrastructure. I think it’s just a request to keep that mantra.
Blockchain can bring trust and thus monetisation to a decentralised network. A good example is the Tor network, which is based on voluntariness, and dVPNs, which can have the same network architecture, but where the nodes are paid for their services.
Stupid
this is a meme right
It is obviously not. Why would it be?
Looks like a meme to me.
Thanks to the fediverse we were all able to read and search old posts on other instances and interact freely with communities on other instances. Pretty damn great i think.
@jimmy90 @PropaGandalf And I was able to finally open up and stop being a lurker
I meant the CDN. The Fediverse per see is great!
Such as?
Nothing. DDoS mitigation is inherently an ISP or someone like cloudflare. You will not have success against anybody who knows what they are doing without their help.
This is bullshit. Just take this as an example. I found it with one quick search and there are plenty more. Perhaps we should broaden our horizons a little rather than entrusting everything to some corpos.
My dude, I think you’re not super familiar with these technologies.
The most basic form of a content delivery network is a set of globally distributed servers that replicate content from a source of truth and a network to direct traffic to the closest server with a valid replica. So the cost here is servers.
With Lemmy, this problem is solved by eliminating the need for individuals to own many servers and a lack of need for trust between servers. The effort and cost is distributed among individual humans, making it manageable.
Now, if you’re familiar with blockchain, you probably perked up when you heard “lack of need for trust.” That’s what the blockchain was built for! Perfect fit, right? Ehh, not so much.
There’s two problems: acting as a proxy for content requires trust, and some single service needs to direct clients to the right local server. If I can arbitrarily join some network of serving content, I can always tell other servers in the network that I’m serving what they ask… and then serve ads. There’s no (reasonable and fast) way for the network to verify that I’m serving the correct content to every client. There’s no way to avoid the need for trust. Additionally, DNS, which directs you from mysite.com to 120.1.2.1, isn’t intelligent. It can’t direct clients to a geographically (or route-efficient, fucking ISPs) local IP. The best it can do is pick a random one from the pool. So when you go to lemmy.world, DNS can’t pick the correct server for you. So some set of servers needs to do the logic to select which local server to actually get content from. Those servers need to be central for the whole content delivery network.
This company you linked is just another company using “blockchain” to get investment money. If you read through their page to get a cursory understanding of how things work, an easy question comes up: what is the purpose of
mediatokens? Sure, maybe you can buy CDN time with it, but when you pay that token to someone providing compute… what do they do with that token? It’s worthless, just like crypto currency. Fucking scams. All that said, blockchain is a super, super interesting technology. There’s just very, very few suitable applications of it.I’ve worked in IT for about 12 years now. Everything from infrastructure monitoring to data analysis to data engineering to DevOps to backend engineering to product management. I’ve worked with systems serving tens of users and tens of millions of users. Happy to answer any questions. I love this shit.
If someone could figure out a trustless, decentralized way to implement a CDN, I’d eat that up in a second, but with my current understanding of the internet and available technologies, I don’t see a way it can work. At least, not with making every web page take >3s to load, which would absolutely kill websites.
Great breakdown
I could a agree with the first part and it does not contradict with the idea of a distributed network for content saving. Think about it this way. Instead of one big local server farm you have multiple small local servers which together form a global network. Now we come to the blockchain. As you pointed out you get these tokens for the CDN time the storage or more generally the server operation costs. Of course the blockchain these tokens are hosted on (Solana) do have to be trustworthy (which in this case they may not be. I don’t like solana that much either). But does that mean that this could not be achieved? It seems logical to me that with a distributed storage and computing network something like this could be achieved very efficiently and cheaply. Heck I’m using a decentralized VPN right now that works with the same principles I mentioned. Or take the Helium network for example? Don’t you see the potential there? Like with all technology these things have to mature but with my understanding they are pretty much doable.
They exist sure, but as others have said, there’s still a lot of links in the chain to smooth out. And for a mission critical application like this you’ll always want to chose the most stable offering.
Sure, its doable, but if we return to OP issue, is it available and usable now? If there’s a service provider I’d trust to do this, it’s CF, they have a good, solid product and they have not given a reason to doubt their business ethics yet.
Two things:
Isn’t there always trust issues though? Also, could SSL passthrough help in that?
Instead of CDN for protection, couldn’t a local WAF help solve this too?
They lost me at building a CDN on top of a blockchain. Why?
It’s rather the other way round. Complement a distributed CDN with a blockchain.
Stupid
Well for now we’ll have to stick around with cloudflare. I’d just would like to see something managed by a decentralized network. I don’t know if it exists, it’s more of a sentiment or a general idea.
If you don’t know what a content delivery network is, here : https://www.cloudflare.com/learning/cdn/what-is-a-cdn/
A CND is very costly to run in an effective way. And because it is an intermediary server between the user and content server, the market is already pretty full. So competing with the CDN giants is practically impossible in a decentralised manner.
Because of what a CDN does (cache website elements closer to the user, protect the website against ddos…), it cannot be a cheap weak server, or it’s the one which will get overwhelmed by the ddos, or even the users.
Another limiting factor is that in decentralisation, that means different companies, and so many separate plans to pay, which is just impossible for a company.
If it was decentralized, a company would have to go and pay 100 different companies (which is more expensive, du to the server costs and each companies having their own staff to may (even if it’s just 1 person per company)) just to offer a quick access to the users around the world, which is just impossible.
A CDN isn’t a great comparison to DDOS mitigations. CDN spreads the load amongst multiple locations that are distinct entities. Any one can be down and the rest functions fine. They generally exist on separate domains and are not inherently codependent.
DDOS requires an inline solution. A layer acting as a man in the middle to deflect or absorb the traffic destined to Lemmy.world, for example. That’s not something that can be readily be decentralized while there’s only one ingress to Lemmy.world.
I know well what a CDN is and that’s why I don’t understand why you build a DISTRIBUTED content delivery network on a single corporation. I mean, the whole architecture is based on decentralised servers that precache the content and share the service load. Why not create an independent network that provides this bandwidth and where each node is rewarded according to its contribution? I know blockchain is a term that pisses a lot of people off, but it’s basically the best way to incorporate trust and monetisation into a decentralised system.
removed by mod
You’re being down voted, but a p2p cdn is something that sort of already exists. IPFS is probably the most mature. As far as I know, it’d only work for static content though. It’s also an entirely different protocol so you’d have to use some sort of local gateway or plugin to make use of it.
I have several vms and dedicated servers that I sort of use as a DIY cdn. No where near as spread out or capable as something like cloudflare, but its also not incredibly expensive to do on a small low performance scale. DDOS mitigation is another story though, generally that is best handled by large networks that can soak up the throughput.
Yeah it’s also more of a potential that I wanted to point out. Over the years that I have been involved with blockchain projects, I have developed a feeling for where blockchains and decentralised networks are suitable and where they are not. In this case, however, it seems very feasible to me. In the end, CDNs are nothing more than a server network that caches the data locally and distributes the bandwidth. This is exactly what an independent network could do with the advantage of the blockchain to remunerate the contributions of the individual node operators. But I see that the notion of blockchain triggers a great aversion in most people.
I don’t have half the knowledge in IT you have, but i totally agree we should find a solution to seperate from mastadons who owns the whole network.
It’s very similar to how we shouldn’t give big corpos like GAFAM willingly our data/privacy or our foodchain shouldn’t be controled by a few corpos who serve poison… (the list goes on).
Most people just don’t care, they have nothing to hide or they won’t die if they eat one cheesburger from McDonald’s a week…
But in the case of lemmy I think (personal opinion) It’s because it’s easier, simpler, faster to setup right now. I’m sure if they had a better solution to not depend on cloudflare they would chose the other solution.
I mean your idea seems great, but how long would it take to put it inplace? How many highly qulified people are needed to make it work? How much will it cost…
I hope that in the long run, lemmy instances are going to find a better solution 😀
I’m only talking about the long run. For now cloudflare is a solid service. I’d love to see some experental approaches tho maybe from other smaller instances.
Build your own then.
Wanna know the beauty of Lemmy? If you don’t like how instances are ran you can create your own🙂
sure, this is just a wish of mine and I’m totally happy with our mods here.
This is a polite way of saying get lost.
It’s an interesting question but the knee jerk reaction towards decentralization isn’t always a silver bullet. Bitcoin always screamed that concept while ignoring the role of clearinghouses. Decentralization can actually compound the issue. Not to dispel the solution but good to keep these things in mind.
It isn’t a silver bullet but in this case it is particularly suitable. I mean, the architecture of CDN is decentralised, but all these servers are controlled by ONE company. So why not leave the whole task to an independent network?
I think the biggest problem with such services is that they require lots of money to run which means that any well-meaning effort will eventually end up becoming a commercial service.
…and that’s where the blockchain comes in. This means that the individual contributions of the node operators can be directly recorded and compensated adequately.
…and that’s where the blockchain comes in.
Sure.
Tell me a good argument why not? How would you reward those people that contribute to said netowork?
That’s easier said than done, DDoS mitigation requires a large amount of servers that are only really useful to persist an active DDoS attack. It’s why everyone uses Cloudflare, because of the amount of customers they serve there’s pretty much always an active attack to fend off. Decentralization wouldn’t work great for it because you would have to trust every decentralized node not to perform man in the middle attacks. But if you know of any such solution I’d love to hear it.
Yeah I see the issue but on the other side you would get a more robust network which could also be incentivised by some sort of underlying blockchain technology. The man in the middle attack could also be mitigated on a technical level.
I block anyone who mentions a blockchain.
Blockchains are bullshit.
Wait whaat?
Blockchains are bullshit.
Chances are that you’re being sarcastic, but in the event you’re not or if others want to learn…
Interesting tech. Almost zero practically useful applications.
Blockchains are effectively reproducible, verifiable ledger systems. But if the ledger grows infinitely, your storage and compute costs also grow infinitely. I’ve heard this has been solved, but I haven’t seen an implementation yet. (If anyone knows of one, please share!)
Another issue is the proofing system. Bitcoin uses proof of work, which means you need to do more computational work to produce new blocks on the chain. If the computational work grows, that means you need more and more powerful computers. This means increased cost which means centralization as participants with less money to pay for compute get pushed out. Alternatively, there’s proof of stake, where having some amount of a token or some similar value/stake allows you to write new blocks. This does reduce the computation cost but still causes those with lots of tokens/stake to get even more tokens/stake, which in turn allows them to spend more for new blocks… which creates a loop towards centralization.
So basically, the technology that preaches decentralization naturally centralizes in practical use over time.
You can’t mitigate a man in the middle attack on a technical level… Because they are a man in the middle… That’s the point of using DDoS mitigation. Nothing’s stopping them from just sending incoming traffic to a phishing site if a bad actor was in control of it.
Dunno if this guy is just so stupid or is trolling at this point. Using random tech buzzwords that have no relevance to the issue.
You’ve never blockchained your decentralized DDoS backend with a bi-duplex CDN enumerator?
Well I did mitigate an attack before using quantum entanglement calibrated against the cosmological constant to mitigated carbon decay. Does that count? Oh and, blockchain and decentralized. Haha
I myself am not sure who here understands anything about blockchain technology. For you it’s just NFT images and shitcoins that you associate with blockchain, isn’t it? That knowledge is enough for you to understand the whole technology. Read my other comments and ask yourself first if you have a balanced information base.
For sure it’s not you. You sound like an amateur developer no-one would hire.
You are smoking crack. You clearly do not know what you are talking about.
You had me until you mentioned blockchain technology. How would a blockchain system help in that regard, anyway?
A blockchain can complement a decentralised network by introducing trust into such a network, where the individual members cannot be trusted. This makes it possible to accurately document actions and reward or punish them accordingly. If you take such a distributed CDN network as an example, a blockchain could help to directly reward the individual members according to their contributions instead of building everything on voluntariness and goodwill as in the Tor network.
Oh man, you lost me at blockchain.
The goal is to mitigate attacks, it costs a lot of money to purpose build world spanning networks than can absorb large amounts of traffic. P2P type options are not a good fit.
Which viable alternative could work to mitigate ddos?
Out of my head, I think OVH offers such a service (but without free tier).
HAProxy has some really good features a server admin can use locally without sending all of our data to Cloudflare or OVH.
https://www.haproxy.com/blog/application-layer-ddos-attack-protection-with-haproxyThere are many protection modules for most reverse proxies that provide basic (limiting) or sophisticated (captcha, calculation challenge, etc) DDoS protection. HAProxy is just a very powerful and easily extensible proxy.
Sure, but you still have to pay for servers to run the proxy instances on. Any DDoS of appreciable size will knock over the number of instances that lemmy.world could stand up. Interesting thought, though. Maybe CloudFlare or others use HAProxy internally? I’m actually not sure what tech they use
It’s usually very effective unless the amount of connections is too much for one machine anyways. Along with bandwidth shaping and connection throttling it can fend off smaller attacks.
A huge botnet would bring down a single proxy instance in any case, true.
OVH is cheap but their anti-spam/abuse departments are ineffective. Too often they do not action blatant spam reports so in effect OVH is part of the problem with network abuse on the Internet. I’ve had to blackhole many of their netblocks while the people who run mxroute (solid email providers) have written about doing the same.
OVH needs to clean up their act.
You sound fanatical with this statement
Anything we can do as “users” to help, other than donating?
If it’s the same people, they’ll probably get tired of it and move on. But the more we talk about it, the more likely it is that new people want to get in on the “fun”. I’d say to not make memes about the downtime and pretty much act like it doesn’t exist (as users, obviously the admins should take action as necessary to mitigate it and post to be transparent).
Hmm, best would be if those kids find a real hobby so they stop bothering us. On the other hand, it helps us understand Lemmy better and secure it.
That’s true. Free stress testing the system I guess? Still they need to touch grass lol
Thank you❤️❤️
A lot of people don’t like that they’ve basically inserted themselves as a man in the middle to a lot of internet traffic.
Where can we donate toward server costs?
Do you prefer one or the other when it comes to donations?
I prefer OpenCollective.
Cool. I didn’t really want to make a patreon account.
Thank you for your efforts, work and results. Those “attackers” only deserve disgust.
Maybe they don’t deserve as much, pity would be enough.
Yeah, this is just growing pains for any website. Get popular enough for it to be “fun” to target. Then get enough data that it’s “profitable” to target. Etc. And the usual way to deal is to first use an external solution at least until it becomes too expensive due to traffic volume. Then make your own solutions for problems you can solve yourself and pay external companies for the ones you can’t.
Thanks a lot for all the work you folks are doing to keep this instance up.
Damn these script kiddies… I don’t like Cloudflare at all but it does its job well. It may just be my paranoia, but putting a single entity in control of so many websites seems dangerous. I think we have all learned about the intentions of big corporations. But hey, it’s better than being taken down tbf.
What are your reasons for hating cloudflair? Best i can tell they run a good service and their free offerings have been great (1.1.1.1)
We said the same thing about chrome 10 years ago. It’s not the quality of the product, which is excellent. It’s the concentration of control.
Exactly my words. I’d love to see a decentralized network to do the job instead. No single point of failure and people can actually earn a bit of money instead of big corpos enriching themselves.
You should change the public IP of the server if you haven’t already
What happens tomorrow? Change the IP again? And again? It’s not a long term solution.
Cloudflare masks the origin IP address and has DDoS protection. Unless it’s a DoS against the software, yes, it is a long term solution.
They mean after adding a ddos mitigation like cloudflare, you should rotate the origin server IP so the origin server’s IP is no longer publicly known and thus not directly reachable by ddos attackers. The only way to now interact with the application is though Cloudflare’s network. You should only have to do this once as long as the origin IP doesn’t publicly leak.
Another step would be to add firewall rules to only allow inbound traffic from cloudflare IPs: https://www.cloudflare.com/ips/
I recall a certain amount of overhead in IPTables “allow only from” situations but I’m not sure whether it’s enough to make a DDOS any kind of viable on a server in this configuration.
Do you happen to know how effective the strategy is?
If your origin servers IP is never revealed then all traffic goes through cloudflare regardless. Firewall restricting the IPs is just good practice since cloudflare is the only IP that is supposed to talk to that server anyway, but it’s not a requirement.
I can see some overhead if you’re maintaining a large blacklist, but I don’t see it happening with a small whitelist and default inbound DROP
Oh absolutely, I agree with the best practice! I just didn’t know the real world efficacy of dropping packets near the NIC to mitigate DDOS load. There is certainly a performance limit but where that limit exists has been nebulous for me.
Cloudflare isn’t bad per se, but having huge amounts of the public internet behind a centralized provider is bad for the flexibility and resiliency of the internet as a whole.
deleted by creator
No, but Cloudflare is providing services to those kids too.
Most of the services that provide DDoS attacks as a service use Cloudflare themself and Cloudflare is absolutely okay with it.Lol. Just for shits and giggles I want to entertain this for a second.
You’d probably want to pay hackers in a country that isn’t friendly with the US to do this. Russia, North Korea, China, Iran.
Three of those countries are heavily sanctioned right now. I wouldn’t want sketchy money flowing to Russia at the moment even if it didn’t technically fall under sanctions since money flow is being scrutinized. Same with NK and Iran.
So that would leave China. I think you could get away with it there pretty easily.
And lo and behold….
https://techcrunch.com/2020/04/28/cloudflare-partners-with-jd-to-expand-its-network-in-china/amp/
:tinfoilhat:
With that said though. Getting that info leaked out would be extremely damaging and totally not worth the risk.
It’s highly unlikely. Cloudflare is (I think) the biggest CDN provider and one of the biggest domain registrars. Whatever lemmy.world is paying them it’s inconsequential to their books. For a sense of scale, they own the IP address 1.1.1.1. (as an aside, 1.1.1.1 is a DNS host, but unlike the other popular ones it has a webpage so it’s very convenient for checking if your internet is down or if you’re having DNS issues)
Basically, the cost reward is way out of whack for them to consider ddosing such a small site.
Maybe CF and spez are paying the script kids… damn…
