Hey everyone, so as I’m sure everyone is aware Lemmy.World has been experiencing several outages throughout the last few days.
We have been investigating the root cause of these outages but believe that they are related to our current hosting provider (Hetzner) blocking access from ClouldFlare as (we think) they believe that our CDN is a DDoS’er, and is causing these disconnects to our backend server, problematic for sure.
We’ve opened support tickets with our current provider and are awaiting a response. We have no issue with being as transparent as possible with downtime. Anyone that is curious, can feel free to check out https://status.lemmy.world and https://dash.lemmy.world for up to the minute outage information. We are also looking into other fediverse friendly methods of posting status and outage updates
In the meantime, we are evaluating alternative hosting options and solutions to provide a high level of reliability to you, our users. Really, we want to say thanks to everyone for soldiering through all our technical growing pains.
Cheers
- LW Infra Team
removed by mod
Thanks for the transparency. I think you lost some trust with the Discord debacle, but this makes me feel a lot better about things.
I don’t blame you for this, but the uptime records are incomplete at best. I’ve experienced the site being down (and confirmed with Down for Everyone or Just Me), yet status.lemmy.world showed all systems operational. As I’m writing this, status.lemmy.world is missing most data up to yesterday and dash.lemmy.world shows 16 days uptime.
I have lots of respect to you for even having these. I also remember status.lemmy.world work mostly fine some time ago. But as of right now, both uptime monitors fail to serve their purpose.
You need to hover over the status bar to see if there is any down time for that day. We can enable it to log incidents every time there is a burp, but we are still tuning alerts as we only have it create a incident when we ACK it in PagerDuty. You can always check the dashboard for up to the minute stats, as well as https://lemmy-status.org/endpoints/_lemmy-world We’ll add this info to make things clearer <3
EDIT: Added more info to our status page, thanks for the feedback Machefi!
EDIT2: Also the missing data is due to us removing and adding more specific monitors for the different infra services.
Excuse me stop being so cool, you’re raising the bar too high for everyone else thank you
deleted by creator
Maybe its just the times I’m accessing but its seem better this week to me compared to the last few ones.
Thanks for the updates and transparency.
On your Cloudflare account, if there was a change in the CNAME/A record being proxied vs. DNS only, that could cause an issue, as Cloudflare would then strip headers off the request that your Apache/Nginx would be looking for.
If you enabled HTTP DDoS protection in your Security -> WAF tab (I think that’s where it is) that could do this too. Might be worth disabling.
Also check for any headers your HTTP load balancer might be expecting, that Cloudflare could be stripping.
Might be worth tailing the webserver logs to see what happens to requests coming in from Cloudflare.
Is there anything we can do to help? Donations? Tech volunteers? Visit hosting company with a baseball bat?
I imagine spreading out across instances, and having accounts in instances beyond the biggest one helps, as it reduces traffic and strain on lemmy.world and its servers.
lemmy.world still deserves support, like with donations, though helping reduce the strain is something anyone can do, albeit small, though free of charge.
For both recurring and one-time donations, follow this link: https://opencollective.com/mastodonworld
For recurring donations only, visit our Patreon page from here: https://www.patreon.com/mastodonworld
The dashboard requires a login to view. At least from here it does. Is there a way to view without making an account?
I was able to access it without a login just now.
Seems to work now. Maybe it was a setting on their side?
As always, the transparency is appreciated. Some growing pains are certainly to be expected
Transparency is key
Key is metal
Transparent Anuminium !!!
Xenonite?
So, by transitivity, transparency is metal?
We had an aquarium with transparent fishes, where you could see all their insides, that was pretty metal
So yeah i think we’re onto something here
Metal is conductive
Conductive is electric
Electric, it is. See it, you cannot.
—Yoda
Electric is magnetic
Thanks for sharing the dash link, and for your work!
Could look into Dacentec if you need more cheap servers. I use them for my stuff. YMMV since you’re getting a hell of a lot more traffic than I am but they haven’t blocked Cloudflare on mine yet so that’s a plus. :)
You guys rule. Thanks for all the hard work.
Thanks for the update!
Everyone talking about the downtime including lemme and me just enjoying lemmy like never before. I’ve experienced no downtime so far
The beauty of a federated app is that I do not have to rely on lw or lemme. Use another instance and go on with my day 😃
deleted by creator
Down for you is not the same as down. If it’s up when you aren’t using it, it’s still up
