Edit: unexpectatly we are back online. Will share how we found a solution later. There might be still some federation issues though.
Alt account of [email protected] here.
Our instance is currently down and I can’t get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can’t work around remotely.
I am still trying a few things, but I am not very optimistic that I can get access.
The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can’t access it physically during the next few weeks and my usual back up that could restart it is not available either.
As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy’s law seems inescapable 😓
I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥
Edit: we might use this “opportunity” to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.
Thanks for letting us know what is going on.
Sorry that it took so long to get the word out. I guess we relied a bit too much on the xmpp server for internal communication.
Even the big operations break in unexpected ways. Facebook accidentally broke the authentication for their door locks when they took the site down a few years back and had to fly people to a datacenter and break down a door to get to servers and restart things.
We managed to resolve the issue and I have many ideas how to avoid this in the future 🤓
Thanks for all that you do for us and this community! A little downtime won’t hurt anyone. I would also be in gavour of PieFed migration 💙
I totally agree with everything you said :)
I just created an account in PieFed and it looks very interesting because there is the option to create your own feeds! I had the wrong impression that you can only choose from the existing ones. This is so exciting!!!
That sounds pretty cool. I don’t know much about Piefed — would we still be able to access posts from Lemmy instances?
Yes, they’re 100% cross compatible. As an example, you can access [email protected] from your Lemmy instance seamlessly.
This is making me want to create an alt on slrpnk.net , since you’re so chill and communicative about issues.
Yeah me too, I’d wanted to for a while but never got around to it. Now I probably never will (unless they get the servers back up at some point in the future).
it seems they will, just not too soon
You’re very welcome! [email protected] / @[email protected] is an incredible admin, and up until now has declined donations, funding the entire instance several years now from his own pocket.
This current downtime is the exception that proves the rule. I hope he’s not stressing out too much. We’ll all be ready to jump right back into it when he gets back.
Already thinking about how to make the best of it 🤷 Maybe we can use this opportunity to try and migrate to Piefed? I had this in the back of my mind for some time already and Rimu seems optimistic that it is possible.
Wouldn’t that mean the instance would completely restart? I mean I don’t know much about piefed but logically it seems like the DB structure would be incompatible. Thus “Migrating” to piefed means a complete reboot, all posts, all users, all content just bye bye.
Not necessarily, no. We aim to preserve users, communities and posts/comments. Image uploads might get lost though.
Such an in-place migration will need extensive database operations and likely some support by the Piefed developer (to add support for bcrypt hashed passwords), but we are hopeful to make it happen and maybe this will result in a database migration script other Lemmy instances could also use.
If this turns out to be infeasible, we will stay with Lemmy rather than reset everything.
Oh awesome, I mean I’ve heard of other platforms like sublinks offering database migration but I didn’t think it would be feasible with piefed since unlike sublinks it’s very different than lemmy.
communicative about issues
Aren’t most admins this way?
*glances at your instance name*
Ah…
I miss BBS.
you can still have it with NodeBB!
Are there modem sounds to play air guitar to?
beep beep boop boop boop boop beep… EEEEEEEEEEeeeeeeee awwwwwww ka-dong ka-dong da kshhhhhhhhhhhhhhHHHHHHHHHHHHHHHHHHHHHHHHHH
eerie silence
I have heard of the migration to Piefed, but will it remove the (very cool) aesthetics of the lemmy ? I loved it :'(
I think we will try to come up with a theme for Piefed with similar colors.
yayyy :D
You guys have always been excellent. As others have stated, no need to worry.
I’ll be giving piefed some love in the meantime 👍
people hate the big cloud solutions, but this is the kind of thing their HA infrastructure prevents against… hardware failures
i dont enjoy using (or paying for) aws, but i will never have a firewall or disk failure.
Their HA infrastructure is all built on open source projects. The thing they have that we don’t is teams of SREs on-call 24/7.
Solarpunks can have a little downtime, as a treat
Seriously, I think a big part of solarpunk ethos is combating the notion that everything has to always be available 24/7. Society pays a lot to deliver every convenience like fruit out of season from the other side of the world.
And on the punk side of things, anybody familiar with volunteer community action, cooperative events, whatever, knows that sometimes a person doing shit for the community fails to do the shit they were supposed to do because some other shit happened in their life. It happens. We just move on to the backup shit-doer-person or find a workaround for the lack of shit.
And this is the cool thing about a distributed network like Lemmy. If a giant centralized database like FB goes down, its users can’t do anything. If our favorite Lemmy node goes down, we can log into other nodes until our home base is back up.
Thanks for the heads up, try not to stress too much.
Alt of [email protected] chiming in, thanks for giving us the sitrep, poVoq!
This is an unusually unlucky set of circumstances, but for the Solarpunks who find their way here, let’s try to make the best of it. If you don’t already have an alternative account somewhere, you may want to create one to use until we’re back up and running. lemmy.dbzer0.com, sopuli.xyz, or lemmy.cafe are some solid choices to consider. After that, you may want to favorite poVoq’s post here so you can check back on any news.
Even without our home base, we can still share the solarpunky stuff we find and create around the threadiverse (though as Tofu mentioned, maybe this could be a time to take a breather to be refreshed when we come back 😄).
Besides, if the past is any indication, time will fly by and we’ll be back before you know it!
Well, on the plus side lots of lessons learned and I think I might move at least the xmpp server to an external vps to have a backup communication channel.
Not sure about your area, but a wireguard accessible OOB connection is a great piece of kit to keep handy. I use a cheap 768kbps SIM in an Ethernet connected switch into my personal systems. It’s saved my skin numerous times.
I’m sure this is obvious, especially in hindsight, but just mentioning because the existence of IoT LTE data plans for a minimal fee ( $100/year for me in Midwest US) was NOT obvious to me until 2 years ago.
Yeah I had plans to set up something like that, but always other priorities and in this specific case I could maybe access other internal servers but i would need KVM access to reboot the firewall or some other way to cut physical power. And exfiltrating hundreds of GBs of lemmy database wouldn’t work over such a small pipe either.
This is something that even larger corps struggle with. My old company would buy some other company, lay off a bunch of people, others would quit, and then it was shocked Pikachu faces all around when That One Thing stopped working in an office they turned into a ghost town and, well, no, nobody was going to be there until Monday morning to power cycle etc. True lights out/OOB reachability is WORK. And there’s always going to be a SPOF somewhere that requires hands on-site.
Do what you can, when you can. And thank you for all that y’all have done so far.
For that, I’d highly recommend a power conditioner with outlet controls (and preferably sequencing).
Most are pretty expensive, but you can get something like a surgex squid or a gude expert power control for a few hundred here and there, which will give you great data on power usage as well. You can also grab an older model with serial and/or relay controls, and use a pi or whatever is handy to trigger it. Actually have a setup like that in a few racks for clients (mostly orange pis and a couple of Asus tinker boards).
Hope your trip goes well!
Lol love the new username, fitting! :)
Yarr, thank ye kindly, Lazy :p
It happens, it’ll be fine. Just remember not to stress out too hard about it and it’ll be fine. The solar punk community knows how to survive a server outage in style.
That sucks. Thank you for your time and running the instance. Gonna miss you all like hell while you’re gone. 😞
Ouch. Best of luck getting it back up and running ASAP.
Thank you for letting us know, good luck!
Thanks for the info, a healthy amount of real life awaits then. :)
I knew it was running on solar energy and old hardware so I guessed something like this had happened.
If you need fail-over to awaken a backup system when the primary fails, things can be designed. :)
Found this post after a couple of days of hopeful reloading of the page! Took another couple of days to log in an old alt just to say I’m glad you communicated this, and let us know how we can help to make it more resilient (if at all necessary).
To the kind soul that was helping me with my greenhouse infested ants (in the unlikely event they are reading here - but iirc they were not from slrpnk.net but just passing by) I’m glad we spoke, it seems like they are harmlessly using my panels for hatching, and then moving on! Related, who has a nice gardening/greenhouse community to suggest? :)
