Hi Beeple!
Here’s a vague version of events :
-
11PM EST: Lemmy.world got hacked
-
12:20AM EST: Blahaj.zone got hacked
-
12:25AM EST: I shut down the server
-
12:30AM EST: I make announcements to tell people about this
-
12:45AM EST: I have an idea of what the problem is but there is no fix
-
2:20AM EST: I go to sleep
-
8:50AM EST: The server is booted back up, steps are applied to mitigate issues (Rotating JWTs, Clearing DB of the source of vulnerability, deleting custom emoji), UI is updated with the fix, CSP and other security options are applied
-
11:40AM EST: We start testing things to make sure are working And well, now here we are.
If you have issues logging in or using an app:
-
Log out if you somehow are still logged in
-
Clear all cache, site data, etc.
-
Hard refresh Beehaw using CTRL+F5
-
Log back in.
If you still have issues, write to us at [email protected]
To be clear : We have not been hacked as far as we know, we were completely unaffected. This was done preemptively.
Oh yeah, in case, you haven’t, this is a good opportunity and reminder to follow us on Mastodon as the communication line was still up despite Beehaw being down : https://hachyderm.io/@beehaw
You must log in or register to comment.
Welcome Back 🤗
12:30AM EST: I make announcements to tell people about this
I think it’d be beneficial to have more backup lines of communication for announcements than just Mastodon.
We have Discord and Matrix channels as well. Do you have anything to suggest?
Just something Google-friendly.
Can you be more precise? What exactly do you recommend? I don’t know what would be more “Google-friendly”
Maybe the front page of the domain could be news and info with the actual forums down a level? Not sure if that works with the software.
That’s not supported by Lemmy unfortunately… Most we could have is a status.beehaw.org, really.
A status website would honestly be excellent.
There you go, courtesy of @[email protected]
Nah.
I’ll be blunt and say that unless you were already in-the-know, Beehaw pretty much ceased to exist when the server was shut down. Not the best result amidst a hacking scare.
Much preferable to the announcement of Beehaw was hacked and lost your user credentials <or more>. Security trumps convenience.
Having an entirely separate website, blog, or social media account for announcements that’s accessible via a Google search wouldn’t factor into how secure Beehaw is.
Something like status-page is always nice. I haven’t used it but it looks like https://cachethq.io/ could be a decent fit as well.
There you go, courtesy of @[email protected]
Heck yeah! Thanks for getting this up
Thank you for shutting down rather than “wait and see”! It was the right choice.
Thank you for all you do, from what I was hearing I was in no way expecting you to have the site back up within 12 hours. Many kudos.
Good job on making the right call and preemptively shutting the server down. Thanks for being alert!
Far more memorable than all the times the service was unavailable was all the times your data was breached. I’ll always prefer the service being down to having it up, and vulnerable
Was the hack related to the lasted patch update?
No it was related to custom emojis.
I’m not sure what you’re asking.
I don’t believe it was related to anything specifically added or removed in a recent patch, I think it was an existing exploit that just hadn’t been noticed.
The shutdown is a good call given the circumstances.
An idea of less-radical preventive action is placing the instance in read-only mode, either as a Lemmy feature, or through reverse proxy settings (eg reply 503 for any POST/PUT/DELETE request). But that’d require some development and/or preparation.
Doing that on the reserve proxy side would block any user-submitted content and more (logins, searches, …). This would hopefully be efficient at blocking many attack vectors, while still keeping the instance partially online, even if that’s a degraded mode.
Would read-only mode help with XSS exploits though, like this particular one? Since the “damage was already done” by the time anybody noticed, wouldn’t putting the site in read-only mode still have kept serving up the XSS payload? It’d stop “infected” people from making any state mutations on Lemmy, but eg. data exliftration would still happen
Note that if this were a Lemmy feature, if we had been infected, an admin could’ve gotten hacked and as a result, disabled that feature. I’m not really sure what can be done to make Beehaw foolproof. That said, the UI has since been hardened by CSP headers so this type of attack should no longer be possible.
deleted by creator
deleted by creator
Great job keeping the site safe guys!
Nice to see it back up again! It being offline was surprisingly palpable. Missed it!
I’m guessing it’s probably not the last big thing that’s going to hit Lemmy instances in the future, everything still being in early development and all. Only things we can do is keep an eye out, have vigilant admins and plenty of backups!
And patient users but we seem to have that. :)
morning thought: I’ve definitely joined the right instance. (also the start from the assumption of good faith guidelines linked to in Gaywallet’s recent post)
Your work is greatly appreciated! Also happy to know that you got some sleep, very important for the process ☺
yayyyyyy
Good work.
Have a non custom beer 🍻
Maybe post to https://hachyderm.io/@beehaw to spread the word outside of Beehaw.org
