While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.
As I am told, this was the issue:
- There is an vulnerability which was exploited
- Several people had their JWT cookies leaked, including at least one admin
- Attackers started changing site settings and posting fake announcements etc
Our mitigations:
- We removed the vulnerability
- Deleted all comments and private messages that contained the exploit
- Rotated JWT secret which invalidated all existing cookies
The vulnerability will be fixed by the Lemmy devs.
Details of the vulnerability are here
Many thanks for all that helped, and sorry for any inconvenience caused!
Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).
For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.
You guys really have my highest respect for spending so much time to keep this running, despite all the recent trouble and now even an attack.
Thank you very much <3 You guys are awesome and I really appreciate how publicly you deal with this.
Good job. I don’t understand very much of that, so that makes me all the more grateful. Thank you.
A lot of images seems to be gone from posts in /c/pics is this related to the hack or the cleanup after?
I heard there was some sort of database rollback to an uncompromised snapshot.
Is there a rough time range when it happened? and any news about other big instances like lemmy.ml? Are those safe? Currently they are not on the same version as lemmy.world.
2:11 UTC is my first record of the event taking place, but keep in mind the attacker could have injected code long before without noticeable impacts. There’s no way to be completely certain they didn’t steal tokens and access accounts before they made themselves known.
With the JWT secret rotation, shouldn’t everyone be forced to re-login? I’m posting with my existing session without any changes.
FWIW, I had to re-log
better to re-login i guess
deleted by creator
I lost some of my post history. Is there a data issue that’s come from this? Why are my comments gone?
If it is only recent post history, maybe it was purged along with many malicious comments/posts
I know but this was over a week of comments lost.
I had to clear jerboa’s app data so I could log in again. so strange
So what happened:
- Someone posted a post.
- The post contained some instruction to display custom emoji.
- So far so good.
- There is a bug in JavaScript (TypeScript) that runs on client’s machine (arbitrary code execution?).
- The attacker leveraged the bug to grab victim’s JWT (cookie) when the victim visited the page with that post.
- The attacker used the grabbed JWTs to log-in as victim (some of them were admins) and do bad stuff on the server.
Am I right?
I’m old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:
- User provided content (post using custom emojis) caused havoc when processing (doesn’t matter if on server or on client). This is lack of sanitization of user-provided-data.
- JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
- How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
- The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.
Am I right? Correct me if I’m wrong.
Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naïve? Maybe.
Marek.
May I ask was is the JWS coockinand if it is automatically changed or if we have to change it in a way?
What are the risks for people who use Jerboa for Lemmy? I logged put and back in and there doesn’t seem to be any issues, so are the app users excluded from this?
So, do we change passwords, esp those who logged on during the attack? (I created this acct right before the attack happened tho.)
Should apps have logged themselves out?
So thats why MalwareBytes gave me this message yesterday.
Is it possible cookies for other websites were scraped? I was logged in to .world at the time; I have logged out of all accounts, and reset passwords as a precaution, but want to know if I should be on the lookout from this.
No, in general it’s not possible because the code in a page cannot access cookies that are bound to other domains. It is only possible if the “other” site misconfigured its own cookies (which is really not likely for stuff you would care about).
