Status update July 4th
Just wanted to let you know where we are with Lemmy.world.
Issues
As you might have noticed, things still won’t work as desired… we see several issues:
Performance
- Loading is mostly OK, but sometimes things take forever
- We (and you) see many 502 errors, resulting in empty pages etc.
- System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)
Bugs
- Replying to a DM doesn’t seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
- 2FA seems to be a problem for many people. It doesn’t always work as expected.
Troubleshooting
We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @[email protected] is also helping with current issues.
So, all is not yet running smoothly as we hoped, but with all this help we’ll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!
deleted by creator
Same here
To all the folks that are worried, don’t be. Let me tell you, Mastodon was a wreck when Musk took over Twitter and that all got sorted out within a week or so. The mods and sysadmin are obviously working hard to get things up and running, but growing pains are growing pains. To paraphrase an old adage: “Facebook wasn’t built in a day.” In the beginning, Zuck and Co literally limited signups to only people with college email accounts and only added universities a few domains at a time… scaling is very difficult, but it’s not impossible. The way things are going, Lemmy is going to thrive!
It’s not perfect but becoming better everyday.
Really appreciate all the time and effort you all put in especially while Lemmy is growing so fast. Couldn’t happen without you!
Thank you to you and everyone behind the scenes working hard to make Lemmy a great place. I am turkey enjoying the experience even with the bugs. Brings back a ton of nostalgia with modern mixed in.
I’m just excited to be back in the Wild West again – all of the big players had bumps, at least this one is working to fix them.
I still remember early reddit days of 502 it went through, 504 try once more.
Early? Haha, reddit still shits the bed regularly
Yee-haw, mother fuckers.
And giving updates!
I’d rather have to deal with hiccups and bumps along the way, because the community only grows more each time.
It’s a ton better this afternoon! Thank you!!
I’ve noticed that too!
It’s a ton better this afternoon! Thank you!!
More props for you and the other admins as you ride the waves. It’s for a great cause.
It seems fine right now. Thank you.
That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
Lemmy has a memory leak? Or, should I say, a “lemmory leak”?
A pretty bad one at that…
But… but… Rust…
Rust protects you from segfaulting and trying to access deallocated memory, but doesn’t protect you from just deciding to keep everything in memory. That’s a design choice. The original developers probably didn’t expect such a deluge of users.
Rust makes holes and that’s how leaks happen
Underrated comment
Leaking memory is safe
I’m calling it - if there’s actually a memory leak in the Rust code, it’s gonna be the in memory queues because the DB’s iops can’t cope with the number of users.
I think I found what eats the memory. DB iops isn’t the cause - looks like the server doesn’t reply before all the database operations are done. The problem is the unbounded queue in the activitypub_federation crate, spawned when creating the ActivityQueue struct. The point is, this queue holds all the “activities” - events to be sent to federated instances. If, for whatever reason, the events aren’t delivered to all the federated servers, they are retried with an exponential backoff for up to 2.5 days. If even a single federated instance is unreachable, all events remain in memory. For a large instance, this will eat up the memory for every upvote/downvote, post or comment.
Lemmy needs to figure out a scalable eventual consistency algorithm. Most importantly, to store the messages in the DB, not in memory.
You should take this entire comment and paste it in as a issue on the Lemmy Github Issues page.
I think the devs have been aware of the issue, theoretically, for a while. A proper solution requires some significant changes, so it was being postponed because this wasn’t considered urgent.
You should consider bringing this up at [email protected]
On some of the latest release candidates I think so.
Lemmy has a memory leak? Or, should I say, a “lemmory leak”?
A lemmory meek, obviously!
Wait isn’t lemmy written in rust how do you create a memory leak in rust? Unsafe mode?
let mut memory_leak = Vec::<u8>::new(); loop { memory_leak.push(0); }That’s not a memory leak though. That’s just hording memory. Leaked memory is inaccessible.
In the example it’s inaccessible.
It’s not. The vec is still accessible and if it goes out of scope rust automatically clears the memory.
But it will never fall out of scope because of the loop
Exactly but it’s still accessible since it’s in scope.
Exactly.
What was that? We’re going to need more and better hardware soon, and you have a Patreon and a paypal on the sidebar?
Yeah, that sounds pretty reasonable, we can work with that.
There is a sidebar?
On web browser, yep.
I’m thoroughly confused by most of what is going on. I’m using kbin on mobile. The interface isn’t terrible and it fills the random surfing void that I am used to.
Just a heads up, you’re using kbin which is a different “website” than lemmy. You are also seeing a thread that originated in lemmy. That means both kbin and lemmy can see each other’s content, but the website UI is different between them so if you read about some UI element and can’t find it in kbin that might be one of the reasons.
That being said, kbin also has a sidebar with useful information. In the mobile site you can find it at the very bottom on the page, since it doesn’t fit to the side in such a small screen.
Omg I never scrolled that far down lmao, now I can go block 196 and the other shit posting magazines
Yesss! Lemmy was difficult to use until I did that. Then eventually I had a long enough list of communities/magazines that I could browse subscribed only and that really supposed the issue. I’m busy not a fan of random porn and bigoted memes y’know?
I mean if a random nude crosses my feed I’m not bothered by it but I really just want to read some cool stuff maybe news etc for like 15 mins at a time throughout the day and this is scratching that itch for me.
If you’re on Kbin mobile, you can also tap the hamburger bar on the top left to bring up that menu directly, instead of having to scroll all the way down.
Also great for if you want to fiddle with dark mode and all of that.
Omg so thats what that goddamn button does… thank you
Every community has a sidebar. If you’re at the page for the community it should be at the top of the website. “sidebar+”
If you’re using a 3rd party app it’s different for each one in terms of how to see it.
Just signed up for the 8€/month tier, hopefully my blue checkmark will arrive soon!
It’s here!
✅
(Please do note that the blue checkmark is for our 1€ “Bean” tier subscribers.)
We’re very pleased to provide you with our premium deluxe green checkmark for your generous 8€ subscription to our “Bacon” tier!
Feel free to reach out to our Lemmy Cares team @ 1-800-FUC-SPEZ for any questions or concerns that you may have.
Thanks u/ruud!
How are you managing all of this in addition to having a life outside of Lemmy? This kind of went from 0-60 in a blink.
Same happened with mastodon.world in November. Family goes first, then work, and then all of my hobbies, of which this is one. (But the one taking up most time at the moment…)
You’re a legend! Thanks so much Ruud!
Glad to hear that your priorities are in the right order. Family comes before helping random strangers posts pictures of beans on the internet.
Blasphemy!
Small remark instead of /u/… you should use the at-sign
@to mention somebody. For example: Hey @[email protected].How do you know which community the user is from?
It should tell you on mousehover if not straight up listed. If there’s no community, it’s because they come from lemmy.world.
TBH its all confusing and all names should be full names like email adress.
In the browser when you hover over the display name, you should see the full username. I don’t know how it’s shown in the various apps but it should appear if you view the user’s profile.
Thanks for putting in all the hard work. Especially on a holiday. We appreciate you guys!
It’s not a holiday here… just a normal work day
Thanks for the hard work and transparency. It gives confidence that you guys will get through this.
It is hard not to root for your success in this. I’m temporarily off .world, but would love to come back when you guys get this under control more.
