So it’s been a few days, where are we now?
I also thought given the technical inclination of a lot of our users that you all might be somewhat interested in the what, how and why of our decisions here, so I’ve included a bit of the more techy side of things in my update.
Bandwidth
So one of the big issues we had was the heavy bandwidth caused by a massive amount of downloaded content (not in terms of storage space, but multiple people downloading the same content).
In terms of bandwidth, we were seeing the top 10 single images resulting in around 600GB+ of downloads in a 24 hour period.
This has been resolved by setting up a frontline caching server at pictrs.blahaj.zone, which is sitting on a small, unlimited 400Mbps connection, running a tiny Caddy cache that is reverse proxying to the actual lemmy server and locally caching the images in a file store on its 10TB drive. The nginx in front of lemmy is 301 redirecting internet facing static image requests to the new caching server.
This one step alone is saving over $1,500/month.
Alternate hosting
The second step is to get away from RDS and our current fixed instance hosting to a stand-alone and self-healing infrastructure. This has been what I’ve been doing over the last few days, setting up the new servers and configuring the new cluster.
We could be doing this cheaper with a lower cost hosting provider and a less resiliant configuration, but I’m pretty risk averse and I’m comfortable that this will be a safe configuration.
I woudn’t normally recommend this setup to anyone hosting a small or single user instance, as it’s a bit overkill for us at this stage, but in this case, I have decided to spin up a full production grade kubernetes cluster with a stacked etcd inside a dedicated HA control plane.
We have rented two bigger dedicated servers (64GB, 8 CPU, 2TB RAID 1, 1 GBPS bandwidth) to run our 2 databases (main/standby), redis, etc on. Then a the control plane is running on 3 smaller instances (2GB, 2 CPU each).
All up this new infrastructure will cost around $9.20/day ($275/m).
Current infrastructure
The current AWS infrastructure is still running at full spec and (minus the excess bandwidth charges) is still costing around $50/day ($1500/m).
Migration
Apart from setting up kubernetes, nothing has been migrated yet. This will be next.
The first step will be to get the databases off the AWS infrastucture first, which will be the biggest bang for buck as the RDS is costing around $34/day ($1,000/m)
The second step will be the next biggest machine which is our Hajkey instance at Blåhaj zone, currently costing around $8/day ($240/m).
Then the pictrs installation, and lemmy itself.
And finally everything else will come off and we’ll shut the AWS account down.
who humour room property punishment road stomach button sun damage family current land cut mass fire pull dependent year see grip yesterday self prose market face fall sand system sky lift touch limit full goat very man short support electric collar if push why harbour stretch well all effect out
@[email protected] I’m pleased that this post has been up for 30 minutes and no one has jumped in with “Acktually…” and proceeded to tell you how you’re doing it wrong. jk /jk
Thankfully I have no clue about this stuff, but appreciate the detailed update nonetheless. #Hajkey #blahajzone
@jo @[email protected] That’s because the ones who are complaining are wisely doing so in their own spaces, as they ought to. They’re also revealing just how much they know about things, in the usual armchair quarterback / bleacher flyhalf fashion.
@ayilathebailey Good points. I’m in a bit of a mood and really need to tone down my ascerbic replies, lest I be accused of the same. @[email protected]
@jo @[email protected] And here I am adding a little more pepper in my pitches.
I think it’s because what they are saying is pretty solid. It’s the exact solution I’d recommend
Awesome! Keep up the good work
How can we donate to keep the infrastructure up and running?
There’s a KoFi! https://ko-fi.com/blahajzone
plant sound meeting enough bright carriage wide eye angle important wood polish sticky rice last blood chance join decision warm clock solid smile earth flower paint waste stem against needle yesterday damage metal material come year punishment division smash development east addition attack early flower here teaching degree pig weather
/qa/ won
free man meat art danger middle chance old delicate knife question part law who wall punishment hammer material hole brother tail sudden memory possible open company dear farm bridge wheel a winter letter danger science front thin wool step flag stick berry sudden camera little needle town tooth present range
/qa/ won
kettle very bottle belief goat motion sun approval run reading woman wound but sweet smash value operation increase test at army shake comb worm healthy crime waste as instrument humour farm protest full bird monkey off paper parallel wet polish foot healthy note bright relation past seat warm leaf umbrella
collar fixed good leaf woman door not berry kiss society payment space person needle up eye man yesterday full father polish church back waiting summer ink road picture copy secret learning this thumb fork only experience fact run military agreement ear polish strong doubt flag sort death potato drain question
digestion oven again wise at mark month steam army apple brain lock bed time cup form history the leather powder potato cause love debt reward smash side whip brother natural bath attempt soft driving roll slip moon ant snow mixed stretch military oil impulse voice and cough driving cart steam
disease payment farm coat poison berry turn goat song chalk gun current chance turn oven able walk value discovery rat cold smell young birth thin middle right young mother curtain taste head hard range vessel frame force responsible why yesterday flight board when pain rat store debt market sock ornament
disgust north angry approval sock wool self cotton flag competition fertile smile rest work able expert even rail first but land skin cruel fixed education protest poor angle sound ornament insect mine probable brass year comfort market nail though plane trousers much oven blow steel brother thick grey observation milk
tail news nation rate horn hearing still care star destruction smell stomach theory disgust great spring like simple metal serious heart equal position growth disease man while act table right wall far though brain direction cat put ever late iron like probable board as flower lead hope mass expert left
colour hospital screw drain carriage quite necessary foot cotton carriage black sort general fear end scissors almost such authority cloth book wax rain top amusement necessary place sudden umbrella young wise sharp page strong lift brown medical tax feeble see complete strong prose hate basin stretch day owner disgust voice
window future thunder structure sex smile come request lead mixed breath true hope sudden distribution week record branch addition now blade land play meal rate digestion vessel knee suggestion system arm sea place breath over high floor important advertisement fork distance hour bucket warm about fiction small winter and nose
gun metal comparison brain light horse love get rough so distribution answer wide late insect level knowledge special top of great past bell sail any almost question healthy shut porter family burst conscious respect friend milk the arch healthy mother that back wet operation box cloud only poison simple cause
