Hey everyone, and happy new year!

Sorry about that super long downtime there. Yesterday (Sunday) morning at 10:03AM PST our server suffered a physical hardware failure, apparently a power supply failure. Unfortunately despite opening a ticket with our hosting vendor (OVH) a few minutes later and them claiming to have 24/7 support, nobody looked at our ticket until this morning when their phone support lines opened and I called them.

They’ve now replaced a defective power supply and we’re back online, after ~26 hours of being offline. Some pretty disappointing response times, to put it nicely.

We’re planning to move away from OVH at the end of this month, onto proper enterprise grade hardware that we own and control. This will give us a HUGE boost in server resources and allow us to scale for the foreseeable future, while also giving us the control to resolve problems like this much quicker. Expect another follow up post about this in the next couple weeks once I’ve put together the migration plan.

Timeline:

  • Jan 5th 10:03am PST - We get alerts to the server being non-responsive.
  • Jan 5th 10:05am PST - I pull up the console via IPMI and it’s completely non-responsive. Attempting to power off / on the server or do anything, does not work.
  • Jan 5th 10:15am PST - Initial support ticket created with OVH. I followed up a couple times over the next few hours, and got no response.
  • Jan 6th 6:32am PST - Called OVH, gave them the case number and asked them to investigate
  • Jan 6th 7:34am PST - I get notified they’ll start their “intervention” in 15 minutes.
  • Jan 6th 11:04am PST - Call them again, the tech is still working on it and they’ll get back to me with an update
  • Jan 6th 11:34am PST - “I was informed by our data centre technician that there is an issue with the power supply unit for the rack on which your server resides. Your server will come back online once they have replaced the power supply.”
  • Jan 6th 12:17pm PST - We’re back up finally!

Edit on Jan 7th @ 8:40am PST: We just had another outage of about an hour. Investigating with OVH.

  • Avid AmoebaEnglish
    9•4 months ago

    Happy new year!

    They’ve now replaced a defective power supply and we’re back online, after ~26 hours of being offline.

    It was a nice break from the firehose.

  • @[email protected]English
    16•4 months ago

    Thank you for all your hard work and top notch comms, Lemmy.ca admin peeps. I spent the day at dbzer0, which is always fun, but happy to be back home!

  • @[email protected]English
    9•4 months ago

    I am simultaneously happy to see you’re all back up and running again and mourning the loss of the sudden increase in productivity I had while you were down.

  • m-p{3}MEnglish
    70•4 months ago

    Thank you for handling the issue :)

    I can’t count how many times I opened my Lemmy.ca bookmark out of habit during the outage 😬

    • IninewCrowEnglish
      8•4 months ago

      Same here … I opened up my bookmark about a dozen times before I realized I should switch to my alt lemmy account.

      It’s also made me realize that I should maintain an extra or a couple extra lemmy accounts for times like these.

      • @[email protected]English
        3•4 months ago

        Yes you should! I was in a bit of a back and forth with someone and had to switch to my alt to continue (once I realized there was no ETA on the server getting fixed lol)

  • @[email protected]English
    12•4 months ago

    I went back to using my .ee account during the downtime and had to see… shudder… Hexbear posts… I guess it was nice to see the clownshow hasn’t changed at all

  • IninewCrowEnglish
    11•4 months ago

    Thank you! … This has become my landing page on the internet so it’s great to see you guys up again.

    Thanks for all the work you guys do!

  • ⓝⓞ🅞🅝🅔English
    14•4 months ago

    Well, thanks to this outage, I got a pretty good understanding of how often I turn on my Lemmy app. It turns out to be a whole heck of a lot. Haha.

    Glad things are back up and running and that you all have a path forward to mitigate this in the future.

  • @[email protected]English
    7•4 months ago

    Thanks for the update, I was wondering what was going on.

    Glad to hear you’re getting your own hardware now. I offered before but again if you need any help I can offer up some free labour on the weekends. I am a net admin so if you need any help with networking or security give me a shout.