This isn’t a gloat post. In fact, I was completely oblivious to this massive outage until I tried to check my bank balance and it wouldn’t log in.

Apparently Visa Paywave, banks, some TV networks, EFTPOS, etc. have gone down. Flights have had to be cancelled as some airlines systems have also gone down. Gas stations and public transport systems inoperable. As well as numerous Windows systems and Microsoft services affected. (At least according to one of my local MSMs.)

Seems insane to me that one company’s messed up update could cause so much global disruption and so many systems gone down :/ This is exactly why centralisation of services and large corporations gobbling up smaller companies and becoming behemoth services is so dangerous.

  • bitwolf
    link
    fedilink
    181 year ago

    Would you really be paying for Crowdstrike for use at home?

  • @[email protected]
    link
    fedilink
    31 year ago

    From my understanding, they have some ring 0 thing that fucked up. Could that not in theory happen on our beloved Linux systems? Or does the kernel generally not give that option?

  • @[email protected]
    link
    fedilink
    35
    edit-2
    1 year ago

    I love how everyone understands the issue wrong. It’s not about being on Windows or Linux. It’s about the ecosystem that is common place and people are used to on Windows or Linux. On windows it’s accepted that every stupid anticheat can drop its filthy paws into ring 0 and normies don’t mind. Linux has a fostered a less clueless community, but ultimately it’s a reminder to keep vigilant and strive for pure and well documented open source with the correct permissions.

    BSODs won’t come from userspace software

    • Nonagon ∞ Orc
      link
      fedilink
      171 year ago

      While that is true, it makes sense for antivirus/edr software to run in kernelspace. This is a fuck-up of a giant company that sells very expensive software. I wholeheartedly agree with your sentiment, but I mostly see this as a cautionary tale against putting excessive trust and power in the hands of one organization/company.

      Imagine if this was actually malicious instead of the product of incompetence, and the update instead ran ransomware.

      • @[email protected]
        link
        fedilink
        English
        31 year ago

        If it was malicious it wouldn’t have had the reach a trusted platform would. That is what made the xz exploit so scary was the reach and the malicious attempt.

        I like open source software but that’s one big benefit of proprietary software. Not all proprietary software is bad. We should recognize the ones doing their best to avoid anti consumer practices and genuinely try to serve their customers needs to the best of their abilities.

      • @[email protected]
        link
        fedilink
        3
        edit-2
        1 year ago

        That’s precisely why I didn’t blame windows in my post, but the windows-consumer mentality of “yeah install with privileges, shove genshin impact into ring 0 why not”

        Linux can have the same issue. We have to keep the culture on our side here vigilant and pure near the kernel.

      • Skankhunt42
        link
        fedilink
        31 year ago

        I deployed it for my last employer on our linux environment. My buddies who still work there said Linux was fine while they had to help the windows Admins fix their hosts.

    • TimeSquirrel
      link
      fedilink
      17
      edit-2
      1 year ago

      Seems to be some sort of kernel-embedded threat detection system. Which is why it was able to easily fuck the OS. It was running in the most trusted space.

    • @[email protected]
      link
      fedilink
      71 year ago

      AV, EDP they offer other solutions as well. I think their main selling point is tamper-proof protection as well.

    • chameleon
      link
      fedilink
      141 year ago

      Company offering new-age antivirus solutions, which is to say that instead of being mostly signature-based, it tries to look at application behavior instead. If Word was exploited because some user opened not_a_virus_please_open.docx from their spam folder, Word might be exploited and end up running some malware that tries to encrypt the entire drive. It’s supposed to sniff out that 1. Word normally opens and saves like one document at a time and 2. some unknown program is being overly active. And so it should stop that and ring some very loud alarm bells at the IT department.

      Basically they doubled down on the heuristics-based detection and by that, they claim to be able to recognize and stop all kinds of new malware that they haven’t seen yet. My experience is that they’re always the outlier on the top-end of false positives in business AV tests (eg AV-Comparatives Q2 2024) and their advantage has mostly disappeared since every AV has implemented that kind of behavior-based detection nowadays.

  • @[email protected]
    link
    fedilink
    English
    18
    edit-2
    1 year ago

    I wanted to share the article with friends and copy a part of the text I wanted to draw attention to but the asshole site has selection disabled. Now I will not do that and timesnownews can go fuck themselves

    • @[email protected]
      link
      fedilink
      12
      edit-2
      1 year ago

      It is annoying. Some possible solutions:

      On desktop: Using Shift + ALT you often can overrule this and select text anyway.

      On mobile: Using the reader mode or the Print preview often works. It does for me on this website.

    • @[email protected]
      link
      fedilink
      111 year ago

      heres the entire article

      Latest Crowdstrike Update Issue: Many Windows users are experiencing Blue Screen of Death (BSOD) errors due to a recent CrowdStrike update. The issue affects various sensor versions, and CrowdStrike has acknowledged the problem and is investigating the cause, as stated in a pinned message on the company’s forum.
      Who Have Been Affected
      Australian banks, airlines, and TV broadcasters first reported the issue, which quickly spread to Europe as businesses began their workday. UK broadcaster Sky News couldn’t air its morning news bulletins, while Ryanair experienced IT issues affecting flight departures. In the US, the Federal Aviation Administration grounded all Delta, United, and American Airlines flights due to communication problems, and Berlin airport warned of travel delays from technical issues.
      In India too, numerous IT organisations were reporting in issues with company-wide. Akasa Airlines and Spicejet experienced technical issues affecting online services. Akasa Airlines’ booking and check-in systems were down at Mumbai and Delhi airports due to service provider infrastructure issues, prompting manual check-in and boarding. Passengers were advised to arrive early, and the airline assured swift resolution. Spicejet also faced problems updating flight disruptions, actively working to fix the issue. Both airlines apologized for the inconvenience caused and promised updates as soon as the problems were resolved.
      Crowdstrike’s Response
      CrowdStrike acknowledged the problem, linked to their Falcon sensor, and reverted the faulty update. However, affected machines still require manual intervention. IT admins are resorting to booting into safe mode and deleting specific system files, a cumbersome process for cloud-based servers and remote laptops. Reports from IT professionals on Reddit highlight the severity, with entire companies offline and many devices stuck in boot loops. The outage underscores the vulnerability of interconnected systems and the critical need for robust cybersecurity solutions. IT teams worldwide face a long and challenging day to resolve the issues and restore normal operations.
      What to Expect:

      -A Technical Alert (TA) detailing the problem and potential workarounds is expected to be published shortly by CrowdStrike.
      -The forum thread will remain pinned to provide users with easy access to updates and information.

      What Users Should Do:

      -Hold off on troubleshooting: Avoid attempting to fix the issue yourself until the official Technical Alert is released.
      -Monitor the pinned thread: This thread will be updated with the latest information, including the TA and any temporary solutions.
      -Be patient: Resolving software conflicts can take time. CrowdStrike is working on a solution, and updates will be posted as soon as they become available.

      In an automated reply from Crowdstrike, the company had stated: CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. Symptoms include hosts experiencing a blue screen error related to the Falcon Sensor. The course of current action will be - our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket. Status updates will be posted as we have more information to share, including when the issue is resolved.
      For Users Experiencing BSODs:
      If you’re encountering BSOD errors after a recent CrowdStrike update, you’re not alone. This appears to be a widespread issue. The upcoming Technical Alert will likely provide specific details on affected CrowdStrike sensor versions and potential workarounds while a permanent fix is developed.
      If you have urgent questions or concerns, consider contacting CrowdStrike support directly.

      • @[email protected]
        link
        fedilink
        61 year ago

        If you have urgent questions or concerns, consider contacting CrowdStrike support directly.

        Something tells me that isn’t going to provide the comfort it was meant to.

  • @[email protected]
    link
    fedilink
    44
    edit-2
    1 year ago

    I work in hospitality and our systems are completely down. No POS, no card processing, no reservations, we’re completely f’ked.

    Our only saving grace is the fact that we are in a remote location and we have power outages frequently. So operating without a POS is semi-normal for us.

    • @[email protected]
      link
      fedilink
      121 year ago

      I’ve worked with POS systems my whole career and I still can’t help think Piece Of Shit whenever I see it

  • suoko
    link
    fedilink
    31 year ago

    A couple of days ago a Windows 2016 server started a license strike in my farm … Coincidence?

  • Angry_Autist (he/him)
    link
    fedilink
    81 year ago

    Is there an easy way to silence every fuckdamn sanctimonious linux cultist from my lemmy experience?

    Secondly, this update fucked linux just as bad as windows, but keep huffing your own farts. You seem to like it.

    • Morphit
      link
      fedilink
      5
      edit-2
      1 year ago

      I’d unsubscribe from [email protected] for a start.

      I’m pretty sure this update didn’t get pushed to linux endpoints, but sure, linux machines running the CrowdStrike driver are probably vulnerable to panicking on malformed config files. There are a lot of weirdos claiming this is a uniquely Windows issue.

      • Angry_Autist (he/him)
        link
        fedilink
        21 year ago

        Thanks for the tip, so glad Lemmy makes it easy to block communities.

        Also: It seems everyone is claiming it didn’t affect Linux but as part of our corporate cleanup yesterday, I had 8 linux boxes I needed to drive to the office to throw a head on and reset their iDrac so sure maybe they all just happened to fail at the same time but in my 2 years on this site we’ve never had more than 1 down at a time ever, and never for the same reason. I’m not the tech head of the site by any means and it certainly could be unrelated, but people with significantly greater experience than me in my org chalked this up to Crowdstrike.

      • CommunityLinkFixerBotB
        link
        fedilink
        English
        21 year ago

        Hi there! Looks like you linked to a Lemmy community using a URL instead of its name, which doesn’t work well for people on different instances. Try fixing it like this: [email protected]

  • @[email protected]
    link
    fedilink
    English
    1
    edit-2
    1 year ago

    Microsoft should test all its products on its own computers, not on ours. Made an update, tested it and only then posted it online.

  • @[email protected]
    link
    fedilink
    English
    621 year ago

    I’ve just spent the past 6 hours booting into safe mode and deleting crowd strike files on servers.

    • @[email protected]
      link
      fedilink
      181 year ago

      Feel you there. 4 hours here. All of them cloud instances whereby getting acces to the actual console isn’t as easy as it should be, and trying to hit F8 to get the menu to get into safe mode can take a very long time.

      • @[email protected]
        link
        fedilink
        English
        61 year ago

        Ha! Yes. Same issue. Clicking Reset in vSphere and then quickly switching tabs to hold down F8 has been a ball ache to say the least!

        • @[email protected]
          link
          fedilink
          English
          2
          edit-2
          1 year ago

          What I usually do is set next boot to BIOS so I have time to get into the console and do whatever.

          Also instead of using a browser, I prefer to connect vmware Workstation to vCenter so all the consoles insta open in their own tabs in the workspace.

      • @[email protected]
        link
        fedilink
        101 year ago

        Since it has to happen in windows safe mode it seems to be very hard to automate the process. I haven’t seen a solution yet.

      • @[email protected]
        link
        fedilink
        English
        51 year ago

        Sadly not. Windows doesn’t boot. You can boot it into safe mode with networking, at which point maybe with anaible we could login to delete the file but since it’s still manual work to get windows into safe mode there’s not much point

        • @[email protected]
          link
          fedilink
          71 year ago

          It is theoretically automatable, but on bare metal it requires having hardware that’s not normally just sitting in every data centre, so it would still require someone to go and plug something into each machine.

          On VMs it’s more feasible, but on those VMs most people are probably just mounting the disk images and deleting the bad file to begin with.

          • @[email protected]
            link
            fedilink
            English
            51 year ago

            I guess it depends on numbers too. We had 200 to work on. If you’re talking hundreds more than looking at automation would be a better solution. In our scenario it was just easier to throw engineers at it. I honestly thought at first this was my weekend gone but we got through them easily in the end.

          • Natanael
            link
            fedilink
            11 year ago

            The real problem with VM setups is that the host system might have crashed too

  • @[email protected]
    link
    fedilink
    81 year ago

    Me too. Additionally, I use guix so if a system update ever broke my machine I can just rollback to a prior system version (either via the command line or grub menu).

      • @[email protected]
        link
        fedilink
        21 year ago

        True, then I’d be screwed. But, because my system config is declared in a single file (plus a file for channels) i could re-install my system and be back in business relatively quickly. There’s also guix home but I haven’t had a chance to try that.

        • totally_notAcat (she/her)
          link
          fedilink
          11 year ago

          I would definitely recommend using guix home because having a seperate config for you more user facing stuff is so convenient (plus no need for root access to install a package declaratively) (side note take this with a grain of salt because I don’t use gnu guix I use nixos)

  • @[email protected]
    link
    fedilink
    91 year ago

    This is exactly why centralisation of services and large corporations gobbling up smaller companies and becoming behemoth services is so dangerous.

    Its true, but otherside of same coin is that with too much solo implementation you lose benefits of economy of scale.

    But indeed the world seems like a village today.

    • DigitalDilemma
      link
      fedilink
      English
      11 year ago

      you lose benefits of economy of scale.

      I think you mean - the shareholders enjoy the profits of scale.

      When a company scales up, prices are rarely reduced. Users do get increased community support through common experiences especially when official channels are congested through events like today, but that’s about the only benefit the consumer sees.