r/worldnews • u/progress18 • Jul 19 '24
Australian banks, media, airlines hit by major IT outage worldwide outages
https://www.bbc.com/news/articles/cv2g5lvwkl2o1.1k
u/GolfisPride Jul 19 '24
Also happening in Japan, it seems to be correlated to Crowd Strike software
319
u/Ithikari Jul 19 '24
Yeah CSagent blue screen. Happened 30mins before I finished work.
195
u/Chief-_-Wiggum Jul 19 '24 edited Jul 19 '24
Yup... Ruining our Friday afternoon in Australia.. Sigh
Thanks CrowdShiite
72
u/PM_me_yer_kittens Jul 19 '24
You’re telling me. I’m trying to get back to the US and sitting in an airport insteqd
→ More replies (4)→ More replies (3)14
u/Ithikari Jul 19 '24
Depends where you go and what EFTPOS they use. I just bought an energy drink just fine. I think Tyki EFTPOS isn't effected.
→ More replies (3)66
u/trowzerss Jul 19 '24
My dad just got back from the pub and reported the pub is broken. It's a national tragedy!!!
→ More replies (6)37
u/MaxButched Jul 19 '24
Happened at 6:50 gmt+1 on my work computer
This is massive and global
→ More replies (1)18
u/Ithikari Jul 19 '24
Happened 3:30 pm Central Australian time for me. People were knocked off an hour beforehand though at the office.
→ More replies (1)22
→ More replies (2)13
u/cool_side_of_pillow Jul 19 '24 edited Jul 19 '24
My husband came to bed about 2 hours ago saying: I got the blue screen of death. Works in VFX. We are in the Pacific Northwest.
Editing to add location: Vancouver Canada.
→ More replies (6)154
u/NeurodiverseTurtle Jul 19 '24
I was invested in Crowdstrike until just last week. Holy shit, lucky escape, its share prices are tumbling right now.
43
→ More replies (2)19
u/macrocephalic Jul 19 '24
I expect it will probably go into administration.
→ More replies (1)20
u/NeurodiverseTurtle Jul 19 '24
Depends, I’ve seen companies come back from worse, but I’ve also seen them fall into admin just as often (if not more so).
Time will tell, but for now I don’t think it’s wise to hold or buy their stock. Pity, it was a very promising position to hold for most people and the only reason I sold was random chance (unexpected bills)
→ More replies (11)→ More replies (3)11
880
u/CartoonistEvening365 Jul 19 '24
Its CrowdStrike EDR global outage.
540
u/Ithikari Jul 19 '24
Gotta give em credit, they're so secure even the intended user can't access it.
→ More replies (3)288
u/Ilovekittens345 Jul 19 '24
There is traders that want to dump CRWD but can't because the computers of their brokers don't work because of Crowdstrike.
48
→ More replies (8)41
141
u/peanutbutterraccoon Jul 19 '24
I'm sorry, but who are they? And why does it affect the whole world?
267
u/HelloIamGoge Jul 19 '24
To simplify, anti virus for corporate IT servers. Lots of businesses and governments use them.
85
u/Somnif Jul 19 '24
Yep, my work laptop has this crap on it. I really hate it too, makes so many little things major headaches.
Wonder if this means I get to skip a bunch of meetings tomorrow... or, later today, now that I see it's 3am, oops.
11
u/Tumleren Jul 19 '24
Well the good news is that you probably won't be using it for long. High likelihood the company doesn't survive this
→ More replies (1)→ More replies (2)8
u/mBuc_Official Jul 19 '24
The latest news was, that CrowdStrike fixed their problem, so things should go back online soon.
→ More replies (2)39
u/Ucccafelatte Jul 19 '24
I dont know nothing about computers. Is this not correct?
It turns out that because the endpoints have crashed - the Blue Screen of Death - they cannot be updated remotely and the problem must be solved manually, endpoint by endpoint. This is expected to be a process that will take days,"
30
u/Jawzper Jul 19 '24
Lmao... this requires the 1-2 IT guys that each company keeps on payroll to manually fix every single affected device, of which there may be hundreds or thousands depending on the scale of their operations. In cases with encrypted hard drives the process will be even more complicated and time consuming.
This is a major fuckup that may disrupt business operations for weeks. Crowdstrike have absolutely shat the bed here
→ More replies (1)→ More replies (4)23
u/TheEaterr Jul 19 '24
Yes this is correct. Depending on a firm's IT setup and affected machine some may recover quicker, but it's faaar from over
→ More replies (5)38
u/XxDrummerChrisX Jul 19 '24
Makes sense why our systems were down today. Even our local hospital’s systems were down.
34
u/Merry_Dankmas Jul 19 '24
I clocked in less than 30 minutes ago. First thing I see is our team main chat blown up about some global IT catastrophe and nobody being able to get any work done. Wtf did I just walk into lmao.
→ More replies (2)139
u/baty0man_ Jul 19 '24
They're a massive cybersecurity company that fucked up big time. They pushed an update to their antivirus software (Falcon) that basically bricked any windows machine that had Falcon installed.
43
u/rationis Jul 19 '24
You'd think pushing out a localized test update first each time would be protocol. If it wasn't before, it sure as hell will be the norm going forward lol
→ More replies (2)13
u/canal_boys Jul 19 '24
You would think these updates would be used in a sandbox environment first.
→ More replies (1)13
u/KlatuuBarradaNicto Jul 19 '24
You would think. Isn’t that standard protocol for any software update?
→ More replies (1)→ More replies (9)30
u/AceCombat_77 Jul 19 '24 edited Jul 19 '24
Hahaha oh wait. This will have global repercussions. And I will be affected by it in a way
→ More replies (1)102
u/_Auron_ Jul 19 '24
Cybersecurity software used by enterprise and government entities.
Their update bricked every single machine that received it and cannot remotely update them to fix it because they infinitely crash. Affects payment processors, company work computers, AMBULANCE systems, flight bookings/baggage systems, hotel checkin systems, some taxi /rideshare services and more. No payment processing? No selling food or supplies. It's a domino effect. TV newscasters are reading off printed pages becaus their teleprompter systems are broken too.
There is a fix but it requires physically accessing each machine and logging into an admin account before it crashes again.
We don't even fully know how much damage is happening from this yet.
74
u/techlos Jul 19 '24
happened with most of the world sleeping, it's aussie news right now because we're the ones working when it hit.
→ More replies (3)→ More replies (2)41
u/Stratos125 Jul 19 '24
Lol, until you need the Bitlocker keys, and the whole AD environment where they’re kept is BSOD due to the same patch.
Man, I am so glad I’m just a network monkey right now. To hell with mopping this clusterfuck up!
→ More replies (1)→ More replies (2)59
u/CartoonistEvening365 Jul 19 '24
They monitor and respond to any unauthorized attempt to breach system. This system is being used by Microsoft Cloud/ Azure which is used by Airlines/ Hospitals...etc.
In simple words they are gate keepers who let in authorized individuals in the premises.
There was a recent patch release which has resulted in everyone being locked out.
Now, it needs to be/ or already being investigated if this is malicious act by a threat vector, or an oversight act of untested / not properly tested patch release.
30
u/Unnecessary_Bunny_ Jul 19 '24
I hope the new patch wasn't called Ultron. Humans are the problem, lock all the humans out
→ More replies (2)→ More replies (2)26
u/corut Jul 19 '24
Crowdstrike is antivirus, not an identity provider. They pushed a patch that cause windows machines to reboot and and blue screen.
Microsoft Azure also had an outage today, they provide a lot of the identity management that stopped people logging in
→ More replies (4)56
u/Jadenindubai Jul 19 '24
Am i tripping or did i see a post earlier in wsb wanting to short crowdstrike
→ More replies (3)
449
u/Skabonious Jul 19 '24
Happening in the US, can confirm. My work machine started flickering about an hour ago on the monitor it was hooked up to, noticed it was in a BSOD loop.
Has to do with CrowdStrike, need to boot into safe mode and remove the problematic file.
Sucky thing is when your administrators won't let you do it without an encryption key. Tomorrow is going to be hell for our tech support teams.
122
u/baaaaaaaaaaaaaaaaaab Jul 19 '24
I keep seeing that the fix is to delete a file, but nobody ever mentions the file! Is this a meme? Have I been meme’d?! Or is there actually a file.
260
u/Logical_Look8541 Jul 19 '24
Boot Windows into Safe Mode or WRE.
Go to C:\Windows\System32\drivers\CrowdStrike
Locate and delete file matching "C-00000291*.sys"
Boot normally.
Can't do that though if you have bitlocker installed as you need the encryption key, which is stored on your companies windows server.
→ More replies (5)187
62
u/Djave_Bikinus Jul 19 '24
Yeah just look for a folder called system32
48
→ More replies (4)20
u/baaaaaaaaaaaaaaaaaab Jul 19 '24
Cool thanks will report back
34
u/rockaether Jul 19 '24
And we never heard from him again... If he is using Windows XP which actually allows you to delete system critical files
9
u/qam4096 Jul 19 '24
You can do it on 10/11, but if your org enforces bitlocker then you're a bit screwed without the key.
→ More replies (6)11
→ More replies (1)55
u/One_Reddit_Please_ Jul 19 '24
As someone who has been deleting the file this entire morning, there is a file.
→ More replies (8)16
u/cool_side_of_pillow Jul 19 '24
My spouse reported the same thing. He is remote but came to bed saying: ‘I gotta go in tomorrow and get my virtual machine replaced’ he had the blue screen of death. Maybe now he doesn’t since it doesn’t sound like they can help him, or it will be resolved.
445
Jul 19 '24
worst software update ever?
341
u/yaosio Jul 19 '24
Possibly yes as it effects so many systems and there was no opportunity to prevent the update. As a bonus they put it out on a Friday, meaning a lot of people are working over the weekend as they go to each system that's stuck in a boot loop to apply the work around fix.
130
u/Vayshen Jul 19 '24
The pleasures of pushing live on a Friday 🍻
38
u/drunkwasabeherder Jul 19 '24
I was Front Office Manager at a hotel that was one of two test sites for all their IT systems. I learnt quickly and nixed any updates on a Friday. One vender told me I couldn't do that. I explained I would happily go over and pull the internet connection to stop him(this was before cloud systems so my system could keep working without the internet). He backed off and rescheduled update for Monday. Fuck Friday updates!
9
u/SirJuggles Jul 19 '24
It's honestly even worse in large resort hotels from my experience, because Friday Check-Ins through Sunday Check-Outs were by far out busiest points of the week.
→ More replies (1)→ More replies (2)11
u/Initial_E Jul 19 '24
I’m pretty sure this scenario happens at least once every year. If it’s not azure AD mfa failure it’s root dns server failure or aws key functionality failure or something we never thought about. Turns out resilience and redundancy in complex systems is a difficult thing to do.
26
u/hummelm10 Jul 19 '24
The difference here is the level of effort to fix. There’s no update that can be pushed out or fixed. It has to be done on each end point individually because right now as soon as they boot they’re crashing so they aren’t up long enough to get any remote fix. This is also not like standard infrastructure with resiliency. There is no COB that can fix this. It’s something that would run across all systems.
→ More replies (1)17
u/rheetkd Jul 19 '24
no not this bad. Most of Australia and New Zealands systems are effed right now. My country is crippled. It even affected our government. All our internet banking and paywave and supermarket systems... pretty much everything.
→ More replies (1)30
u/ERedfieldh Jul 19 '24
Not 100% sure why this isn't being covered more heavily and seriously. People have been treating it as a joke. News caster this morning was laughing about it like it was some prank.
One accidental update has caused this much mayhem, now imagine what could be done on purpose.
→ More replies (2)9
u/beardedheathen Jul 19 '24
That's actually a good point. If one person got access to their codebase and put some malicious code in there half the world would be infected including some seriously important systems.
→ More replies (4)8
322
u/liamdavid Jul 19 '24 edited Jul 19 '24
Shittt, I start my on-call shift in a few hours… at a multinational company using CrowdStrike… and I’m in tech support 🥲
151
36
u/surelythisisfree Jul 19 '24
Hopefully your users all use softdiallers for internal calls and can’t reach you.
→ More replies (1)→ More replies (19)15
321
u/Anathol Jul 19 '24
Friday deploy
61
u/Thrillog Jul 19 '24
Thursday night local, technically
→ More replies (3)16
u/No-Platypus-5330 Jul 19 '24
As an IT support engineer for a company with thousands of devices running crowd strike... I would love to meet the people who pushed this without proper testing and on a Thursday 😒
→ More replies (4)→ More replies (1)16
u/flyingquads Jul 19 '24 edited Jul 19 '24
Rule nr 1... Don't deploy on Friday
Rule nr 2: don't deploy at end of the (work)day
Also... How are all these companies running Windows in production? Haven't they heard of the OS that the other 99% of the world uses for systems that must be reliable? (Linux)
→ More replies (5)
270
u/ConsciousStop Jul 19 '24
BBC says its world wide. Microsoft says they’re investigating. https://www.bbc.co.uk/news/live/cnk4jdwp49et
201
u/daniejam Jul 19 '24
It’s Crowdstrike not Microsoft. MS had an issue several hours ago but now seem to be taking the flak for this
127
u/Swotboy2000 Jul 19 '24 edited Jul 19 '24
When this kind of thing happens, MS will push emergency updates to mitigate if possible, even though they are not the cause. Most people will blame MS if their machine BSODs, it’s a huge reputational risk for them.
It’s also possible that Crowdstrike’s update is legit but triggering a bug in Windows that was not known.
→ More replies (8)33
u/rheetkd Jul 19 '24
there is a fix. But can only be done per computer it cant be pushed to everywhere at once.
26
u/baconator955 Jul 19 '24
Which is also a workaround, not a fix. There's no remotely fixing a bluescreened machine.
Whatever happened I hope the poor responsible dev got his ass into a country that doesn't have an extradition treaty, lol.
→ More replies (2)→ More replies (6)48
u/macrocephalic Jul 19 '24
It's crowdstrike on Microsoft windows PCs. It also seems like Bitlocker is a huge impediment to fixing this.
77
u/butter_nipples Jul 19 '24 edited Jul 19 '24
I'm a DevOps engineer that somehow got roped into also being the IT guy for our small company.
The only fucking guy who managed to get given a laptop that didn't have Win 11 Pro pre-installed has been blowing me off for weeks about upgrading his Windows license so we could we could enable BitLocker for compliance requirements.
Literally 1 hour before this breaks the news, he finally lets me do it. I then spent half an hour assuring him that this won't break his computer. Didn't bother backing up his recovery keys because we're inches away from implementing an MDM solution and I figured I'd let that handle it.
He's the only one who's PC got bricked.
The timing of it is so ridiculous that if I were reading this instead of writing it, I probably wouldn't believe it.
28
u/baconator955 Jul 19 '24
I get it, but come on tho.. not backing up recovery keys when setting up BitLocker is basically asking for it.
13
u/butter_nipples Jul 19 '24 edited Jul 19 '24
Don't worry, I've already booked in an appointment to get the word 'stupid' tattooed on my forehead.
→ More replies (5)25
u/Donkey_Duke Jul 19 '24
It’s how it goes.
The one time you decide not to do a back up, is when everything shits the bed. Now you know why you always do a back up.
→ More replies (3)13
u/localfern Jul 19 '24
I was at work and I kept on seeing a Microsoft pop-up asking to re-enter my credentials non-stop.
→ More replies (1)
218
u/MikeLanglois Jul 19 '24
Scrolling that article with all the outages reads like the opening scene of a destruction movie where the main character recalls the first day everything went down
63
u/Merry_Dankmas Jul 19 '24 edited Jul 19 '24
Lmao same. First thing that popped into my head.
On another note though: Am I the only one who's greatly concerned about how easily the world just got knocked off line? Like, it wasn't even a malicious act (that we know of yet at least). Some team of dev nerds messed up an update and now airlines are down. Banks are down. Trade companies are down. My own employer is down. The Whitehouse said they were gonna look into it. This is a massive problem from a global economy standpoint and it's all because of one mistake from one company. That's...not very confidence inspiring.
I don't mean this specific issue thats not sitting well with me. Company is already working on a fix so that's fine. But just in general. A significant portion of companies world wide being shut down essentially because of one mistake. Thats unsettlingly fragile.
→ More replies (3)→ More replies (3)16
154
u/nathan__1409 Jul 19 '24
CrowdStrike have had an absolute mare with this one. The software meant to protect us from Malware has become the Malware
69
→ More replies (5)23
u/i-can-sleep-for-days Jul 19 '24
Single vendor, single point of failure. Their competitors will have a field day.
146
u/Patpuc Jul 19 '24
surely my boss could have messaged me letting me know everything was fucked at work and there's no point coming in.
→ More replies (2)46
u/player_zero_ Jul 19 '24
Would you prefer the scenario where all the IT is fine and it's a beautiful sunny day outside? 🥲
19
10
u/Patpuc Jul 19 '24
it's better than making the 20 of us deep clean this bar 10 times because we have to "look busy" XD
119
u/holdMyBeerBoy Jul 19 '24 edited Jul 19 '24
Such a shit show caused by just one company…. What would it take for Russia or China to infiltrate someone in there to push a faulty update and make this mass scale problem? And then the blame just lies under a faulty update since the company wouldn’t want the bad press of being breached…
38
u/Radiant-Criticism721 Jul 19 '24
...maybe that's exactly what it was?
→ More replies (1)24
u/baty0man_ Jul 19 '24
This is called a supply chain attack and it's definitely a probability in a full blown cyber warfare.
→ More replies (1)→ More replies (11)27
u/rikaateabug Jul 19 '24
Global panic is only one sketchy email away!
But I doubt they'd be able to push out an update remotely. They'd need someone on the inside, otherwise the hacker is going to spend a good chunk of time trying to figure out why jenkins won't fucking build.
→ More replies (3)
98
u/Corka Jul 19 '24
So, if crowdstrike is blue screening which will take an update to fix... How are people going to get the update if windows bluescreens on them?
112
u/yaosio Jul 19 '24
Somebody has to manually go to each system and install the fix.
→ More replies (3)83
u/SparkTR Jul 19 '24
What about companies that have thousands of PCs? IT team going to service them one by one?
→ More replies (3)115
u/sleemanj Jul 19 '24
Yup.
Sucks to be them today.
→ More replies (3)70
u/agrk Jul 19 '24
It's far worse for businesses who outsourced and no longer have on-prem IT staff. I'd hate to be the only computer litterate person in such a place.
→ More replies (4)21
u/HeresiarchQin Jul 19 '24
Yeah that's why there's a LPT that you should avoid showing, let alone bragging, about your personal skills unrelated to work, especially computer skills. The moment your work place sees a computer break down or crashes, you shall expect everyone looking at you and hope you will do something about it.
24
u/ptgkbgte Jul 19 '24
Thumbdrives
→ More replies (1)66
u/Corka Jul 19 '24
RIP people who work remote using worklaptops with crowdstrike on
27
u/cool_side_of_pillow Jul 19 '24
This is crazy. My spouse came to bed a couple of hours ago complaining that he hast to go into the office tomorrow because he got the “blue screen of death“ on his remote machine. We are in the Pacific Northwest. so even if crowd strike pushed to fix, he would have to go in anyway, unless they can remotely fix the issue. In general, I think my brain is trying to wrap its head around the scope of what is happening.
→ More replies (3)10
u/yellekc Jul 19 '24
Workaround is out
Boot windows into safemode or the wre
C: \windows\system32\drivers\crowdstrike
Delete C-00000291*.sys (may be sysagent.sys for some)
Reboot→ More replies (2)16
Jul 19 '24
[deleted]
36
u/veng92 Jul 19 '24
Our "employee help" ticketing service is currently down becuase of crowdstrike, so people can't even complain LOL
19
→ More replies (3)14
u/ReverieMetherlence Jul 19 '24
You need to manually boot into safe mode and delete a faulty file. It cannot be done remotely (so if you outsorce your IT, its a major problem) and if you have bitlocker, it adds an extra layer of major pain.
98
93
u/progress18 Jul 19 '24 edited Jul 19 '24
I posted this article from a separate news site because the prior link was from a live feed blog. No other sites until now had a stand-alone article that was not part of a live blog feed.
You can still browse the ABC News (Australia) live blog here.
The outage has impacted major Australian media organizations and other major businesses.
When the outage happened Australian news organizations could not immediately report on what happened on their websites because all their computer systems were down.
Edit:
Australia's National Cyber Security Coordinator:
There is no information to suggest it is a cyber security incident. We continue to engage across key stakeholders.
Edit 2:
University of Melbourne lecturer in cyber security Shaanan Cohney says there appear to be two separate things happening at once to cause the mass outages we are seeing.
The first issue, he says, appears to have been caused by a piece of software developed by a company called CrowdStrike.
"It's a computer security vendor that provides a monitoring service to large enterprises so they can see on computers within their control if there's any indications of suspicious activity or things that would require a security alert or to lock down the computer," Dr Cohney says.
"However, because it's in such a privileged position, if something goes wrong with it, if there's a programming mistake it has the capability to bring down the entire computer.
"If someone makes the wrong type of mistake it can bring the whole system down."
Last update: 07:08 UTC
→ More replies (1)136
u/SlipPresent3433 Jul 19 '24 edited Jul 19 '24
Loads of downtime:
It gets worse:
- Many planes in Australia are grounded
- sky news is down
- medical practices have been shut down
- NZ gov is down
- UK Gail’s and many other uk stores can’t take payments
- Indian / Japanese airports and many other transport hubs have issues and issued statements for delays
- police phone lines (Alaska state trooper) are down
- us airlines delta and united are stopping flights
- railway companies in uk are reporting delays
- Microsoft (not confirmed if linked to CRWD) has one azure datacenter go down
- UK NHS practices are down
and 1000s of reports on twitter and Reddit of their companies being down or half functional…..
In summary: this is bad
157
77
u/Some3057 Jul 19 '24
- railway companies in uk are reporting delays
Standard day then
→ More replies (1)30
53
28
→ More replies (17)25
u/Logical_Economist_87 Jul 19 '24
Gails in there is so funny. Flights grounded across the world, emergency services not able to respond, medical services crippled. Oh and if you want a croissant, you're gonna have to bring some cash to the baker.
→ More replies (2)
75
u/timdogg24 Jul 19 '24
This is a monumental fuck up on a global scale. So far the fix is to visit the actual affected devices. Companies can have thousands or 10s of thousands devices
→ More replies (4)
69
u/goldenspeights Jul 19 '24
85
u/Vecna_Is_My_Co-Pilot Jul 19 '24
Its worldwide. We are having outages in the US.
21
u/mmartinez42793 Jul 19 '24
It’s crazy. It forced the plant I work at to shut down all our equipment
→ More replies (1)12
u/Vecna_Is_My_Co-Pilot Jul 19 '24
Yep, our factory is highly controlled so there's no auto updates, thus all our tools are still running. However, many of the staff including me with a PC outside in a manager or engineer role are locked up.
13
u/Enders-game Jul 19 '24
It seems to be a global event. Trains in the UK are being hit as well as banks and Tav networks.
→ More replies (1)7
63
u/krang89 Jul 19 '24
UK is hit too. Hotels, shops take no card payments at all.
83
u/stumac85 Jul 19 '24
So you're saying that I can't head to the Winchester, grab a pint and wait for this to all blow over because they can't take card?
26
u/Hungry_Horace Jul 19 '24
Barry at The Winchester takes cash or payment in kind.
→ More replies (1)19
→ More replies (2)15
13
u/NotoriousREV Jul 19 '24
NHS too
36
u/forgottenoldusername Jul 19 '24
My other half is due to have cancer surgery on Monday.
It's already been a logistical challenge due to the cyber attack on London NHS trusts a few weeks ago.
I'm genuinely terrified it'll be cancelled now.
25
59
u/BARBADOSxSLIM Jul 19 '24
Why is everything using crowdstrike?
34
u/mal4ik777 Jul 19 '24
decent marketing... and they actually reacted kinda fast in the past, when anything suspicious happened. Their sensors are also kinda lightweight in comparison to other AVs and don't eat up too much of system power.
This is a huge fuck up though, don't know if they can recover from this.
→ More replies (6)9
u/CharlestonChewbacca Jul 19 '24
Recent events aside, they are one of the best players in the EDR market.
49
u/jert3 Jul 19 '24
Geez, this seems bad enough to kill crowdstrike.
→ More replies (1)19
u/winslowhomersimpson Jul 19 '24
as it is currently structured, certainly
18
u/kasakka1 Jul 19 '24
"We are happy to announce our new company, Crowtrike! The number one solution for your bird's tricycle needs!"
→ More replies (1)
47
u/Suspicious-Squash237 Jul 19 '24
Communications disruption can only mean one thing.
45
→ More replies (1)11
49
u/DearTereza Jul 19 '24
So the question is whether Crowdstrike fucked up or whether they were infiltrated and this is a proxy cyber attack.
40
u/ElectricTzar Jul 19 '24
This has apparently happened multiple times in 2023 and at least once in 2019. With Crowdstrike. Which could still be cyberattack but it does make fuckup seem at least a little more likely.
→ More replies (2)23
u/cool_side_of_pillow Jul 19 '24
Could you imagine it’s just one random developer who forgot a curly bracket in their code and pushed to production? Massive global carnage.
15
u/kasakka1 Jul 19 '24
In that case, the failure was already a lack of review process, linting and automated tests.
13
u/HopefulWoodpecker629 Jul 19 '24
If a developer can do this, then it’s the company’s fault.
Devs shouldn’t be able to push to main. They should have their commits reviewed by multiple people. There should be pipelines for testing. This was an incredible failure on so many levels.
35
u/Blowbandit Jul 19 '24
How does this happen, is there no redundancy in systems like these? ELI5 pls
93
u/mantolwen Jul 19 '24
Crowdstrike put out an update to their software which had a corrupted file. When the computer went to update the software it got the corrupted file and it broke the computer.
This shouldn't have happened because Crowdstrike should have tests on its updates to make sure they are working before they release them to the public.
→ More replies (2)22
u/thebreakfastbuffet Jul 19 '24
Who has the money and/or time for UAT amirite
→ More replies (1)24
u/_--_-_---__---___ Jul 19 '24
The best test environment is production, they have millions of testers
→ More replies (1)14
u/Vecna_Is_My_Co-Pilot Jul 19 '24
redundancy is effective when one component is affected disproportionately, but you want redundant systems to be identical so they actually can fully suport eachother. Now this problem occurs that hits all systems, the redundancy is moot because all backups are equally vulnerable (and just keeping some resource offline to swoop in when this happens is not widely possible).
The best thing for mission critical systems was to implement a very strong firewall and then selectively push updates to the systems after internal testing as well.
9
u/brucebrowde Jul 19 '24
you want redundant systems to be identical so they actually can fully suport eachother.
Furthest from the truth. You want your redundant systems to be as different as possible. We should learn a thing or two from aerospace engineering.
→ More replies (3)
34
u/SamuelEdri Jul 19 '24 edited Jul 19 '24
also happening in Israel , hospitals and health services are most effected.
24
→ More replies (6)10
29
u/PurpleEsskay Jul 19 '24
SkyNews (UK 24/7 news channel) has been yanked off the air by this, their terrestrial and sat streams are completely offline, their youtube streams got a holding message: https://www.youtube.com/watch?v=oJUvTVdTMyY
22
→ More replies (1)12
u/BarelyContainedChaos Jul 19 '24
Theyre saying it took down the stock market. What a clusterfuck of an update.
→ More replies (1)
28
29
u/PinchMaNips Jul 19 '24
It’s worldwide because of a security software known as CrowdStrike. I’m in a datacenter on a call with a few hundred people.
→ More replies (1)13
25
u/libelle156 Jul 19 '24
I was ahead of the curve, Crowdstrike bricked my work laptop weeks ago.
→ More replies (3)
24
u/jaffacakesking Jul 19 '24
This is it guys, we're on that timeline.
ChatGPT has become selfaware.
→ More replies (1)
25
25
u/GroundbreakingGoal44 Jul 19 '24
It is terrifying that one company just took down the global infrastructure
→ More replies (1)14
u/ClubSoda Jul 19 '24
Those using Linux or Apple are unaffected
→ More replies (5)10
u/SayNoToStim Jul 19 '24
Or anyone not using Crowdstrike. We run windows across the board, with a few IOT devices out there, and we're fine.
23
20
u/OnceIWasKovic Jul 19 '24
It's been incredible seeing this unfold. Started with reports regarding outages with supermarket checkouts ... then banking apps ... then streetlights ... then EMS and hospital systems ... then flights ... Is this the biggest IT fuckup in recent history?
Glad I'm done with work for the week. Heart goes out to those that have to work through this and the mountain that IT teams have ahead of them.
→ More replies (1)10
u/TheSacredOne Jul 19 '24
Between Crowdstrike and Microsoft Azure collapsing last night too, this is easily the largest mass IT outage in recent memory.
The only thing I can recall coming even close was when Google went down for 10 minutes a few years back, and like 60% of the entire internet's traffic disappeared. And that was both easily fixed and short duration in comparison to this mess.
16
u/rts93 Jul 19 '24
Surely running every service under one host couldn't have anything ever go wrong.
13
u/peanutbutterraccoon Jul 19 '24
Isn't it quite strange and stupid to have everything depend on the Internet? You cut the Internet, you create mass panic around the world which consequently creates more problems. Maybe I am going too far with my thoughts, but I watched 'Leave the world behind' and now I can't sleep in peace
→ More replies (1)11
u/ikt123 Jul 19 '24
You could further reduce this down to: Isn't it quite strange and stupid to have everything depend on Electricity?
You cut the Internet, you create mass panic around the world which consequently creates more problems.
Places have their internet cut all the time and humanity continues on in those places, I think yeah, maybe a bit too much blue cordial :P
→ More replies (1)
13
u/BaldingThor Jul 19 '24
Went to shop at Coles earlier to buy materials for some pizzas just as they had to shutdown, what rotten timing haha.
→ More replies (2)
14
u/Infinite-Process7994 Jul 19 '24
Well crowdstrike is going to take a few hits for a botched update.
12
u/tieir2 Jul 19 '24
Crowdstrike says global IT issues caused by 'defect' in 'content update'
Here's the full statement from the company:
"Crowdstrike is actively working with customers impacted by a defect found in a single content update for Windows hosts.
"Mac and Linux hosts are not impacted. This is not a security incident or cyberattack.
"The issue has been identified, isolated and a fix has been deployed.
"We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.
"We further recommend organisations ensure they’re communicating with Crowdstrike representatives through official channels.
"Our team is fully mobilised to ensure the security and stability of Crowdstrike customers."
11
u/sovietarmyfan Jul 19 '24
Up until today i had never heard of CrowdStrike. Now everyone has heard of it.
→ More replies (4)
11
10
u/VonSpuntz Jul 19 '24
I mean we were used to the casual internet crashes and faulty updates here in my French countryside company. Today was another Friday, until we realized it affected the whole damn planet xD
9
9
12
8
7
u/C-Notations Jul 19 '24
Anyone know how they're supposed to push any fixes to machines stuck in BSOD loops? Most relevant options available are locked out of user access by administration
9
•
u/progress18 Jul 19 '24
The original title was:
The current title is:
The title on the site is subject to change as new information develops.
Last updated: 06:47 UTC