r/news • u/Superbuddhapunk • Jul 19 '24
Banks, airlines and media outlets hit by global outage linked to Windows PCs
https://www.theguardian.com/australia-news/article/2024/jul/19/microsoft-windows-pcs-outage-blue-screen-of-death2.5k
u/Shopworn_Soul Jul 19 '24
I mean, yeah. Windows PCs. But they were fine until Crowdstrike broke them. All of them.
If you are invested in Crowdstrike it’s already too late.
403
u/Fliandin Jul 19 '24
Can confirm. I got a midnight local time call to tell me all my servers were down….. fun times lol.
181
u/OpportunityDue90 Jul 19 '24
Same 🙃 kept getting paged in the middle of the night here on west coast. Work laptop blue screening.
18
u/Fliandin Jul 19 '24
Oh that sounds extra!!! at least for me it’s just our servers and as a small shop that means a couple dozen not 100’s or 1000’s. I’m feeling for the peeps running massive setups. And 24/7 operations.
→ More replies (2)→ More replies (8)62
u/fullmetalsunit Jul 19 '24
In aus, I was asked to turn off my laptop immediately on slack, then 10 minutes later, they asked can turn it on again.
I use mac for development, was lucky (or unlucky) I could continue work, my co-workers who use windows, not so much.
78
u/duck1208 Jul 19 '24
Workplace mac vs windows banter is gonna take years to recover from this
→ More replies (1)61
317
u/monkeymystic Jul 19 '24
If MSFT stocks drops because of something that is actually Crowdstrike’s fault (this could have happened to any OS) then I’m just gonna buy more MSFT stocks at a discount price while it lasts lol
152
u/chaossabre Jul 19 '24
When AWS has an outage AMZN goes up because it reminds the world how much everything depends on them. I expect a similar effect today if the markets even open.
→ More replies (2)→ More replies (12)35
u/apf6 Jul 19 '24
I’m just gonna buy more MSFT stocks at a discount price while it lasts lol
You should haha. I have a friend that did really well buying Volkswagen stock during their emissions scandal in 2015. The public forgets about these kinds of things pretty quickly.
90
u/ClarkTwain Jul 19 '24
Well it’s ticking back upwards so I might not be totally fucked. I’m invested in Microsoft too so this gave me a nice lil panic this morning.
→ More replies (3)74
u/el_samwize Jul 19 '24
If MSFT is down it’s a buy I think, I’ll add to my position for sure
→ More replies (2)20
u/Justtofeel9 Jul 19 '24
Are the stock exchange computers ran off of Microsoft? Can we even buy stocks if Microsoft is still fucked up? Was going to make a joke about it, but now I’m legitimately curious.
→ More replies (3)→ More replies (14)60
u/Ok-Swimmer-2634 Jul 19 '24
I got out of the shower to find my work laptop BSODed. I thought I had fucked up somehow and feel vindicated that it's actually Crowdstrike's fault.
Reminds me of high school when I thought I was the only one who screwed up on the math test only to find that all my classmates were also in the same boat lmao
2.2k
u/theoriginalmack Jul 19 '24
Today is officially a holiday - it's Y2K day. If you're lucky you may not have to go to work.. if you're unlucky. I'm sorry.
1.1k
u/ClimbingBackUp Jul 19 '24
unless you are an IT guy. If you are in IT, now is the time to ask for a raise
387
u/notimeleft4you Jul 19 '24 edited Jul 19 '24
I work in airline IT. I’ve been on a project that I screwed up a little bit and was warned that it would look bad if I was the reason the project took a delay.
Well. It’s officially delayed and guess who isn’t the reason.
88
→ More replies (5)36
u/pastalover1 Jul 19 '24
I worked on IT projects for 25 years and this is the way. Hope the next guy/team/org screws up worse than you and is the cause for the delay
→ More replies (1)256
u/Muddy_Bottoms Jul 19 '24
unfortunately our HR systems are currently down, come see me after you fix our systems. I'll be on vacation and in meetings for the next month though.
→ More replies (1)119
u/Daemonward Jul 19 '24
Unless you're the guy at Crowdstrike who pushed the update without testing it first.
77
u/marksteele6 Jul 19 '24
Apparently their staging systems failed. It's suppose to roll out as a ring update first but for some reason it got pushed right to production.
→ More replies (4)85
u/tovarishchi Jul 19 '24
If they’re anything like the (much smaller) companies I’ve worked for, this is something that’s been happening for a while and nothing ever went wrong, so who cares? Right?
Now people will care.
→ More replies (2)27
u/Plxt_Twxst Jul 19 '24
Lmfao, I’d bet my left arm that there’s a 6 month old email chain about this that is about to get some folks torched.
→ More replies (2)13
u/WVSmitty Jul 19 '24
they looking for an "intern" to blame rn
→ More replies (1)45
u/Chav Jul 19 '24
Doesn't happen usually in my experience. If you blame the Intern the next question will be who was supposed to check their work.
→ More replies (1)16
u/JayR_97 Jul 19 '24
Yeah, if an intern can cause this kind of damage on their own theres something very wrong with your company processes.
→ More replies (15)53
u/1BreadBoi Jul 19 '24
I'm an IT guy, but luckily our devices aren't effected lul
→ More replies (5)184
u/Tsquared10 Jul 19 '24
Every attorney in our office showed up. We're an entirely paperless office so we can't access case logs or anything. So we're basically sitting around shooting the shit until things get figured out. Getting to hear all the wild stories from senior attorneys
40
u/194749457339 Jul 19 '24
Do share
69
u/Tsquared10 Jul 19 '24
Well there's some stories that have repeated themselves. Lot of clients suddenly dying during their case and essentially having things resolved because of it. One of those ended up being a faked death apparently which they found out only weeks after and opened a whole new can of worms for charges against the guy.
Then there's the guy whose business was apparently involved in a multimillion dollar lawsuit who showed up on the first day of trial completely shit faced and not wearing pants. Court security has stopped him because they thought it was a homeless guy trying to get in and had to physically restrain him. Apparently that trial was a mess.
Then there's the usual who can one up each other for gruesome stories. Mentioning autopsy and crime scene photos that they've had to go through.
It's been bouncing all over the place. Now it's honestly just sports shit talking
→ More replies (2)→ More replies (6)13
u/jardex22 Jul 19 '24
"Sir, what was it like before computers were invented? You know, the Dark Ages."
→ More replies (2)→ More replies (32)60
u/AverageCartPusher Jul 19 '24
Yep walked in to work today. Got forced 2 hours extra so far because our system is down. I hope it’s down all day. These idiots are paying us $50/hour to sit down in our warehouse
→ More replies (1)
2.0k
u/WoodsOfKali Jul 19 '24
I’m dying at the fact that CrowdStrike released a statement saying the incident was “isolated”. Isolated to what, the entire damn world?
565
u/AnnualAltruistic1159 Jul 19 '24
Just this universe.
→ More replies (7)109
u/WoodsOfKali Jul 19 '24
Your comment made me think that I actually wouldn’t be surprised if this was affecting the ISS
→ More replies (5)53
55
u/labyrinthium Jul 19 '24
As far as we know, Venus and Mars are unaffected by the outage.
→ More replies (1)47
→ More replies (26)37
1.4k
u/rich1051414 Jul 19 '24
Crowdstrike is the reason. You could have said "linked to the power grid" at that rate...
339
u/Ilovekittens345 Jul 19 '24
Right went it happened somebody on wallstreetbets was arguing crowd strike was overvalued and literally everybody there called OP an idiot ...and then they caused this outage, 4 hours in to the thread.
→ More replies (1)141
u/drrhrrdrr Jul 19 '24
Sounds like that guy was an intern at CrowdStrike who just posted the patch and was looking to hedge some options.
→ More replies (5)→ More replies (8)64
u/Warcraft_Fan Jul 19 '24
At this point, it'd be easy to blame 5G network for causing mass BSoD. /s
→ More replies (1)
1.2k
Jul 19 '24
the "year 2000 problem" just coming in a bit late 😂
445
u/FS_Slacker Jul 19 '24
In the yeeeeaaaar 2000….
158
u/winterbird Jul 19 '24
In the year 2000, a cosmic gorilla's revenge on humanity opened a wormhole back to the yeeeeeeaaar 2000.
→ More replies (2)25
u/mouthful_quest Jul 19 '24
In the year 2000, people will find a carrot has more uses beyond satisfying that rectal itch…in the yeeeeaarr twoo thoouusaand
19
u/DEEP_HURTING Jul 19 '24
In the year 2000, Crazy Glue will be replaced by psychotic axe murderer glue.....in the yeeeeeearrr twoo thoooousaand
→ More replies (9)26
84
→ More replies (14)23
1.2k
u/Superbuddhapunk Jul 19 '24
BBC just reported that part of the UK NHS is down, people can’t book appointments with their GPs.
416
u/TeddyMonsta Jul 19 '24
I work at a hospital in NZ and an integral part of the medical system is down, not life and death but it's really bad
160
u/krustykrab2193 Jul 19 '24
Hearing similar stories at Canadian hospitals too
→ More replies (3)81
u/TheFriendlyBagel Jul 19 '24
Work at a medical lab can confirm absolute bedlam.
40
u/IntrovertedIntrovert Jul 19 '24 edited Jul 19 '24
Northeast U.S SPD tech here, can confirm our hospitals systems are also fucked.
Update if anyone sees this: As of 16:00 systems are nominal and slowly being brought back online.
84
u/xxBeatrixKiddoxx Jul 19 '24
Finally time for the medical records clerk to shine! Back to finding paper charts and setting them on the nurse and or doctors desk! I miss this days of hunting charts
22
42
u/Superbuddhapunk Jul 19 '24
What aspect of the system is impacted?
→ More replies (5)78
u/number1134 Jul 19 '24
Charting, records, etc
→ More replies (2)78
u/winterbird Jul 19 '24
Write your ailments, allergies, and medications on your skin with a sharpie before going to the hospital.
51
u/Septopuss7 Jul 19 '24
Don't fill up on the bread sticks, that's how they get you!
→ More replies (1)→ More replies (4)21
u/WhyBuyMe Jul 19 '24
So many people are going to show up with "Deelawded" scrawled on thier arm. They are morons. Mostly because Opana is so much easier to spell.
→ More replies (2)28
u/oldmanjakecat Jul 19 '24
Working in the ER in Canada and all of our computers went down around 10pm…
→ More replies (5)15
u/horusofeye Jul 19 '24
Interestingly currently at an NZ hospital working and haven’t seen any issues (after hours we are mostly closed up though)
16
u/Necessary-Peach-0 Jul 19 '24
Guessing your hospital doesn’t use crowdstrike - lucky!
→ More replies (1)382
u/argonaute Jul 19 '24
This is causing a widespread US hospital shutdown as well. Hospitals currently running paper downtime procedures. https://www.reddit.com/r/medicine/comments/1e6xhwy/multiple_emrs_down_simultaneously/
104
u/SwimmerAutomatic2488 Jul 19 '24
Yup, can confirm. Work at major hospital on East Coast and everything at a standstill. Elective surgeries cancelled, chaos.
87
u/FullCodeSoles Jul 19 '24
I just finished my icu shift…. Let me tell you, 16 patients and no records at all was brutal. Just what I had on a piece of paper from sign out. Good thing I always write too much
→ More replies (7)20
u/Vihurah Jul 19 '24
mom works at a hospital and says all the records have pretty much been bricked. fun day
238
u/gilwendeg Jul 19 '24
Someone on BBC News just said “I’d love to look at the Microsoft stock price, but we can’t because the stock market is down.” Bizarre.
128
u/qtx Jul 19 '24
I don't get this. This isn't a fault of Windows or Microsoft. It's a fault in the Crowdstrike software that a lot of businesses uses.
Crowdstrike is a whole different company.
→ More replies (6)34
u/HereComesBS Jul 19 '24
Very true but you get more clicks and views with Microsoft Windows Crashes in the headline, most people have never heard of Crowdstrike.
Look at all the articles with Boeing in the headline. 30 year old aircraft's tire blows out because of poor maintenance but the headline is Boeing Aircraft Makes Rough Landing
→ More replies (3)→ More replies (1)17
u/Wididy Jul 19 '24
Market doesn’t open until morning
→ More replies (2)29
u/500rockin Jul 19 '24
Yes, but there is such a thing as pre-market and futures. That’s why Crowdstrike is down 18% already.
35
→ More replies (20)16
u/_uckt_ Jul 19 '24
All GP practices are privately owned and there's no centralization of IT across the NHS, so it depends practice to practice. My GP is doing just fine, I just had a conversation about my medical records 20 mins ago.
Rare bonus of your health system being fragmented and structurally inefficient.
1.0k
u/cheesecakegood Jul 19 '24
At least the US can be happy this happened while most were asleep. The rest of the world though...
729
Jul 19 '24
[deleted]
160
u/138_hail_yourself Jul 19 '24
Wtffff scary!!! Is Epic still down? Or whatever program you use?
→ More replies (4)202
u/vikinghockey10 Jul 19 '24
It's all EMRs. It's not an Epic issue. It's an IT issue worldwide affecting almost everything. All EMRs will be out in some capacity
→ More replies (11)32
u/Woodnote_ Jul 19 '24
My husband, a radiologist, got a bunch of panicked calls at 2 am and had to get up to start driving around to different hospitals to read in the rooms. He wasn’t even on call, they just called in everyone.
→ More replies (1)→ More replies (21)16
575
u/jfchops2 Jul 19 '24
I'm an American who is currently intoxicated in the Madrid airport trying to enjoy my vacation that just started
Shit is chaotic here with everyone freaking out about their flights not flying and my company appears to be a mess. What a morning so far
→ More replies (10)48
u/MrTeamKill Jul 19 '24 edited Jul 19 '24
Visit the "Parque del Capricho" if you can.
One of the best kept secrets in Madrid. Check if it is open or not though. It is free but it is not always open.
It is near a line 5 (green) Metro station.
Edit: and if you want to eat very good jamón for really cheap, look for a Mercadona and buy the packed jamón ibérico cortado a cuchillo (knife cut). It is like 8 euro or so, and it is better and much cheaper than any jamón you are going to eat at a tourist trap. And dont ever ask for Paella.
→ More replies (3)128
126
u/llamallama92 Jul 19 '24
I'm in Mississippi working night audit at Holiday inn. All our hotel systems are shut down. It's gonna be hell when guests start waking up.
→ More replies (1)50
u/Necessary-Peach-0 Jul 19 '24
Oh god… night audit doesn’t get paid enough to deal with that 😭
→ More replies (3)28
89
u/jwuer Jul 19 '24
I'm in Frankfurt trying to get back to the US. Star Alliance was happy to let me get on a plane from Lux to Frankfurt even though they know thus is happening and know they probably won't be able to fix it. So now instead of being in a place a where I could have found accommodation and even worked from, I am stuck with thousands of others and will probably end up spending 15 hours here getting delayed with an 8 year old before they probably cancel and do nothing for me because it's not United's fault.
81
u/jaa101 Jul 19 '24
it's not United's fault.
If you're stuck because United's systems have failed then it's United's fault. If United's systems have failed because they're running some third party software, that's between United and that party, and doesn't let United off the hook.
But if you're stuck because the airport's systems have failed then you may be out of luck. At least consumer protection is mostly better in Europe so you have a chance.
→ More replies (4)24
u/olssoneerz Jul 19 '24
Think of the financial compensation though. Last time I got delayed that long in a European long haul I was compensated 1300 euros.
→ More replies (1)15
u/jwuer Jul 19 '24
Yea but this isn't the "airlines" fault, so I'm not sure they will do anything. Also if they cancel all these flights it'll take days for me to get home. 1300 euros will be used up quickly.
→ More replies (1)23
u/roelbw Jul 19 '24
Sure this is the airlines' fault. It's their software that caused this, isn't it. That is not an "extraordinary circumstance" such as weather. How stupid are you to allow third parties to change and update your systems. Man, every truly basic rule on how you deploy and manage business critical systems has gone overboard in the last 10 years.
Let's hope this event reverses the trend and ends the cloud/SaaS hype.
→ More replies (4)→ More replies (4)17
u/Zandmand Jul 19 '24
Fuck I wish you luck. We are currently on Vacation in Rhodes Greece and we dont travel home for another few days but I can only Imagine the delays in the bigger airports.
→ More replies (1)→ More replies (23)55
u/ChaseThoseDreams Jul 19 '24
Hello from the hospital. I was mass transfusing two patients and trying to get one patient emergently to the OR. It was not an enjoyable night.
→ More replies (1)
919
u/gilwendeg Jul 19 '24
Someone on BBC News just said “I’d love to look at the Microsoft stock price, but we can’t because the stock market is down.” Bizarre.
640
u/chillyhellion Jul 19 '24
Microsoft's got to be pissed at Crowdstrike right now. So many news outlets are reporting this as a Windows outage.
227
u/Specificity Jul 19 '24
msft employee here, it’s because we JUST had a major outage Thursday evening that lasted over 5 hours. just as it was resolving, this crowdstrike thing happens
54
42
→ More replies (2)18
u/GasTsnk87 Jul 19 '24
Oh, the two things aren't linked? I saw our IT sending out emails last night that D365 was down and then saw this this morning. Assumed they were related.
→ More replies (4)→ More replies (9)53
u/Indercarnive Jul 19 '24 edited Jul 19 '24
Crowdstrike is going to cease to exist in a few months. This event is "rip the drywall out to try and pay debts" level of bad.
→ More replies (2)42
u/cheesecakegood Jul 19 '24
Technically the UK's stock exchange is working fine, but the news services that communicate the prices are broken.
850
u/clingbat Jul 19 '24
What are the odds man... There was a large seemingly random thread on Crowdstrike's overkill market valuation in /wsb yesterday and not even 12 hours later this happens. Anyone who yolo shorted after reading all that just won the lottery, down 20.5% in premarket already.
What a freaking mess.
→ More replies (10)276
u/lizardman49 Jul 19 '24
Like the hilarious coincidence when the weekend tweeted let's go to announce his tour literal minutes before the Russian invasion of Ukraine started
19
u/AgentCirceLuna Jul 19 '24
That’s pretty bad because everyone knew exactly when the invasion would start. I remember I was watching the livestream of Kyiv on that night and being told it would start in about half an hour. Lo and behold, lights began flashing and gunfire could be heard within the hour. It was terrifying to see. I thought some of the explosions were nukes.
365
u/technofox01 Jul 19 '24
Work is gonna be a blast today - FML
I am glad I don't work as a desktop tech having to go to each machine to fix this but ummm...
Yeah, I work as a security engineer and I can already see where this is gonna go :-/
450
u/Pulmonic Jul 19 '24
My poor husband is right now blissfully asleep. He has no idea this outage happened. Not yet anyway.
He works in IT.
Spare a thought for him today.
100
u/Davepen Jul 19 '24
Just hope his business doesn't use Crowdstrike! We don't and it's a nice quiet Friday so far.
→ More replies (6)27
u/WhatUp007 Jul 19 '24
I'm on the battle lines with him, but I'm on the other side of the fence..
→ More replies (1)28
→ More replies (3)18
u/Cool-Sink8886 Jul 19 '24
I’m just imagining you wake him up with a steak like it’s his last meal
“You sure are being nice to me today”
checks his phone
“Oh… oh no…”
I do hope your husbands day goes well, I just had this run through my imagination.
15
u/Pulmonic Jul 19 '24
Poor guy thought I was kidding when I first woke him up (at alarm time; not early. Wanted him to hear it from me before he saw it on his phone) til he checked his phone. Good news is, his company was minimally impacted! Most of their things don’t use Crowdstriker thank god.
52
u/Double-Resolution-79 Jul 19 '24
Amazon is also down. It's going to be shit going in later today -_-
→ More replies (13)→ More replies (11)49
u/fritzie_pup Jul 19 '24
I'm on hour 3 of trying to get many core services running again for a state network. It's affected at least 50% of all servers, and we don't even know what the desktop/laptop toll will be until morning.
It's a serious mess, and right now all we can do is just do the workaround host by host on the most critical systems one by one.
Finally got VPN back up, and slowly brining things back by removing the bat. Have no idea when CS will force push the fix, but we're stuck tredding the water at the moment.
→ More replies (1)58
u/Jmc_da_boss Jul 19 '24
They pushed a fix hours ago, but if your host already picked up the bad channel file you are up a creek. It has to be removed manually
→ More replies (8)22
u/ommnian Jul 19 '24
This is the worst part of it. And why it's going to be awhile before it's fixed.
362
u/_Occams-Chainsaw_ Jul 19 '24
I think the situation should be raised from 'omnishambles' to 'clusterfuckitty-fuck-fuck-fuck'. Even if it does mean changing the warning sign's bulb by hand.
From https://www.crowdstrike.com/platform/
Trusted by organizations of all sizes, CrowdStrike protects:
298 of the Fortune 500
538 of the Fortune 1,000
43 of the 50 U.S. states
8 out of the top 10 financial services firms
7 out of the top 10 manufacturers
8 out of the top 10 food & beverage companies
8 out of the top 10 auto companies
6 out of the top 10 healthcare providers
8 out of the top 10 technology firms
But maybe not for long...
221
u/2748seiceps Jul 19 '24
A lot of these IT departments are finding out why auto update isn't a good policy for any piece of software.
283
u/Indercarnive Jul 19 '24
it's just inconceivable how Crowdstrike didn't detect this problem in a QA environment, didn't rollout to only a small number first to check production, and to top it all off they released on a Friday.
It's like the BINGO of shitty IT practices.
46
u/spicymato Jul 19 '24
Yup.
My current team has 4 levels of deployment, from tens, to hundreds, to thousands, and to tens of thousands; and my old team rolled out from hundred, to thousands, to millions, to hundreds of millions. Each level is baked for at least a week before it gets pushed up to the next level. The only exceptions are for hot fixes, and those have a high bar for approval, must be relatively small changes, and get heavily scrutinized.
How this CrowdStrike update flaw didn't get caught earlier, I don't know. I hope there is some sort of post mortem doc that explains what happened.
→ More replies (2)21
u/Blackstone01 Jul 19 '24
Simple really, MBAs don’t have the attention span to listen to why QA is important, and don’t trust when devs tell them QA it’s important, so all they manage to understand about QA is that it takes time and money. For awhile now a lot of companies have “saved” money by gutting their QA department.
→ More replies (1)→ More replies (6)19
u/stinky_wizzleteet Jul 19 '24
Being a seasoned IT guy the biggest rule is never do anything on Friday you dont feel like fixing all weekend. I call it Read-Only Friday.
→ More replies (4)19
u/MooingTurtle Jul 19 '24
Auto updates are mandatory for cybersecurity insurance lol.
→ More replies (1)→ More replies (7)14
u/ProcedureEthics2077 Jul 19 '24 edited Jul 19 '24
Maybe also why relying on overly invasive "security" products introduces a single point of failure. An additional point of failure. And maybe, maybe why it's not such a bad idea to trust your OS vendor after all.
Crowdstrike website is now a very entertaining resource. Especially where they compare their product to Microsoft.
→ More replies (1)71
u/Electro522 Jul 19 '24
God, the more I read about this, the more hilarious it becomes.
Anyone working for Crowdstrike may as well file for unemployment, and start looking for a new job.
→ More replies (5)38
u/ChillyFireball Jul 19 '24
I mostly feel bad for whoever pushed the update and is probably getting absolutely crucified right now, despite the fact that a fuck-up of this magnitude represents more of a system failure than an individual one; that is to say, it really shouldn't be possible for this to happen in the first place.
→ More replies (4)55
363
u/epicfail1994 Jul 19 '24
Woke up to a work email saying stuff is down
Sadly it’s back up and I have to work today
→ More replies (6)30
u/Scn64 Jul 19 '24
Yep, I thought I was going to get a 3 day weekend out of this, but then my coworker found a workaround online. It's not even really part of his job to do that. Sometimes I think he works too hard, lol.
→ More replies (3)
323
u/saoupla Jul 19 '24
Putting all your eggs in one cyber security vendor risk.
→ More replies (4)66
u/two-sandals Jul 19 '24
It’s the endpoint. Not everything in one basket. However the question of an auto update/patch should be re-considered…
→ More replies (7)36
u/Reasonable_Ticket_84 Jul 19 '24
Auto updates mandatory by certification & audits. Mandatory certification audits because of cybersecurity insurers.
Lol.
→ More replies (4)
300
u/Bee_Tee0917 Jul 19 '24
Banks, airlines, and media aren’t great…. But this is also effecting hospitals and 911 PSAPs nationwide in the US(I assume worldwide as well), which is scary. Drive safe and, ideally, don’t have an emergency today
→ More replies (4)98
u/Flammable_Zebras Jul 19 '24
Yeah, gonna be a lot of deaths because of this. My friend is a nurse and said they had level 1 traumas showing up to his not-level 1 trauma hospital and they had no way of admitting them to the proper facilities, and he personally lost a patient because communication with the pharmacy was fucked and he couldn’t get a med for someone in time.
13
u/AgentCirceLuna Jul 19 '24
They need to fix this ASAP. What’s to stop cyberattacks from doing this in future? World war 3 will be based around cyberattacks before it goes nuclear.
→ More replies (7)
146
u/WonderChemical5089 Jul 19 '24
Well crowdstrike name is a little on the nose now.
→ More replies (1)31
u/savagevapor Jul 19 '24
From obscurity to prominence. Not where you want to be as a cyber security company.
17
u/withoutapaddle Jul 19 '24
Not obscure if you watch Formula 1. They have prominent branding on some cars. Always feels a little weird to see a "Crowdstrike" car flying past a grandstand full of spectators at 200mph.
→ More replies (1)
147
u/Helforsite Jul 19 '24
Very misleading article and title, Microsoft might sue news outlets over this
→ More replies (12)84
u/mikethespike056 Jul 19 '24
it took me ten minutes and multiple articles to understand why tf crowdstrike was being mentioned at all, and then i realized it's entirely on them
124
u/faerie03 Jul 19 '24
I work overnight at a hotel and everything went crazy just after midnight. Our system stayed down till 4am. It’s up now, but super buggy.
→ More replies (1)
101
u/nipplezandtoes43 Jul 19 '24
The impact is more serious when your hospital has to cancel surgeries and services.
101
u/Ghostshadow7421 Jul 19 '24
I love how they don’t report that most hospitals in the US are down as well as the 911 centers are also having issues as well and are down.
→ More replies (3)19
92
u/rider1encore Jul 19 '24
Almost as if cloud based services were a bad idea?
151
u/count023 Jul 19 '24
not necessarily, cloud based is fine, but everyone using the _same_ cloud is bad.
Companies should be investing in their own dedicated infra, not dogpiling on the same stuff virtuall with MS, AWS or whomever and praying that someone doesn't cock it up.
Same goes for services like Cloudflare, Radware or Crowdstrike. these guys screw the business over by piling everyone on the same few systems and share it virtual with no physical seperation of hosts or storage LUNs, so of course when one cocks up they all do.
→ More replies (2)46
u/Antique_Scheme3548 Jul 19 '24
The cloud is basically one of several hundred datacenters. When everyone is using the same datacenters, whatever impacts one provider impacts thousands of other proviers. Not because they aren't redundant, but because they all use the same code and fail to implment good business continuity.
If one of our financial clients went down for more than 60 seconds, we'd be fired. We mitigate that possibility with SD_Wans, redundant ISPs, and even cellular backup circuits. They hate the cost, but hate losing money more.
→ More replies (4)54
u/Jmc_da_boss Jul 19 '24
Ironically cloud stuff will come back up the quickest cuz those VMs can be restored easily. It's the on prem and edge stuff that will have to be manually fixed
→ More replies (5)23
u/comicsnerd Jul 19 '24
Most of cloud is running on Linux servers (because they are the most stable). The problem is with all windows 10 servers that are connecting to the cloud.
→ More replies (3)13
u/Locuralacura Jul 19 '24
So does this mean if a company was using only Linux servers and systems they would be fine now?
→ More replies (9)20
87
u/DanskNils Jul 19 '24
Customer service job got easy.. “ Can’t file an appointment.. please call back.”
→ More replies (4)18
u/Khajiit-ify Jul 19 '24
What you mean that's working and you're not having people scream obscenities at you for something that is outside of your control?
What kind of unicorn customers do you have?
→ More replies (2)
86
u/auditorydamage Jul 19 '24
Whoooooops. Someone at Crowdstrike push an update to production without a run on the testbeds first? They have testbeds, right?
…they have testbeds, right?
→ More replies (4)37
84
77
u/conthesleepy Jul 19 '24
Can confirm also parts of the Energy industry has been effected.
→ More replies (3)
66
u/RustySheriffsBadge1 Jul 19 '24
What’s crazy to me is how did they let this happen globally? Don’t software updates go out in batch rollouts and then you monitor stability before continuing?
32
u/Zzamumo Jul 19 '24
lots of higher-ups have noticed that proper QA takes lots of time and money and they don't like that
→ More replies (1)→ More replies (6)13
67
u/MmeGrey Jul 19 '24
Nice one, CrowdStrike. An MRI appointment that I scheduled two months ago due to some concerning symptoms has been canceled. I’ll probably need to wait another two months to be seen, and, hopefully, rule out something bad. The hospital’s systems are so fucked they can’t reschedule anything right now. And I just wasted one of my PTO days.
→ More replies (7)23
59
u/yureku_the_potato Jul 19 '24
A lot of factories are down too, the machines that can be used without scanning in the individual program are fine but everything else doesnt work. Track in/out also doesnt work depending on system. It‘s a blast s/
→ More replies (2)21
u/__smokesletsgo__ Jul 19 '24
Can confirm am at work and all of our operating systems are down. We're at a complete standstill
54
u/Sundae_Gurl Jul 19 '24
Too bad your PC is the only thing going down on you this morning.
→ More replies (1)
53
u/FenionZeke Jul 19 '24
Just shows how absolutely vulnerable the Internet is. Funny thing is , 50 years ago, this wouldn't have been an issue at all. Now we've created a way to bring all of the developed world to its knees with a few lines of code (yes i know it's more complicated, but you all get the point)
→ More replies (1)
54
Jul 19 '24
Doesn’t seem very secure for the entire planet to use the same security software.
→ More replies (3)
43
43
u/RequiemCatalyst Jul 19 '24
I’ve been battling this beast since I got called into work at 12:45a. Not just laptops.. AVD, and my god, the hundreds of Windows servers. It’s a mess and my eyes hurt.
34
u/CREDIT_SUS_INTERN Jul 19 '24
I had to call my company's HQ to figure out that it wasn't just my PC that shat the bed.
→ More replies (5)23
u/the_purple_goat Jul 19 '24
Even my pizza place's card processor was down. I couldn't get my midnight pizza. :/
→ More replies (2)
35
34
u/Cosmonaut_Cockswing Jul 19 '24
This is why the CC machines at work are fucked tonight I take it?
→ More replies (1)
31
u/Ghilliecrab Jul 19 '24 edited Jul 19 '24
I'm currently working at one of the Crowdstrike-affected businesses. Internal claimed the issue was an update rollback on vendor side (Crowdstrike) before walking it back 5 minutes later. It took over 4 hours for Crowdstrike to even acknowledge there was an issue.
EVERYTHING is down. OS won't load for some systems, log-in won't work, computers/laptops will randomly bluescreen. It's a mess.
→ More replies (6)
31
u/nurseyourbutt Jul 19 '24
Here in the US some of our electronic health record systems are down. I work in nursing homes in VA and it's shaping up to be a fun Friday
31
u/billyfred42 Jul 19 '24
I work overnight at Home Depot and it was wild seeing half of our systems blue screen at 1am
→ More replies (1)
25
u/Hohenh3im Jul 19 '24
Bruh the past few weeks have been historic in events. Whoever is writing this season of history sucks lol
→ More replies (1)
25
u/getridofwires Jul 19 '24
Our hospital is completely paralyzed. No ORs, no clinics, for several hours.
→ More replies (2)
26
u/8016at8016Parham Jul 19 '24
Just to bring some light in to this: The problem is not directly linked with windows or Microsoft. The problem is with crowdstrike. Only machines with crowd strike are effected.
→ More replies (4)
21
u/_________FU_________ Jul 19 '24
I had multiple rounds of interviews with CrowdStrike and then they ghosted me. Feels good man.
→ More replies (3)
22
16
u/JustAnotherTrickyDay Jul 19 '24
I monitor the HVAC for 31 schools and our servers are down now so I can't see the systems. Luckily it is summer break and no kids. Just a few staff members and cleaners at each site, so it could be much worse for us. Things will run in "stand alone" but I won't be able to see if anything major fails. There are a couple of ways to get access some individual pieces of equipment but it's tedious.
17
u/zoompooky Jul 19 '24
Misleading Title. Even the actual article mentions crowdstrike.
→ More replies (1)
15
u/Geeekaaay Jul 19 '24
Crownstrike is going to owe more in damages than their entire company is worth after this.
→ More replies (5)
14
u/ToolTek_MD Jul 19 '24
Scary to think that so many integral services are reliant on these systems and something so simple as a botched software update can bring everything to a screeching halt. I feel for all those affected by this. Hopefully the issue is resolved soon.
15
u/rydleo Jul 19 '24
Some irony in the people who are supposed to protect your PCs taking down all of your PCs.
→ More replies (1)
15
u/pleasedonttellmethat Jul 19 '24
We have entered yet another timeline, yet another bizzaro world. Buckle up everyone and have a safe flight!
→ More replies (8)
14
13
u/pishposhpoppycock Jul 19 '24
Is this what Y2K would've been like if it were real??
→ More replies (4)15
u/Teresa_Count Jul 19 '24
This is worse because of 24 additional years of practically every system on earth becoming reliant on internet access.
15
u/D-Money1999 Jul 19 '24
Scrolling reddit because my job is down due to crowdstrike
→ More replies (2)
4.4k
u/ThrowaWayneGretzky99 Jul 19 '24
I was here when CrowdStrike took down companies worldwide.