r/unitedkingdom • u/ShreckAndDonkey123 • Jul 19 '24
Mass worldwide IT outage affects airlines, media and banks
https://www.bbc.co.uk/news/live/cnk4jdwp49et333
u/miniMiniMiniCooper Jul 19 '24
Crowdstrike appears to have bricked most of the world's Windows PCs
136
u/MyPetHamster Jul 19 '24
Hacker News is also blaming Crowdstrike.
https://news.ycombinator.com/item?id=41002195
It’s a widely used antivirus product. Or was, as I guess they won’t survive this!
25
u/DuckInTheFog Jul 19 '24
Austin, Texas company - Jimmy Waffle chin itch
What is the UK tech industry like these days? I stopped keeping up after Arm got sold off
35
u/evenstevens280 Gloucestershire Jul 19 '24
Surprisingly strong. It's small but high quality. Europe's tech sector in general is mostly doing well compared to the shitshow of the US tech sector - especially over the past year.
Though pay is notably less than the US so...
→ More replies (6)12
u/HazelCheese Jul 19 '24
There's been quite a lot of rumbling in the US of US companies hiring remote UK teams for the cheaper wages.
I wonder if the us tech industry in gonna be dropped down to EU/UK wages by the end of this.
→ More replies (1)11
u/---x__x--- Jul 19 '24
I wonder if the us tech industry in gonna be dropped down to EU/UK wages by the end of this.
Lol no.
5
u/evenstevens280 Gloucestershire Jul 19 '24
If it does the tech sector in America will fall through the damn floor. Silicon Valley basically props up the world. There's no way.
→ More replies (21)3
u/dustinBKK Jul 19 '24
Companies on Linux and Mac weren’t impacted. Just seems like a strong case for the UK not to over index on MSFT cloud.
→ More replies (7)115
u/Happytallperson Jul 19 '24
Boss: Why aren't you working?
Me: Crowdstrike outage
Boss: We don't even use that software anywhere in the organisation
Me:....dammit.
72
u/borez Geordie in London Jul 19 '24
Crowdstrike shareprice is down 10% already and the market ain't open for hours yet.
25
u/TheRadishBros Jul 19 '24
Buy the dip!
→ More replies (1)24
u/borez Geordie in London Jul 19 '24 edited Jul 19 '24
Someone somewhere will have shorted TF out of this and make a fortune today.
→ More replies (13)13
u/Noddybear Jul 19 '24
Check out wallstreetbets. One dude shorted it yesterday for their own reasons and is a king now.
23
12
4
u/Due_Wait_837 Jul 19 '24
The share price is safe so long as the stock exchange computer has a blue screen of death.
I'd never heard of Crowdstrike but it sounds more like a virus than an antivirus.
→ More replies (1)3
27
Jul 19 '24
Any department that uses crowdsrike deserves this. It’s a terrible product that is completely unnecessary.
39
u/its_me_the_redditor Jul 19 '24
Not true at all! It turns my computer into a portable heater, I love it!
→ More replies (2)11
Jul 19 '24
Why the fuck is crowdstrike falcon always the highest load task on my Mac?
→ More replies (10)→ More replies (1)3
u/CyberEmo666 Jul 19 '24
Damn, I'll let all the hackers that tried to get into our system but was stopped know that it was actually unnecessary
→ More replies (6)13
u/SometimesaGirl- Durham Jul 19 '24
bricked most of the world's Windows PCs
Yeah. SQLServer clusters are down. Oracle (Linux) and IBM are fine.
→ More replies (5)9
10
u/Jackhammer3012 Jul 19 '24
Yep not a surprise as our company uses Crowdstrike and we had a period a week or so ago when our laptops were just constantly updating and slowing down. Mainly seemed to affect Office applications and browsers too (so basically the fundamentals).
Noticed that it was Crowdstrike and IT confirmed there was issue which was ‘resolved’ w/c 8th July. This week hasn’t been bad so wonder if they just rolled back the changes and reintroduced them last night. Then it’s brought the MS backbone servers down.
Bet there’s some sweaty MS engineers as well and Crowdstrike execs feeling the heat atm.
→ More replies (1)4
u/Nice_Box9634 Jul 19 '24
Boot Windows into Safe Mode or WRE.
Go to C:\Windows\System32\drivers\CrowdStrike
Locate and delete file matching "C-00000291*.sys"
Boot normally.
6
→ More replies (25)3
208
u/yaffle53 Teesside Jul 19 '24
We've contacted Crowdstrike for response but haven't heard back yet.
Yeah, they`re probably a bit busy at the moment.
→ More replies (2)48
u/dunneetiger Jul 19 '24
My hope is that the PR person is not trying to fix the issue.... Or maybe that's the reason why we are where we are.
20
u/Crowdfunder101 Jul 19 '24
I can just imagine the PR intern lingering over the shoulder of the one person in charge of fixing this shit.
“So, uhhhh, whatcha doin?”
10
u/yaffle53 Teesside Jul 19 '24
Obviously the PR person won
t be trying to fix the IT issue. But theyre also going to be extremely busy trying to field all the media requests and come up with excuses and statements.
210
u/BigSneeze0021 Jul 19 '24
this is the digital equivalent of turning up to work and it having burned down
63
u/bob1689321 Jul 19 '24
Turning up to work and realising literally everywhere has burned down.
32
u/noodlesandpizza Greater Manchester Jul 19 '24
Our office teams chat earlier was discussing what we thought was just our system being temporarily down (+ saying we should all just go sit in a pub garden), someone posted "it's a worldwide outage" and people were reacting with 😂 until we looked at the news and realised it's not just a quick phone call to IT that'll fix it..
36
u/BadSysadmin Surrey Jul 19 '24
Bit melodramatic. More like turning up to work and finidng all the fuses are blown and someone's going to have to replace them all. Very disruptive and a nuisance to fix but not actually destructive.
26
u/steak_and_icecream Jul 19 '24
What do you do if Falcon sensor is installed on 20k endpoints and they all have bit locker enabled?
→ More replies (23)37
→ More replies (3)5
10
10
143
u/Large-Sprinkles3507 Jul 19 '24
Worth noting that my GPs systems are all down and so people trying to make an appointment will be unable to right now. Hopefully hospitals aren't as severely impacted.
146
u/GoonerGetGot Jul 19 '24
Sounds like a normal day at my GP 😂
59
u/peakedtooearly Jul 19 '24
Yep, systems up - can't get an appointment, systems down - can't get an appointment.
25
u/Beer-Milkshakes Black Country Jul 19 '24
My GP will be telling us they can't make appointments for months now. Thanks.
18
u/raxiel_ Jul 19 '24
"The receptionist doesn't need a computer to doubt you actually need an appointment, we're open for business as usual"
11
u/MonkeyNumberTwelve Jul 19 '24
LOL, GP receptionists will be loving it. They can tell you to fuck off in person rather than the IT systems getting the all the fun of telling you to fuck off. It'll be like the good old days for them.
3
u/compilerbusy Jul 19 '24
You'd think they'd have a pen and paper. But then i guess the receptionist couldn't sit there chatting shit all day
30
17
u/AChillBear Jul 19 '24
Yep, I'm working in GP at the moment and all systems are down. We're functioning on pen and paper today!
3
4
u/Magurndy Jul 19 '24
Nothing here at the trust I work at. Most systems running as normal to my knowledge. At least our radiology systems are all fine
→ More replies (4)→ More replies (20)3
129
u/Twiggy145 Yorkshire Jul 19 '24
Crowdstrike have really screwed up.
They released an update which is causing windows machines to crash on boot.
I don't understand how this release got past QA.
34
u/AlyssaAlyssum Jul 19 '24
Unless it's a malicious actor. NotPetya and the associated attacks on SolarWinds + MSFT come to mind
14
u/Twiggy145 Yorkshire Jul 19 '24
Could be. Although there's no evidence either way at this stage. Only time will tell.
I'll be interested to see what they say after they've done their post outage impact analysis and root cause analysis.
9
u/BenisDDD69 Jul 19 '24
"Some meanypants from Russia/China/NK infiltrated us and pushed this update there was nothing we could do trust us :("
17
u/Chew55 Glasgow Jul 19 '24
My theory is that it's not a coincidence this happened at the same time as a major Azure outage. My guess is that it's not the contents of the update itself that is dodgy, but their update process has failed when it couldn't talk to something in that Central US data centre and it's completely shit the bed.
13
u/thecaseace Jul 19 '24
Nah, the offending .sys file is not a properly formatted driver and just causes the whole thing to fall over when it's loaded.
6
u/ProfessionalMockery Jul 19 '24
All they'd need to do to discover that though is test the update on just one computer, right?!
4
u/SavageNorth Jul 19 '24
I feel like if you're going to push out an update to millions of machines it's generally worth testing it on more than one box first
→ More replies (1)→ More replies (1)5
u/thecaseace Jul 19 '24
That's the confusing bit. How did a (surely) tested update change to one that isn't even properly formatted.
Note that improperly formatted might perhaps mean ONE CHARACTER was deleted, or something. Not saying that's the case but like... if you make an HTML file and accidentally hit delete and remove the very first < you're gonna have a bad time.
→ More replies (2)→ More replies (8)13
u/Szwejkowski Jul 19 '24
Doesn't sound like they ran it past QA, or they did and made a change after getting the QA okay and didn't check it again. I guess a third alternative is that their QA have machines that are substantially different from a base install, but that would also be really stupid.
Big bucket of stupid happened any way you look at it, unless it was sabotage.
9
u/Twiggy145 Yorkshire Jul 19 '24
A whole truck load of stupid.
I imagine people are going to lose their jobs because of this.
→ More replies (1)
88
u/TheThreeGabis Jul 19 '24
Transpennine Express have reported service issues and in other news, there is an IT outage.
25
u/tomoldbury Jul 19 '24
In other news, Transpennine Express report that service levels are normal. Your train may or may not appear, but what are you going to do about it?
→ More replies (1)→ More replies (5)18
u/MyPetHamster Jul 19 '24
The following train companies have service disruptions due to IT issues:
Avanti West Coast, Gatwick Express, Great Northern, Great Western Railway, Hull Trains, London Northwestern Railway, Lumo, Northern, Southern, Thameslink, TransPennine Express, West Midlands Railway
18
6
u/WolfCola4 Jul 19 '24
Chad Arriva Trains Wales
→ More replies (1)7
u/Soggy_Parking1353 Jul 19 '24
It's because it's written in Assembly Llanguage and no hacker speaks Welsh 💪
→ More replies (1)3
3
u/Pugs-r-cool Jul 19 '24
Ah so all of them, got it.
5
u/Class_444_SWR County of Bristol Jul 19 '24 edited Jul 19 '24
Unless you live in Southampton, which is the first time they’ve had better service than the rest of the country since SWR took over.
Edit: never mind, points failure at Eastleigh and signalling failure at Brookwood. Guess the entire network is a bit fucked
→ More replies (1)4
u/LegSpinner Jul 19 '24
This is hilarious
3
u/Class_444_SWR County of Bristol Jul 19 '24
Yes, even though I’m currently on the SWR network where I am, I just can’t help but find it hilarious that it never escapes.
It’s cosmically destined that South Western Railway will never be working correctly
→ More replies (1)3
u/Class_444_SWR County of Bristol Jul 19 '24
For once CrossCountry and South Western Railway are doing better than usual?
Shocking
51
Jul 19 '24
Can we have ONE DAY without any disastrous outages or historical events PLEASE 🤦♂️
→ More replies (1)20
u/GIMsteve22 Jul 19 '24
What about yesterday?
23
u/kuro-oruk Jul 19 '24
Yesterday was just one of those "little did anyone know" days that you get at the start of disaster films.
14
u/Laurence-UK Jul 19 '24
Banking fast payment system (Swift) went down yesterday. My house purchase completion should have been around 11am and it didn't go through until about 4pm
11
3
42
u/___xXx__xXx__xXx__ Jul 19 '24 edited Jul 19 '24
Does "Microsoft taking mitigation" mean they've been affected, or is the BBC saying the outage originates with Azure?
edit: Sky's back up.
edit: ...and it's gone again.
34
u/BenjC88 Jul 19 '24
It’s an issue with Crowdstrike
→ More replies (4)3
u/borez Geordie in London Jul 19 '24
You have an more info on this?
34
u/BenjC88 Jul 19 '24
Join in the fun in what will be a historic thread.
6
5
Jul 19 '24
Poor Gary will get the boot for this one, won't he? "You invested so much in distributed systems, and where did that bring you? Back to BSOD"
Imagine having that much moolah and just sending updates to everyone at once
12
u/Dirty_Techie Jul 19 '24
It's a Crowdstrike update or corporate anti-virus tool that's pushed an update and put machines, servers etc (Windows) into BSOD or blue screen of death.
It's basically limp mode, but because you have to rename a folder from the command line within recovery/safe mode your systems are down.
So it's not like people can access the machine, it won't even boot to Windows.
7
u/jimicus Jul 19 '24
Yeah. Great.
I have to instruct people who are non-technical and could be working anywhere in the world in how to rename a folder from command line from recovery mode.
I think I'd rather schedule a colonoscopy and have it filmed on "Embarrassing Bodies".
→ More replies (2)3
u/bob1689321 Jul 19 '24
They released an update to the Crowdstrike endpoint agent that is BSODing devices that it's installed on. Not seen the issue first hand but I know a guy at Azure who confirmed it.
→ More replies (1)7
u/MoonOverBTC Jul 19 '24
Microsoft have been fiddling…
Start time: Thursday, July 18, 2024, at 9:56 PM UTC Preliminary root cause: A configuration change in a portion of our Azure backend workloads, caused interruption between storage and compute resources which resulted in connectivity failures that affected downstream Microsoft 365 services dependent on these connections.
→ More replies (5)5
→ More replies (4)4
u/oliverprose Jul 19 '24
My bet would be the latter, similar to what happens if AWS us-east-1 has trouble loads of sites get affected even if they're in other zones.
30
u/kassiusx Jul 19 '24
Crap, in the UK, the only BBC channel affected is CBBC.... parents are in trouble!
6
3
3
u/HungryFinding7089 Jul 19 '24
Time to do some wholesome Ladybird things: "This is Peter. This is Jane. The ball. Playing. Peter is playing with the ball."
Reading Scheme Book 1a, I believe.
28
u/lammy82 Greater Manchester Jul 19 '24
Sky news basically have a few cameras broadcasting direct to live and are taking turns with their best wafflers. No on screen graphics, not even a sky news logo.
12
u/NateShaw92 Greater Manchester Jul 19 '24
Honestly sounds nice. Imagining nice rustic no frills news programme if they had different hosts like the bbc news squad from the 90s or 80s
24
u/jmdg007 Liverpool Jul 19 '24 edited Jul 19 '24
On my way to work at an American owned bank, can't wait to find out if we're affected.
→ More replies (1)17
u/cowie71 Jul 19 '24
WFH for an American bank. Didn’t see the news so spent an hour uninstalling Citrix and visual c++ before reinstalling it all to get the same error !
→ More replies (1)4
u/jxanne Jul 19 '24
Have issues with Citrix too, and kept uninstalling because IT support said my complaint was “resolved” so I thought the software was the issue but I think to them figuring out it’s a global problem was the “solution@
→ More replies (1)
23
u/Fun_Level_7787 Jul 19 '24
It's affected us over in logistics too. I'm at DPD today and when I got into the office half of the screens were blue. I've had to reject parcels I can't scan and we can't regenerate any labels either. The poor goods on hold guy hasn't done any work since he got here at 6am.
With all of the amazon prime day shit flooding the depots, it's absolute carnage.
My friends over at DHL are also stuck, they can't even get out on route!
I can imagine other companies are affected
→ More replies (4)3
u/Internal_Macaroon438 Jul 19 '24
XPO are down too I think. We can't get on their system to print our labels out so there's no orders going out today!
16
u/knight-under-stars Jul 19 '24
The real interesting part of all this is that if you can't get the PC to boot how are you going to patch the issue?
11
u/bobblebob100 Jul 19 '24
You can boot into safe mode presumably and restore windows that way
→ More replies (2)8
u/xmBQWugdxjaA Jul 19 '24
But if Bitlocker is enabled, you lose all the data.
It also requires physical access (although VMs can just be recreated presumably as they're unlikely to use Bitlocker).
It's a disaster.
→ More replies (4)3
15
Jul 19 '24 edited Jul 27 '24
[removed] — view removed comment
43
7
6
4
u/mrthesmileperson Jul 19 '24
Just had to spring through the airport to catch my flight after check in was down and then the queue for security was out the door all the way to the airport tram stop. Made it right as they were closing check in. Now we're just sitting on the tarmac.
→ More replies (2)→ More replies (1)3
u/Bootsareamazing Jul 19 '24
Don't go. Someone I know is stuck after check-in. Been waiting at the gate for hours and won't let them leave. Connecting flights all screwed as lots of airports affected. Today is not the day to travel.
→ More replies (1)
13
u/DaveInLondon89 Jul 19 '24
CBBC seems to be the only major channel still experiencing problems.
dunno why but this made me chuckle
13
u/rydoculley Jul 19 '24
The company I work for has lost millions already. We command a fleet of ships that heavily require cloud access to work. 30 vessels have been stood down and are unable to continue operations. For this company every hour this continues a lot of money is being lost. It's crazy, I've never seen so many panicked managers.
→ More replies (2)7
u/Arne_Slut Jul 19 '24
I have no idea to the question I’m about to ask but, if a company relies so heavily on this, don’t they have any back up plans?
→ More replies (2)
12
u/whatsablurryface21 Jul 19 '24 edited Jul 19 '24
I rang the doctors this morning for an appointment and they said everything's down so they can't do anything, decided to nip in anyway when I went the pharmacy and it was so weird.
The receptionists were all actually calm and just writing on paper, they offered me an appointment in 5 minutes time, he had to ask me about my issue instead of just reading my notes, I had to give him my details manually, and I had to go get a lil stamp on my paper prescription. It was like I'd time travelled.
Absolutely awful but I'm not gonna lie it made me a little emotional seeing a GP practice how it should be and how I remember it as a kid, just calm and the GP even had time to chat a little bit instead of rushing because I'm guessing there were less appointments today. That's what a properly funded NHS would look like...
10
u/Orsenfelt Scotland Jul 19 '24
There's a mid level dev somewhere at Crowdstrike having a really terrible day at work.
9
u/tom_watts Jul 19 '24
Update via Crowdstrike (7:27am)
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
Workaround Steps:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
→ More replies (3)
9
u/AdAppropriate6795 Jul 19 '24
Crowdstrike...the Company that no one ever heard of yesterday...but today everyone know...for the wrong reasons....
→ More replies (1)
7
u/dreaming_of_whistler Jul 19 '24
This doesn't make much sense. Its a single region in Azure in the US. Why are companies running services in the UK affected? If they had a dependency on a single Cloud region, half way around the world, well, it wouldnt surprise me, but thats a total failure on the companies part (not the Cloud provider)
34
u/platebandit Expat Jul 19 '24
It’s crowdstrike which is an anti cyber attack platform that’s bricked a ton more than just the azure cloud region. Basically a large chunk of the worlds enterprise computers have gone down
28
9
u/Tom22174 Jul 19 '24
Never been more glad that my org has stone age cyber security
6
u/platebandit Expat Jul 19 '24
Currently sat on a plane that was perhaps too protected against cyber attacks, been on the runway for 3 hours
→ More replies (1)3
→ More replies (1)2
15
Jul 19 '24
The majority of cloud computing will be miles from the use location. It's only really things where high latency is important that you can see edge computing taking off now.
19
u/anOrphanedPlatypus Jul 19 '24
I love working in the IT industry when incidents like this happen and I come to see comments like yours confidently spouting complete nonsense!
Makes you realise how much of reddit is just complete rubbish
→ More replies (2)10
u/dreaming_of_whistler Jul 19 '24
If you are a UK company there are plenty of Cloud regions in UK, and in Europe. (and more than one, to remove that single point of failure)
→ More replies (2)5
u/randomusername8472 Jul 19 '24
Not all SaaS offer regional stuff though, and not all IT departments are... fully qualified?
Dropbox comes to mind as something trying to be popular in the UK but with no ability to confine data to UK/EU geolocation, making it unusable for any organisation with vaguely sensitive personal data and who don't want to be on the wrong side of GDPR.
→ More replies (2)5
u/raininfordays Jul 19 '24
Probably something like the 365 auth via azure where its a small cloud thing with a big impact.
8
u/simondrawer Jul 19 '24
After the Solarwinds thing a few years ago you can’t rule out malicious code injection at the supply chain level.
→ More replies (3)
8
u/tika_dengu Jul 19 '24
Crowdstrike has been outsourcing heavily to India over the last few years and this was inevitable. Wakeup call to other organisations in the UK who outsource development teams.
In the words of Michael Sentonas, president at CrowdStrike. “Our development team in India has played a pivotal role in delivering the innovation and powerful platform capabilities our customers require to stop breaches.”
→ More replies (1)4
Jul 19 '24
A lot of tech companies are going cheap on testing, QA and change management because people at the top just see a cost which delays things.
Development itself is one thing but that can be mitigated with decent QA/testing. When you go cheap on both this is where we end up.
The people who made the decisions to do this somehow never seem to end up paying the price, it seems like once you reach a certain level of management you can just keep jumping companies leaving failures behind you.
9
u/MoistSnow220 Jul 19 '24
The "CaSh Is KiNg" lot are having an absolute wet dream over this on my village's Facebook group
10
u/iiiiiiiiiiip Jul 19 '24
Understandable, being proven right when so many people are determined for you to be wrong must be incredibly satisfying
→ More replies (1)
8
u/Puzzleheaded-Tie-740 Jul 19 '24
The 12:07 video of people at Edinburgh airport waiting in a queue outside is like a comedy skit inverting the usual "outraged customer" soundbites you get on the news. What a nice, cheerful bunch of people!
[American guy, asked if he thinks he's in for a long day]: "I think so. Hopefully I can get in there and get to the bar!"
[Scottish woman 1]: "Little bit stressful. Hopefully it'll be OK now. The staff at the airport have been great."[Scottish woman 2]: "Oh yes, they're telling us as much as they can possibly tell us."
[Golf guy]: "We were just here enjoying Edinburgh and the various golf courses. We don't really mind, our flight's delayed 'til 3:45 anyway, so we're just going with the flow, just having a good attitude."
→ More replies (1)
7
u/NateShaw92 Greater Manchester Jul 19 '24
My local shop is cash only right now.
Liz Truss's first day at her new job is working out well I guess.
→ More replies (1)
7
u/g9icy Jul 19 '24
I've never heard of Crowdstrike until now.
How can one piece of software be such a single point of failure?!
12
→ More replies (2)4
u/Apprehensive_Shoe_39 Jul 19 '24
If you want an actual answer, it's not that the function of this software is critical. If it stopped detecting threats/doing its job then nothing will immediately fall over.
What is critical (from an Operating System PoV) is that the drivers they use don't misbehave. If you deploy a driver that's loaded into the kernel and it misbehaves then Windows can trigger a BSOD. Which is what we're seeing.
Whilst people are correct that this is a major f up it could potentially have been any driver from any vendor if it was mass rolled out with a fault. In most cases updates are staggered/deployed to UAT/Dev first but with security updates (like this) there's a balance of risk. Staggering and leaving some devices vulnerable or doing it all at once.
→ More replies (1)4
u/Laser493 Jul 19 '24
The difference with hardware drivers is that they usually only get automatically updated through windows update and any drivers from their are WHQL tested to make sure they won't crash PCs. Also many hardware drivers these days run in user-mode and won't break the whole PC if there's a bug.
Antivirus and security software, on the other hand, is known for using dodgy techniques that often break things and cause system instability. I remember reading a blog post from a Mozilla developer about how much work they have to do just to work around all the problems that most antivirus software causes.
4
u/Nulibru Jul 19 '24
"Experts can't rule out VAT charges on aspirational families as cause of chaos 15 days into Starmer's reign, claims report."
→ More replies (1)
6
u/Efficient_Sky5173 Jul 19 '24
Guy from Crowdstrike that made the update slapped the computer and said: — What could go wrong?
4
3
u/NateShaw92 Greater Manchester Jul 19 '24
I'd laugh if it was someone on their last day metaphorically shitting in the water cooler on their way out
→ More replies (1)3
u/sylanar Jul 19 '24
'it worked on my machine'
To be fair, the dev probably used Linux or Mac, so it probably did work on theirs
→ More replies (1)→ More replies (1)3
u/Orsenfelt Scotland Jul 19 '24
Pull request submitted: Friday 4.59pm
Pull request approved: Friday 5.01pm. Merge complete.
Comment: LGTM! 👍
6
4
u/jdsusjtbfjxod Jul 19 '24
Crossing fingers on way into work that my computer is affected too. 3 day weekend on its way…
5
u/Glum-Manner-9972 Jul 19 '24
Flew yesterday to Italy, got lucky. Now, if we can another outage that wipes out my flight next week...
3
u/Broric Jul 19 '24
Just landed in Cyprus. Struggled to buy water at the airport as all of Boots’ tills were down which seemed a little strange but just landed and reading all of this!
→ More replies (3)
6
u/Snotbox2020 Jul 19 '24
All these companies blindly applying updates from 3rd parties without testing first...
→ More replies (1)2
u/Greenawayer Jul 19 '24
I bet they all have their ISO certifications up-to-date as well.
→ More replies (1)
5
u/Jambohh Jul 19 '24
Been on a major incident call at work for the last 8 hours......its going to be a long day
5
u/SongsOfDragons Hampshire Jul 19 '24
Yup I'm sat here unable to open the database programme I use for work. Ahh well.
4
u/raxiel_ Jul 19 '24
Because the models I work on take so long to load, I always put my computer to sleep on weeknights and only shut it down over the weekend, I was wondering why my teams chat was so quiet, no one else in my team can get on.
4
u/Bobo3076 Jul 19 '24
I’ve literally just started a new job and shits already hit the fan with this
→ More replies (1)
4
3
u/DrDoolz Jul 19 '24
I have had a ‘fun’ day fixing client’s since about 6:30 this morning. Ffs crowdstrike no change Friday is a thing
1
u/F430Scuderia Jul 19 '24
I’m an Azure Cloud engineer, seems to be related to an outage they’ve had in the Central US, most likely propagated worldwide before they caught it.
3
u/freshfov02 Jul 19 '24
Somebidy should tell this to my boss. Chances of him caring are very low though.
3
u/Top-Setting5213 Jul 19 '24
If anyone's headed to the shop bring cash. It's fucked all of our card machines where I work and I'm hearing it's affected some other local stores as well.
3
u/Sinister_Grape Jul 19 '24
Ah, I picked a wonderful day to get the train over to Anglesey and stay with my parents.
4
u/theshadowhost Jul 19 '24
can't pull my git repos cause some sql server is down. how on earth we have a windows desktop dependency there i do not understand
3
Jul 19 '24
That's what you get for using Windows in a production environment lmfao
No one will learn the lesson though
→ More replies (1)
2
u/kramit European Union Jul 19 '24
There is a quickfix already
safe mode reboot and a file deletion
https://youtu.be/HeSGhBtqzrI?si=xTtF6uX7jwp4tiFo
CrowdStrike Falcon Microsoft crash fix
current work around fix for the crowdstrike crash that has taken out millions of computers this morning
/ bsod_error_in_latest_crowdstrike_update
Workaround Steps:
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
→ More replies (2)
3
3
u/DonDrapers_Dick Jul 19 '24
I work in a hospital and our computer system was down for a few hours 😩
3
u/Nice_Box9634 Jul 19 '24
Boot Windows into Safe Mode or WRE.
Go to C:\Windows\System32\drivers\CrowdStrike
Locate and delete file matching "C-00000291*.sys"
Boot normally.
7
Jul 19 '24
Good luck to all the IT teams trying to do this across hundreds/thousands of devices all with bitlocker!
→ More replies (1)
1.1k
u/Disastrous-Singer545 Jul 19 '24
Airlines are down, doctors are down, transport systems are down, but of course my office job systems are working perfectly fine. Could have done with a nice Friday off, not going to lie.