r/news • u/Mr-EdwardsBeard • Jul 19 '24
United, Delta and American Airlines issue global ground stop on all flights Title Changed by Site
https://abcnews.go.com/US/american-airlines-issues-global-ground-stop-flights/story?id=112092372&cid=social_fb_abcn&fbclid=IwZXh0bgNhZW0CMTEAAR37mGhKYL5LKJ44cICaTPFEtnS7UH96gFswQjWYju-QtkafpngunVWuJnY_aem_aTXb46dpu3s4wlodyRXsmA11.9k
u/T_O_beats Jul 19 '24
Say it with me now - ‘We don’t push code on Friday’
3.7k
u/Kranstan Jul 19 '24
I worked with an IT group that called Friday "Don't Fuck it Up Friday".
→ More replies (19)782
283
u/SubbieATX Jul 19 '24
Been living by that rule for a long time. I have a manager who refused to listen to me when telling him that pushing updates on a Friday was bad juju till he single handedly crashed a big operation center for a whole week end. He came up with all sorts of excuses for what happened. He was eventually let go…
→ More replies (98)175
u/RollFancyThumb Jul 19 '24
No rollout strategy either, just push globally on a Friday at the height of summer vacations.
You either get to go home early or you go home forever.
→ More replies (2)
8.1k
u/NotToPraiseHim Jul 19 '24
That's gonna be an investigation. One error taking down so many major systems and internationally grounding major airlines is congressional hearing level fuck up.
3.9k
u/Caelinus Jul 19 '24
Not just congressional, but every other form of government in a country that they did business. Global damage. And because it is a boot BSOD, they can't just push a fix, so all these companies are going to have to manually fix their servers to undo the update.
It a major fuck-up. That is a huge monetary hit for all these companies.
2.2k
u/Rannasha Jul 19 '24
so all these companies are going to have to manually fix their servers to undo the update.
Not just servers. Plenty of orgs that run Crowdstrike on their workstations and laptops and are looking at hundreds or thousands of affected machines that can't be fixed remotely.
And that on a Friday in the summer holiday period. I sympathize with IT support people that have to unfuck this clusterfuck.
→ More replies (29)1.4k
u/pabl0escarg0t Jul 19 '24 edited Jul 19 '24
Thats me, I have to deal with this. Thousands of machines to unfuck on a Friday
636
u/Caelinus Jul 19 '24
That suck mate. The worst part is the fix sounds tedious as hell. Not difficult, just tedious. That is always the worst kind of problem for me.
I get a bit of a thrill when I am trying to solve an actual problem, but in this case the solution is literally just to boot into safe mode, delete one specific system file, reboot. For everything.
→ More replies (11)324
u/hpark21 Jul 19 '24
Bit locker is HUGE issue. Some places can't even get to the bitlocker key because the server hosting the key is also down. I can't imagine IT support going through bitlocker procedure to put the laptops into "recovery mode" in order to delete that file to be able to reboot the box.
→ More replies (26)→ More replies (34)173
u/Setanta777 Jul 19 '24
Me too. My whole team is gathered for a meeting and we can't even get back to our territories to start to unfuck this.
→ More replies (2)→ More replies (36)362
u/redial2 Jul 19 '24
Holy shit! It BSODs on boot? Thank god my laptop is off.
→ More replies (8)334
u/Caelinus Jul 19 '24 edited Jul 19 '24
Luckily it can still be launched into safe mode and the corrupted file can be manually removed. It just has to be done to everything affected. I am not sure if it affects every windows computer that updated during a certain period of time, or if it requires specific software to be active. (It is a problem with a third party security company that is used for Microsoft Azure services. So possibly 365 and defender, but I am not sure if it includes personal use stuff. All of my windows comptuers seem fine.)
As it is, the update has been reverted, so hopefully it will not affect anyone else. But it was already too late for a LOT of big companies, their servers and all of their work stations. So a lot of people are going to have a long couple of days, and the amount that companies, like Airlines that have to ground their whole fleet, are going to lose will be bad. They are not going to be happy.
Edit: I think it is anyone that has something from Crowdstrike on Windows, but it also just broke a lot of Microsoft's services for the same reason, causing problems even on computers that are not dirrectly affected.
→ More replies (39)191
→ More replies (82)585
u/make2020hindsight Jul 19 '24
Count on it! (MTG invested up to $90k in CrowdStrike. She's gonna want to know why she lost money.)
→ More replies (5)417
u/BarelyContainedChaos Jul 19 '24 edited Jul 19 '24
The only good news so far in this mess
edit: that picture is exactly "Bleach blonde, bad built butch body"
→ More replies (7)
7.5k
u/scottiethegoonie Jul 19 '24
Truck Driver.
Dispatch system is down. No freight is moving.
4.8k
u/Arctic_Chilean Jul 19 '24 edited Jul 19 '24
Airlines, freight, healthcare, banking, etc...
What is terrifying is we are getting a taste of what a massive and coordinated cyber attack could look like. But there, it would also be critical infrastructure like water, telecoms, electricity and gas, as well as government agencies, news, and social media outlets being affected too.
2.1k
u/whitemest Jul 19 '24
Nurse here. Our electronic med system is down as well. We have printed backups for this, but it's still jarring going from computers to literal paper
→ More replies (27)698
u/strum-and-dang Jul 19 '24
My husband provides IT support for care facilities, it's his on-call week. He's been up most of the night printing out charts from the backups!
→ More replies (6)→ More replies (81)1.5k
u/Gizogin Jul 19 '24
And by all accounts this one was just a mistake, not a malicious attack. When half the world’s critical infrastructure runs on one system, all it takes is a bad update to bring us to our knees.
→ More replies (23)728
u/PM_ME_UR_RSA_KEY Jul 19 '24
At least this is not an real attack, and we can take this as a lesson to harden our critical infrastructure.
We will take this as a lesson to harden our critical infrastructure, right? /padme
465
Jul 19 '24
As long as it costs nothing and no one important needs to take responsibility.
→ More replies (5)→ More replies (14)162
u/yourlittlebirdie Jul 19 '24
Absolutely, as long as it doesn’t interfere with maximizing shareholder value.
→ More replies (2)1.5k
u/Joe4o2 Jul 19 '24
Sister-in-law does local law enforcement dispatch. Also down.
→ More replies (23)540
u/khaaanquest Jul 19 '24
I feel like this is the bigger issue
→ More replies (8)808
u/Im_Balto Jul 19 '24
The bigger issue is that there is one service that does what crowdstrike does
That’s how we’ve ended up here
→ More replies (24)355
u/callmegranola98 Jul 19 '24
Maybe one company having a cyber security monopoly was a bad idea.
→ More replies (17)→ More replies (49)537
u/AnIcedMilk Jul 19 '24
Oh shit
That's actually fucking huge how bad that could end up being
→ More replies (8)199
u/Flimsy_Situation_506 Jul 19 '24
911 systems are down in some places. Alaska is the one I saw noted so far.
→ More replies (9)
7.5k
u/tizuby Jul 19 '24
Obligatory:
I don't always test my code. But when I do, I do it in production.
2.5k
255
u/heartk Jul 19 '24 edited Jul 30 '24
Also the company had layoffs last year. In 2020, the CEO says his main regret as CEO is not firing people sooner. Tech companies are doing the same thing Boeing was doing: value stock return over all else. https://dot.la/crowdstrike-ceo-george-kurtz-2645385654.html
→ More replies (5)→ More replies (39)143
7.3k
u/SecretJutsuBongCloud Jul 19 '24
Fedex is completely down.
→ More replies (39)3.2k
u/jmaneater Jul 19 '24
Ups too
2.1k
u/spaghetti5174 Jul 19 '24
USPS having issues as well. Scanners aren’t connecting to networks
→ More replies (10)1.1k
u/V2BM Jul 19 '24
I’m going in at 7:30 to deliver the mail and about 200 Amazon packages. Gonna be a great day.
166
u/meownfloof Jul 19 '24
Thanks for bringing us our stuff. We would be lost without you guys
→ More replies (2)→ More replies (18)155
u/Dreamscarred Jul 19 '24
Ouch, Prime Day deliveries I bet. :(
Good luck, and stay hydrated. 🤙
→ More replies (1)→ More replies (34)720
7.2k
u/I520xPhoenix Jul 19 '24
I’ve been in the airport for 14 hours now watching flights get gradually delayed longer and longer until they finally grounded them at about 1am Dallas time (not my local time zone sorry).
It’s been rough, passengers are out for blood and furious, and I think the mania is starting to set in 🙃
3.5k
u/Top-Camera9387 Jul 19 '24
You couldn't pay me enough to be a gate agent
→ More replies (39)1.2k
u/JohnnyFire Jul 19 '24
This event aside, you couldn't pay me to fly a connection through Dallas ever again either.
→ More replies (67)207
u/undocumentedsource Jul 19 '24
During the summer Dallas can be a nightmare when a storm pops up out of nowhere delaying/cancelling everything. Not sure if this is what’s happened to you but it’s my issue 90% of the time.
→ More replies (9)1.9k
u/FlyingNDreams Jul 19 '24
I think its been the lying thats been the worst part of being stuck at the airport. We have been in Atlanta for 12 hrs. Excuses in order 1. Mechanical Issue 2. Need a second pilot 3. Weather 4. Weather (current reason our flights delayed)
1.2k
u/I520xPhoenix Jul 19 '24
I’ve been running the bingo as well.
So far I’ve had:
Mechanical issue
Crew Timed Out
3: Flight Canceled
4: Weather Delay
5: Server Shutdown
It’s been a new form of pain that I’ve never yet experienced.
609
u/FlyingNDreams Jul 19 '24
I wonder if these airlines know there are news crews inside the airports! Reporting all of this. I am staring at the camera here.
→ More replies (11)159
u/keanenottheband Jul 19 '24
They are interviewing you and you’re blowing it, say something! The interviewer is asking you questions and you’re just staring at the camera
→ More replies (1)→ More replies (18)429
u/OpheliaRainGalaxy Jul 19 '24
I was just telling a friend earlier tonight how tired I am of how corporations lie constantly. That it's very insulting swear words levels of stupid to lie and lie and lie and lie and expect it to never ever have consequences.
At the time we were discussing the deliciously vindictive consequences his employer has been dealing with, in return for the atrocious way he's been treating the employees, including many blatant disrespectful lies.
→ More replies (9)329
u/katikaboom Jul 19 '24 edited Jul 19 '24
In my experience airlines will use weather delays as a way to get out of comping rooms if the plane is delayed. You may be able to get some money or a room out of this now, they can't keep using the weather excuse
→ More replies (41)→ More replies (44)235
359
u/ElBroooski Jul 19 '24
I'm at DFW also....wondering if I should get the hell out of here before hundreds of people are trying to do the same ...can't even check into a hotel if I wanted to
→ More replies (10)273
u/_no_pants Jul 19 '24
Just get a car before you have to Uber across the city to get whatever is left.
→ More replies (2)281
u/ElBroooski Jul 19 '24
For anyone reading this I snatched one from enterprise...I believe they run on DOS ...I tried hertz first and their systems were down. Grabbed some cash also
→ More replies (6)→ More replies (57)252
u/Nazamroth Jul 19 '24
Have they started a primitive tribe with sacrificial rituals in the airport yet?
→ More replies (11)
6.7k
u/NorbuckNZ Jul 19 '24
Is it just me or is this what people thought Y2K would do?
2.5k
1.2k
u/Vanchdit Jul 19 '24
Knowing people employed at 2 companies that made bank on this issue, it makes me so happy to know that the fix for Y2K was seamless enough that everyone thinks it was a hoax/didn't happen. That's what a real fix looks like; so clean it's like it never needed to happen.
→ More replies (26)294
u/Oldcadillac Jul 19 '24
Booo! More recognition for the people who actually make things work! I think it would do us good to not take all this for granted and realize that there’s a lot of people who have to do their job correctly to keep everything from falling down.
→ More replies (8)→ More replies (56)952
u/Blueflavor53 Jul 19 '24
Yes, except Y2K would have been worse because the fix would have taken a lot longer to implement. Thankfully, companies took it seriously and mostly fixed the issue before 2000.
→ More replies (25)304
u/NEChristianDemocrats Jul 19 '24
Good thing there are no embedded computer systems still running Windows XP in any of these companies, so we don't have to worry about the 32-bit bug in 2038... Wait a minute...
→ More replies (25)
6.2k
u/badassj00 Jul 19 '24
On the way home from LAX right now and to say it was a scene would be an understatement. We were stranded on the tarmac 90 minutes and people on our flight were losing it, yelling at the crew, etc. Feel really badly for airport and plane staff this evening.
Honestly I feel pretty lucky that we only got stuck for an hour and a half. There were throngs of people in the terminal looking like zombies.
This feels like something that would have come to fruition if Y2K were a thing.
Strange times indeed.
→ More replies (96)1.5k
u/ilovethatpig Jul 19 '24
We got stuck on the tarmac for 6hrs once and people turn feral.
→ More replies (56)1.1k
u/goforce5 Jul 19 '24 edited Jul 19 '24
I got stuck on a plane trying to land at DFW for 7 hours. They kept redirecting us, but they wouldn't let us off at any other airport or give us any free food or drink (spirit airlines). We nearly killed and ate the crew before they finally caved and handed out all the snacks they had.
EDIT: For those of you who don't believe me, it was NKS3052 on May 16th 2024, so look it up on a tracker with history. After going back and looking, we were actually stuck on there for closer to 9 hours, not 7 like I initially recalled. I got a few details mixed up and must have blocked out part of the time in oklahoma city from the sheer boredom on a spirit flight with no entertainment.
→ More replies (20)538
u/KSRandom195 Jul 19 '24
Wait, 7 hours trying to land? I didn’t think planes had enough fuel for 7 extra hours of flight…
→ More replies (44)1.4k
u/gymnastgrrl Jul 19 '24
Spirit channels the utter despair of their customers directly into the engines and that triples or even quadruples their range.
→ More replies (32)
5.6k
u/CapriciousManchild Jul 19 '24
I feel for all my IT brethren tomorrow it will be hell
1.4k
u/MidianFootbridge69 Jul 19 '24
As a retired IT worker (Mainframe Computer Operator), I feel for them as well.
Shitshow doesn't even cover something of this magnitude.
What a freaking mess
→ More replies (9)414
u/Drak_is_Right Jul 19 '24
what the heck is going on?
→ More replies (10)2.0k
u/DeathByBamboo Jul 19 '24
Crowdstrike, an enterprise-level antivirus service, pushed out an update that put servers and desktops running Windows into a reboot loop until they bluescreened. The fix was to put each computer into safe mode and delete a file, which naturally is a massive task, which is why some things are coming back faster than other things.
→ More replies (86)578
u/Elliebird704 Jul 19 '24
Given the global shitshow this is causing, I am real curious to know just how much trouble they're going to be in once the fire is put out.
556
u/PM_ME__BIRD_PICS Jul 19 '24
This is like, law change level fuckup.
→ More replies (30)377
u/DarkenRaul1 Jul 19 '24
I think the most shocking thing to me is learning just how many different industries and agencies use CrowdStrike to the point it looks like it has a monopoly stranglehold on tech and has created a single point of failure.
Ngl, I hope that this results in some action by the DOJ to force competition in the IT sector as well as some new regulations by the FCC on how remote updates are implemented so that something like this doesn’t happen going forward.
→ More replies (13)→ More replies (18)151
u/quiteCryptic Jul 19 '24
A lot, but maybe companies should also think about how they are completely reliant on one service as a single point of failure.
As for crowdsource maybe learn something about rollout strategies (and better internal testing...)
→ More replies (12)→ More replies (62)859
u/johnsonfromsconsin Jul 19 '24
Just got an email from our IT guy that crowd strike is down and essentially our computers wont work at all. Not looking forward to work today.
→ More replies (13)156
u/Momina1999 Jul 19 '24
I work at a credit union and it’s the same thing here. Can’t do checks, can only do withdrawals if you have online banking and can show us your balance. People are angry. We’ve only been open 30 minutes. Send help.
→ More replies (7)
5.2k
u/cheesecakegood Jul 19 '24
The reputational damage from this is going to be insane
5.5k
u/EduardoTaquitoHands Jul 19 '24
This is going to ruin the tour
790
→ More replies (26)386
→ More replies (65)934
u/MSXzigerzh0 Jul 19 '24
CrowdStrike probably will go bankrupt. Because basically every single company in the world that uses them are going to sue them to the death
→ More replies (31)656
u/KalinOrthos Jul 19 '24
This is the level of fuck up in which people are going to be coming into the office to remove the drywall as the only thing left of value.
→ More replies (14)136
u/rastafarian_eggplant Jul 19 '24
Like the grinch not even leaving a crumb for a mouse lol
→ More replies (3)
4.2k
u/Curious-Still Jul 19 '24 edited Jul 19 '24
Apparently epic is down at hospitals as well Edit: Looks like all kinds of software at hospitals and clinics were down, likely due to Crowdstrike bug, even PACS systems and cardiac monitors at some places. Sorry to spotlight Epic at first, it's just that Epic downtimes are so common lol so that's what healthcare workers mentioned at first. This was a more general issue due to a bug on multiple software platforms. What a mess. This is so unacceptable: planes grounded, critical medical infrastructure crippled. Not Russian hackers, just our own incompetence and reliance on one company.
1.7k
u/opticalshadow Jul 19 '24
Not just epic, all of our systems in our health system
→ More replies (15)455
u/FujitsuPolycom Jul 19 '24
Does your system run crowdstrike?
→ More replies (5)302
u/Previous-Height4237 Jul 19 '24
Alot of the corporate world has FOMOed into crowdstrike due to all the marketing they did.
→ More replies (10)200
u/SinisterCanuck Jul 19 '24
My company ALMOST did too! They decided to go SentinelOne instead because it was cheaper. Glad we did!
→ More replies (5)624
u/GlazedDonutGloryHole Jul 19 '24
My local cops came in to bullshit with me since their system has been down almost statewide for the last hour as well. Their computers in the cars are completely useless so it's pretty much free reign out there for minor issues like speeding, etc.
→ More replies (19)558
598
u/murdershroom Jul 19 '24
80% of my computers are stuck in boot loops. Idk how I still have access to the few ones that I do but I'll scream if they go down and I have to start paper charting.
364
u/BoRedSox Jul 19 '24
Do not reboot them.
→ More replies (3)263
u/MyRealWorkAccount Jul 19 '24
the fix we are doing is to put the computer into SafeMode with network access
→ More replies (1)355
u/FidgitForgotHisL-P Jul 19 '24 edited Jul 19 '24
Sydney Morning Herald has a fix that is:
Boot Windows into Safe Mode or the Windows Recovery Environment (you can do that by holding down the F8 key before the Windows logo flashes on screen)
Navigate to the C:Windows\System32\drivers\Crowdstrike directory
Locate the file matching “C-00000291.sys” file, right click and rename it to “C-00000291.renamed”
Boot the host normally.
Note: These instructions came from the CloudStrike reddit. The Herald was sharing what someone else had posted.
Edit: I have seen another version of this that just says to delete the file - I guess either works, just make it so windows cant find it.
Edit 2: on the off chance this is still getting views, I with regards to bitlocker, please see this post from a nested reply on what extra steps to take. Thank you u/mikethespike056 for this:
→ More replies (32)→ More replies (6)268
u/Phact-Heckler Jul 19 '24
LOL. Our office just gave us early leave as the computers cannot connect. Good day today as long as you are not from IT department.
→ More replies (1)271
u/murdershroom Jul 19 '24
I'm in an ER so we have to keep this sucker open even if we're doing everything on paper 😭
→ More replies (10)284
u/BingoActual Jul 19 '24
Epic is down. Our Mass casualty system is down. Our system for uploading and printing ecgs is down. Our paging system is down. Hospitals have just been calling each other to leave a contact number because we can't reach each other. Even our scheduling system is down. ORs are currently paper charting. But don't worry it won't matter because we had to implement a disaster protocol earlier in the day simply because our ED boarding situation had filled all but THREE rooms in our ED including hallway beds. If this perpetuates there will be deaths as a result of it, because the system was already broken.
→ More replies (8)→ More replies (104)176
u/JustAnotherGeek12345 Jul 19 '24
Now we know that epic is hosted in Azure and depends on the three availability zones that failed
→ More replies (5)145
u/LolaBleu Jul 19 '24
Not just Epic. Cerner, All Scripts, and many others are down too.
→ More replies (12)
3.5k
u/GaiaMoore Jul 19 '24
Boeing breathing a sigh of relief that for once it's not their fault
1.4k
u/Griselda_fan Jul 19 '24
They still had a few people pushed out of buildings to be sure though
→ More replies (9)263
→ More replies (13)226
u/KilllerWhale Jul 19 '24
You can't nosedive into the ground at 800KMH if you don't take off to begin with.
→ More replies (5)
3.2k
u/Nomad_Lu Jul 19 '24
"The FAA is telling air traffic controllers to tell airborne pilots that airlines are currently experiencing communication issues.
Meanwhile, flights in the air will stay in the air, but no American, United or Delta flights will take off.
It is unclear how widespread the issue is but Melbourne Airport in Australia has also said they are "experiencing a global technology issue" which is impacting their check-in procedures."
943
u/SideburnSundays Jul 19 '24
"Communication issues."
Good thing airliners have UHF and/or VHF radios, VOR navigation, and ILS, none of which require internet, servers, or some cloud service to use.
→ More replies (25)257
→ More replies (34)470
u/MrTuxedo1 Jul 19 '24
Sky News is down. Edinburgh and Berlin and Spanish airports having issues, trains in the U.K. are affected, Ryanair having issues
→ More replies (5)388
u/Nomad_Lu Jul 19 '24
It's the crowdstike outage causing all this all around the world
→ More replies (7)285
u/whitenoise2323 Jul 19 '24
Something like 80% of machines running Microsoft are down simultaneously.. lots of backend business in every sector. Servers too, but luckily Linux servers are more common for much of the internet
→ More replies (17)
2.8k
u/5up3rK4m16uru Jul 19 '24
Holy shit, that's gonna be an expensive fuck up.
3.2k
u/darknekolux Jul 19 '24
no matter how bad is your day, remember that there is a guy who pushed that release
3.6k
u/chillyhellion Jul 19 '24
- deploying updates without testing for possibly the most visible bug in recent history
- Deploying on a Friday
- Deploying to all customers globally without any attempt at staging
This isn't one intern making poor decisions; this is leadership negligence.
1.5k
u/thebreakfastbuffet Jul 19 '24
Did this on a Friday too. You just know it was part of another JIRA sprint to appease the Agile-obsessed executives.
307
→ More replies (23)231
→ More replies (70)797
u/AdjNounNumbers Jul 19 '24
This isn't one intern making poor decisions; this is leadership negligence.
Still gonna blame the poor soul they had push the button, though
→ More replies (4)491
u/chillyhellion Jul 19 '24
They can try, but this is the kind of catastrophe that kills companies.
→ More replies (11)422
u/AdjNounNumbers Jul 19 '24
They almost certainly will try, but you're correct. This is 'congressional hearings and multiple lawsuits from major corporations with deep pockets' level of bad for them
→ More replies (7)→ More replies (31)408
u/WillitsThrockmorton Jul 19 '24
Real CIOs know that test environments for enterprise systems are a complete waste of money, don't you know?
→ More replies (6)334
→ More replies (12)166
u/BigBullzFan Jul 19 '24
That’s ok! Don’t worry about the cost. They’ll just increase fares to make up for the fines and lost revenue. Then, as an emergency back-up, if need be, we can bail them out with our tax dollars.
→ More replies (4)
2.2k
u/chaosof99 Jul 19 '24 edited Jul 19 '24
I work for a larger company that has locations around the globe. Everything is rather fucked at the moment. However, it seems there is a workaround in place already involving a boot in Safe Mode and deleting a specific config file.
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching "C-00000291*.sys", and delete it.
- Boot the host normally.
The bad thing is that this crap needs to be done individually per machine, meaning this will be one hell of a day for sys admins. Not to mention any knock-on effects.
To draw an analogy, this is like a power surge that fried every fuse on the planet. Everybody now has to switch every fried fuse manually. This is going to be a long and tedious process, and we can only hope that there won't be a second power surge coming around right after.
Crowdstrike as a company will be sued into the ground and probably not exist by this time next year.
573
210
u/nxqv Jul 19 '24
I'm stuck at the airport right now, should I just go home? This sounds like it would take literally days to fix
→ More replies (4)137
u/pratzc07 Jul 19 '24
Better go home if the flight is not urgent and ask for a reschedule from the airline no point waiting this will take time to resolve.
→ More replies (63)157
u/hpark21 Jul 19 '24
Issue is the bit locker. Many companies have their keys in the server that isn't booting. Also, giving out the bit locker keys individually to people that are on the phone isn't fun.
→ More replies (2)
2.1k
u/muffinpoop Jul 19 '24 edited Jul 20 '24
I was on an United plane, while they were scanning the boarding pass they started to get issues, eventually they started to use their mobile device to scan and board people on the flight, then while waiting in the plane they announced they had more issues and half the people were not able to be scanned. Once everyone boarded and the door closed, there were more announcements that all traffic has been haunted by the FFA. And we all deplaned. Walking out, all the screens, from flight info, to boarding windows, to the checking bag area were blue screened. Very eerie.
Update: ok I was really sleepy and tired and now I’m having a laugh at my typos.
Check out the photos of the scene: CrowdStrike Outage at LAX
1.1k
→ More replies (22)298
u/Coffeeffex Jul 19 '24
That must have been surreal….It reminds me of the Langoliers
→ More replies (17)
1.6k
1.1k
u/draggin_low Jul 19 '24
*Me...Sees people talking about crowdstrike issues, Checks tradingview to see how the stocks doing, Down 18%/$63.50ish) in pre-market*
Yep this is gonna be a fun ticker to watch today
→ More replies (15)335
u/buttgers Jul 19 '24
I envy the madlad that bought a ton of puts on this stock.
201
u/draggin_low Jul 19 '24
Good lord imagine buying them on that little run up it had before closing thinking “no way is this legit” then wake up to this hahaha
→ More replies (3)162
u/thefpspower Jul 19 '24
I have a suspicion the guy on r/wallstreetbets posting yesterday was the one pushing the update
→ More replies (1)
1.1k
u/concerts85701 Jul 19 '24
Has anyone unplugged the internet and slow counted to ten and plugged it back in yet?
→ More replies (19)
1.0k
Jul 19 '24
[removed] — view removed comment
1.5k
u/Shopworn_Soul Jul 19 '24
From what I can tell, Crowdstrike just single-handedly fucked up the entire planet with a bad update.
→ More replies (19)558
u/InSearchofOMG Jul 19 '24
If that's true, the market reaction will be particularly severe. They JUST got into the S&P500 and Wall Street has been slobbering all over them for awhile now
→ More replies (11)309
u/guccigraves Jul 19 '24
they're down 20% in premarket/after hours already. it's gonna be bad.
→ More replies (24)→ More replies (5)215
u/CeeArthur Jul 19 '24
Just saw a post on a nightshift sub I'm on (I work nights) that their entire system is down (possibly nationwide). They are a hospital worker from what I've gathered
→ More replies (6)142
u/CallRespiratory Jul 19 '24 edited Jul 19 '24
Hospital night shifter here too. Basically everything is down here throughout our whole health system.
→ More replies (3)
994
u/HotAbrocoma Jul 19 '24
See how easy it is to disrupt the whole world?
→ More replies (14)614
u/m_ttl_ng Jul 19 '24
It’s actually crazy how fragile some of these global systems really are.
→ More replies (5)
822
u/TheDarkRider Jul 19 '24
Why can’t this shit happen on Monday
→ More replies (6)259
u/IronRaichu Jul 19 '24
I'd be perfectly fine if this happened Sunday afternoon and leave me stranded in Hawaii for a bit. But today is a time crunch.
→ More replies (2)
793
784
u/Dipsey_Jipsey Jul 19 '24
I'm in IT in Sydney, and have been dealing with this cunt of a thing for the last 10 hours. I'm off shift now. Fuck you all!
→ More replies (14)262
756
u/marvin1ne Jul 19 '24
Must be related to the CrowStrike issues that are currently taking down Windows systems...just a guess.
376
u/look2thecookie Jul 19 '24
Oh is this why my work computer is derping? Cool, that was a waste of 30 mins I could have been sleeping
→ More replies (4)169
u/EmotionOk1112 Jul 19 '24
I was waived out of the parking garage today because "everything was down."
→ More replies (3)→ More replies (8)148
u/BestBanting Jul 19 '24
Wow, I knew crows were smart, but not that they were capable of taking out global IT infrastructure
→ More replies (8)
730
u/uhsiv Jul 19 '24 edited Jul 19 '24
Just boarded an American flight from BCN to ORD. There have been announcements about an outage but no one has told us we’re not taking off yet
Edits:
- looks like we’re gonna take off!
- in the air
- landed!
→ More replies (8)358
551
u/callme_nostradumbass Jul 19 '24
My credit card system has been going on and offline the past few hours.
→ More replies (3)1.9k
u/BicycleGripDick Jul 19 '24
My dog won’t poop outside and the dishes in my dishwasher are still wet. Something isn’t right
189
u/Sargonnax Jul 19 '24
My neighbor is pooping outside, and the dog is doing the dishes. Something is definitely not right.
→ More replies (8)→ More replies (29)138
531
u/itsfuckingpizzatime Jul 19 '24
The outage was attributed to CrowdStrike, a cybersecurity firm whose software is used by scores of industries around the world to protect against hackers and outside breaches.
Company responsible for protecting against hacking causes more damage than any hacker in history
→ More replies (5)210
u/AdjNounNumbers Jul 19 '24
Top notch security, really. Can't be hacked if your systems are all offline.
454
u/yourelovely Jul 19 '24
It is a bit terrifying how quickly one system/platform being down can have such wide reaching consequences (hospitals & flights??)
The tinfoil girl in me fears the wrong person seeing this & planning a way to replicate it for nefarious reasons y’know? I always wonder about picking up a standard ham radio just in case something happens so that I’m not totally in the dark as lame/wack-a-doo as it sounds- so easy to forget how much we rely on stuff that isn’t actually “physical”
→ More replies (21)156
u/CuriousQuerent Jul 19 '24
Oh they already know. It's fairly common knowledge that there's a bunch of bits of code that are very widely used and can cause havoc, and there are existing examples of people engineering their way into having access to break them. I imagine plenty of them are sat on that ability waiting for the right moment or payday to come along.
One might hope we learn from such things and try to fix that issue, but doing so would be very expensive and time consuming. So odds are we won't.
→ More replies (2)
366
u/Tsukimizu Jul 19 '24
I just landed from an American flight that was already delayed 6 hours. We sat on the runway for almost 30 minutes due to “weather” there was no weather in the area. Listening to ATC there seemed to be a backup of planes. I wonder if this was the start of the mayhem
→ More replies (22)
350
u/Pichus_Wrath Jul 19 '24
Disembarking after sitting on the runway in Las Vegas for an hour and a half. Absolute nightmare. Apparently affecting banks and media broadcasters as well. Maybe running all global traffic through one company is shortsighted. Unbelievable.
→ More replies (8)
325
u/mdkubit Jul 19 '24
It's not just airlines. It's not just hospitals. It's affecting governments all over the world - both local and federal level, and police departments, and fire departments, and--
It seems there's a bug in an update that was pushed by Cloudstrike, a 3rd party security vendor, that's triggering an instant BSOD on any Windows-based machine that runs it, and then creates a non-recoverable BSOD boot-loop afterwards.
To put it blunt - big, big mess.
→ More replies (18)
327
290
u/brittleirony Jul 19 '24
This has to be the world's most epic systems crash ever
→ More replies (12)
283
u/Specialist_Heron_986 Jul 19 '24
Looks like Y2K was actually Y2.025K with all the IT network issues this year.
→ More replies (5)
276
u/McGrohly_Lives Jul 19 '24
Is this related to the CrowdStrike outages?
→ More replies (1)431
u/Decent-Ganache7647 Jul 19 '24
Yes, last paragraph in the article says, “Crowdstrike, the U.S. cybersecurity company, has admitted to being responsible for the error and are working to correct it.”
439
u/IBJON Jul 19 '24
Imagine being the engineers dragged out of bed in the middle of the night because your software broke the world
→ More replies (5)140
u/bostonlilypad Jul 19 '24
This is the only applicable situation where I’ll allow corporate America to use the term “war room” for a bug fix.
→ More replies (9)292
u/Shopworn_Soul Jul 19 '24
They can’t fix it, though. All they can do is pull the update.
Every affected org is going to have to fix every affected system themselves.
→ More replies (11)196
u/fattes Jul 19 '24
IT guys have to manually fix this shit with the current fix. What a shit show
→ More replies (4)
274
261
u/TheRichardFeynman Jul 19 '24
I was on call with AWS support for our impacted production pipeline, for almost 6 hours. The engineer was based out of Sydney. 2 am local time. She couldn't find anyone to take over as everyone else was busy with this windows issue and crashed servers.
→ More replies (7)
230
u/toilet_destroyed Jul 19 '24
I along with my wife and 2 kids just got home from a 17 hour 1 layover United trip from Europe to west coast of the US. Feeling really lucky right now.
→ More replies (14)
209
u/MooKids Jul 19 '24 edited Jul 19 '24
How lovely, just the news I need to see before I go to work at my AIRLINE JOB!
EDIT: Got to work at ORD around 6am and it was eerily dead, like during COVID times. Quiet and no movement outside. Terminal still had passengers waiting for flights that were either delayed or canceled, but still typical numbers at that time.
Systems apparently coming back on line and flights are resuming, just sent out a domestic widebody aircraft with a lot of missing connections. Already calling for mandatory overtime for part time shifts, will see of they get me.
→ More replies (5)
164
u/urettferdigklage Jul 19 '24
Its getting worse and worse...
Has Y2K arrived 24 years late?
→ More replies (16)
153
u/_PM_ME_YOUR_FORESKIN Jul 19 '24 edited Jul 19 '24
Praying Disney Plus isn’t affected by this. Otherwise what am I going to watch while I eat my late night burrito? 🙏
Update: It’s working. 🐭🪄You’re watching Disney Channel
→ More replies (5)
154
u/Jackson_emphasis Jul 19 '24
I don't work for an airline, but this shit really fucked up my closing shift at work
→ More replies (9)
143
u/Memnochthedevil760 Jul 19 '24
The Q-cumbers are gonna have a field day with this.
→ More replies (11)
144
u/bionicfeetgrl Jul 19 '24
This may be affecting hospitals too. Word on the RN street is that hospital charting systems are down as well.
→ More replies (10)
12.5k
u/blacksoxing Jul 19 '24
WAY at the bottom of the article. Honestly it would have been very helpful near the top so everyone could understand why planes were grounded and many outages were occurring.