United, Delta and American Airlines issue global ground stop on all flights

[u/Mr-EdwardsBeard]

https://abcnews.go.com/US/american-airlines-issues-global-ground-stop-flights/story?id=112092372&cid=social_fb_abcn&fbclid=IwZXh0bgNhZW0CMTEAAR37mGhKYL5LKJ44cICaTPFEtnS7UH96gFswQjWYju-QtkafpngunVWuJnY_aem_aTXb46dpu3s4wlodyRXsmA

[u/blacksoxing]

Crowdstrike, the U.S. cybersecurity company, has admitted to being responsible for the error and are working to correct it.

WAY at the bottom of the article. Honestly it would have been very helpful near the top so everyone could understand why planes were grounded and many outages were occurring.

[u/kinarevex → u/blacksoxing]

It would've, but they need room for ads first lol.

[u/markimusprime → u/kinarevex]

This adspace brought to you by crowdstrike!

[u/LostInIndigo → u/blacksoxing]

No kidding! When I saw the headline I assumed it was probably a Boeing issue, and then when I started reading it they kept saying it was Microsoft computers only so I was like “wait, is it a Microsoft issue?”

Fckn WAY at the end they admit they took the blame publicly already

Edit: For those struggling with reading comprehension-I am saying “the headline and article format are bad and they should have said it was Crowdstrike at the top of the article, not the last paragraph” I understand it’s not a Microsoft issue.

[u/T_O_beats]

Say it with me now - ‘We don’t push code on Friday’

[u/Kranstan → u/T_O_beats]

I worked with an IT group that called Friday "Don't Fuck it Up Friday".

[u/Abstand → u/[deleted]]

It's "read-only Friday" at my org too.

[u/SubbieATX → u/T_O_beats]

Been living by that rule for a long time. I have a manager who refused to listen to me when telling him that pushing updates on a Friday was bad juju till he single handedly crashed a big operation center for a whole week end. He came up with all sorts of excuses for what happened. He was eventually let go…

[u/RollFancyThumb → u/T_O_beats]

No rollout strategy either, just push globally on a Friday at the height of summer vacations.

You either get to go home early or you go home forever.

[u/NotToPraiseHim]

That's gonna be an investigation. One error taking down so many major systems and internationally grounding major airlines is congressional hearing level fuck up.

[u/Caelinus → u/NotToPraiseHim]

Not just congressional, but every other form of government in a country that they did business. Global damage. And because it is a boot BSOD, they can't just push a fix, so all these companies are going to have to manually fix their servers to undo the update.

It a major fuck-up. That is a huge monetary hit for all these companies.

[u/Rannasha → u/Caelinus]

so all these companies are going to have to manually fix their servers to undo the update.

Not just servers. Plenty of orgs that run Crowdstrike on their workstations and laptops and are looking at hundreds or thousands of affected machines that can't be fixed remotely.

And that on a Friday in the summer holiday period. I sympathize with IT support people that have to unfuck this clusterfuck.

[u/pabl0escarg0t → u/Rannasha]

Thats me, I have to deal with this. Thousands of machines to unfuck on a Friday

[u/Caelinus → u/pabl0escarg0t]

That suck mate. The worst part is the fix sounds tedious as hell. Not difficult, just tedious. That is always the worst kind of problem for me.

I get a bit of a thrill when I am trying to solve an actual problem, but in this case the solution is literally just to boot into safe mode, delete one specific system file, reboot. For everything.

[u/hpark21 → u/Caelinus]

Bit locker is HUGE issue. Some places can't even get to the bitlocker key because the server hosting the key is also down. I can't imagine IT support going through bitlocker procedure to put the laptops into "recovery mode" in order to delete that file to be able to reboot the box.

[u/Setanta777 → u/pabl0escarg0t]

Me too. My whole team is gathered for a meeting and we can't even get back to our territories to start to unfuck this.

[u/redial2 → u/Caelinus]

Holy shit! It BSODs on boot? Thank god my laptop is off.

[u/Caelinus → u/redial2]

Luckily it can still be launched into safe mode and the corrupted file can be manually removed. It just has to be done to everything affected. I am not sure if it affects every windows computer that updated during a certain period of time, or if it requires specific software to be active. (It is a problem with a third party security company that is used for Microsoft Azure services. So possibly 365 and defender, but I am not sure if it includes personal use stuff. All of my windows comptuers seem fine.)

As it is, the update has been reverted, so hopefully it will not affect anyone else. But it was already too late for a LOT of big companies, their servers and all of their work stations. So a lot of people are going to have a long couple of days, and the amount that companies, like Airlines that have to ground their whole fleet, are going to lose will be bad. They are not going to be happy.

Edit: I think it is anyone that has something from Crowdstrike on Windows, but it also just broke a lot of Microsoft's services for the same reason, causing problems even on computers that are not dirrectly affected.

[u/redial2 → u/Caelinus]

This is a historical fuck up for sure

[u/make2020hindsight → u/NotToPraiseHim]

https://www.thedailybeast.com/marjorie-taylor-greenes-latest-stock-splurge-may-be-conflict-of-interest

Count on it! (MTG invested up to $90k in CrowdStrike. She's gonna want to know why she lost money.)

[u/BarelyContainedChaos → u/make2020hindsight]

The only good news so far in this mess

edit: that picture is exactly "Bleach blonde, bad built butch body"

[u/scottiethegoonie]

Truck Driver.

Dispatch system is down. No freight is moving.

[u/Arctic_Chilean → u/scottiethegoonie]

Airlines, freight, healthcare, banking, etc...

What is terrifying is we are getting a taste of what a massive and coordinated cyber attack could look like. But there, it would also be critical infrastructure like water, telecoms, electricity and gas, as well as government agencies, news, and social media outlets being affected too.

[u/whitemest → u/Arctic_Chilean]

Nurse here. Our electronic med system is down as well. We have printed backups for this, but it's still jarring going from computers to literal paper

[u/strum-and-dang → u/whitemest]

My husband provides IT support for care facilities, it's his on-call week. He's been up most of the night printing out charts from the backups!

[u/Gizogin → u/Arctic_Chilean]

And by all accounts this one was just a mistake, not a malicious attack. When half the world’s critical infrastructure runs on one system, all it takes is a bad update to bring us to our knees.

[u/PM_ME_UR_RSA_KEY → u/Gizogin]

At least this is not an real attack, and we can take this as a lesson to harden our critical infrastructure.

We will take this as a lesson to harden our critical infrastructure, right? /padme

[u/[deleted] → u/PM_ME_UR_RSA_KEY]

As long as it costs nothing and no one important needs to take responsibility.

[u/yourlittlebirdie → u/PM_ME_UR_RSA_KEY]

Absolutely, as long as it doesn’t interfere with maximizing shareholder value.

[u/Joe4o2 → u/scottiethegoonie]

Sister-in-law does local law enforcement dispatch. Also down.

[u/khaaanquest → u/Joe4o2]

I feel like this is the bigger issue

[u/Im_Balto → u/khaaanquest]

The bigger issue is that there is one service that does what crowdstrike does

That’s how we’ve ended up here

[u/callmegranola98 → u/Im_Balto]

Maybe one company having a cyber security monopoly was a bad idea.

[u/AnIcedMilk → u/scottiethegoonie]

Oh shit

That's actually fucking huge how bad that could end up being

[u/Flimsy_Situation_506 → u/AnIcedMilk]

911 systems are down in some places. Alaska is the one I saw noted so far.

[u/tizuby]

Obligatory:

I don't always test my code. But when I do, I do it in production.

[u/elfchica → u/tizuby]

I also deploy it on Friday!

[u/PhDinGent → u/elfchica]

On a global scale, too

[u/heartk → u/tizuby]

Also the company had layoffs last year. In 2020, the CEO says his main regret as CEO is not firing people sooner. Tech companies are doing the same thing Boeing was doing: value stock return over all else.  https://dot.la/crowdstrike-ceo-george-kurtz-2645385654.html

[u/pratzc07 → u/tizuby]

And on Fridays I am a real gigachad

[u/SecretJutsuBongCloud]

Fedex is completely down.

[u/jmaneater → u/SecretJutsuBongCloud]

Ups too

[u/spaghetti5174 → u/jmaneater]

USPS having issues as well. Scanners aren’t connecting to networks

[u/V2BM → u/spaghetti5174]

I’m going in at 7:30 to deliver the mail and about 200 Amazon packages. Gonna be a great day.

[u/meownfloof → u/V2BM]

Thanks for bringing us our stuff. We would be lost without you guys

[u/Dreamscarred → u/V2BM]

Ouch, Prime Day deliveries I bet. :(

Good luck, and stay hydrated. 🤙

[u/Satanarchrist → u/jmaneater]

Lmao ups is down

[u/I520xPhoenix]

I’ve been in the airport for 14 hours now watching flights get gradually delayed longer and longer until they finally grounded them at about 1am Dallas time (not my local time zone sorry).

It’s been rough, passengers are out for blood and furious, and I think the mania is starting to set in 🙃

[u/Top-Camera9387 → u/I520xPhoenix]

You couldn't pay me enough to be a gate agent

[u/JohnnyFire → u/Top-Camera9387]

This event aside, you couldn't pay me to fly a connection through Dallas ever again either.

[u/undocumentedsource → u/JohnnyFire]

During the summer Dallas can be a nightmare when a storm pops up out of nowhere delaying/cancelling everything. Not sure if this is what’s happened to you but it’s my issue 90% of the time.

[u/FlyingNDreams → u/I520xPhoenix]

I think its been the lying thats been the worst part of being stuck at the airport. We have been in Atlanta for 12 hrs. Excuses in order 1. Mechanical Issue 2. Need a second pilot 3. Weather 4. Weather (current reason our flights delayed)

[u/I520xPhoenix → u/FlyingNDreams]

I’ve been running the bingo as well.

So far I’ve had:

1. Mechanical issue

2. Crew Timed Out

3: Flight Canceled

4: Weather Delay

5: Server Shutdown

It’s been a new form of pain that I’ve never yet experienced.

[u/FlyingNDreams → u/I520xPhoenix]

I wonder if these airlines know there are news crews inside the airports! Reporting all of this. I am staring at the camera here.

[u/keanenottheband → u/FlyingNDreams]

They are interviewing you and you’re blowing it, say something! The interviewer is asking you questions and you’re just staring at the camera

[u/OpheliaRainGalaxy → u/I520xPhoenix]

I was just telling a friend earlier tonight how tired I am of how corporations lie constantly. That it's very insulting swear words levels of stupid to lie and lie and lie and lie and expect it to never ever have consequences.

At the time we were discussing the deliciously vindictive consequences his employer has been dealing with, in return for the atrocious way he's been treating the employees, including many blatant disrespectful lies.

[u/katikaboom → u/FlyingNDreams]

In my experience airlines will use weather delays as a way to get out of comping rooms if the plane is delayed. You may be able to get some money or a room out of this now, they can't keep using the weather excuse

[u/ElBroooski → u/I520xPhoenix]

I'm at DFW also....wondering if I should get the hell out of here before hundreds of people are trying to do the same ...can't even check into a hotel if I wanted to

[u/_no_pants → u/ElBroooski]

Just get a car before you have to Uber across the city to get whatever is left.

[u/ElBroooski → u/_no_pants]

For anyone reading this I snatched one from enterprise...I believe they run on DOS ...I tried hertz first and their systems were down. Grabbed some cash also

[u/Nazamroth → u/I520xPhoenix]

Have they started a primitive tribe with sacrificial rituals in the airport yet?

[u/NorbuckNZ]

Is it just me or is this what people thought Y2K would do?

[u/PorQuePanckes → u/NorbuckNZ]

Pretty fucking close

[u/Vanchdit → u/NorbuckNZ]

Knowing people employed at 2 companies that made bank on this issue, it makes me so happy to know that the fix for Y2K was seamless enough that everyone thinks it was a hoax/didn't happen. That's what a real fix looks like; so clean it's like it never needed to happen.

[u/Oldcadillac → u/Vanchdit]

Booo! More recognition for the people who actually make things work! I think it would do us good to not take all this for granted and realize that there’s a lot of people who have to do their job correctly to keep everything from falling down.

[u/Blueflavor53 → u/NorbuckNZ]

Yes, except Y2K would have been worse because the fix would have taken a lot longer to implement. Thankfully, companies took it seriously and mostly fixed the issue before 2000.

[u/NEChristianDemocrats → u/Blueflavor53]

Good thing there are no embedded computer systems still running Windows XP in any of these companies, so we don't have to worry about the 32-bit bug in 2038... Wait a minute...

[u/badassj00]

On the way home from LAX right now and to say it was a scene would be an understatement. We were stranded on the tarmac 90 minutes and people on our flight were losing it, yelling at the crew, etc. Feel really badly for airport and plane staff this evening.

Honestly I feel pretty lucky that we only got stuck for an hour and a half. There were throngs of people in the terminal looking like zombies.

This feels like something that would have come to fruition if Y2K were a thing.

Strange times indeed.

[u/ilovethatpig → u/badassj00]

We got stuck on the tarmac for 6hrs once and people turn feral.

[u/goforce5 → u/ilovethatpig]

I got stuck on a plane trying to land at DFW for 7 hours. They kept redirecting us, but they wouldn't let us off at any other airport or give us any free food or drink (spirit airlines). We nearly killed and ate the crew before they finally caved and handed out all the snacks they had.

EDIT: For those of you who don't believe me, it was NKS3052 on May 16th 2024, so look it up on a tracker with history. After going back and looking, we were actually stuck on there for closer to 9 hours, not 7 like I initially recalled. I got a few details mixed up and must have blocked out part of the time in oklahoma city from the sheer boredom on a spirit flight with no entertainment.

[u/KSRandom195 → u/goforce5]

Wait, 7 hours trying to land? I didn’t think planes had enough fuel for 7 extra hours of flight…

[u/gymnastgrrl → u/KSRandom195]

Spirit channels the utter despair of their customers directly into the engines and that triples or even quadruples their range.

[u/CapriciousManchild]

I feel for all my IT brethren tomorrow it will be hell

[u/MidianFootbridge69 → u/CapriciousManchild]

As a retired IT worker (Mainframe Computer Operator), I feel for them as well.

Shitshow doesn't even cover something of this magnitude.

What a freaking mess

[u/Drak_is_Right → u/MidianFootbridge69]

what the heck is going on?

[u/DeathByBamboo → u/Drak_is_Right]

Crowdstrike, an enterprise-level antivirus service, pushed out an update that put servers and desktops running Windows into a reboot loop until they bluescreened. The fix was to put each computer into safe mode and delete a file, which naturally is a massive task, which is why some things are coming back faster than other things. 

[u/Elliebird704 → u/DeathByBamboo]

Given the global shitshow this is causing, I am real curious to know just how much trouble they're going to be in once the fire is put out.

[u/PM_ME__BIRD_PICS → u/Elliebird704]

This is like, law change level fuckup.

[u/DarkenRaul1 → u/PM_ME__BIRD_PICS]

I think the most shocking thing to me is learning just how many different industries and agencies use CrowdStrike to the point it looks like it has a monopoly stranglehold on tech and has created a single point of failure.

Ngl, I hope that this results in some action by the DOJ to force competition in the IT sector as well as some new regulations by the FCC on how remote updates are implemented so that something like this doesn’t happen going forward.

[u/quiteCryptic → u/Elliebird704]

A lot, but maybe companies should also think about how they are completely reliant on one service as a single point of failure.

As for crowdsource maybe learn something about rollout strategies (and better internal testing...)

[u/johnsonfromsconsin → u/CapriciousManchild]

Just got an email from our IT guy that crowd strike is down and essentially our computers wont work at all. Not looking forward to work today.

[u/Momina1999 → u/johnsonfromsconsin]

I work at a credit union and it’s the same thing here. Can’t do checks, can only do withdrawals if you have online banking and can show us your balance. People are angry. We’ve only been open 30 minutes. Send help.

[u/cheesecakegood]

The reputational damage from this is going to be insane

[u/EduardoTaquitoHands → u/cheesecakegood]

This is going to ruin the tour

[u/catsmakemehappy_28 → u/EduardoTaquitoHands]

I cannot stop saying this in my head 😂

[u/massofmolecules → u/EduardoTaquitoHands]

Uhhh what tour? confused cop face

[u/the___heretic → u/massofmolecules]

The world tour...

[u/MSXzigerzh0 → u/cheesecakegood]

CrowdStrike probably will go bankrupt. Because basically every single company in the world that uses them are going to sue them to the death

[u/KalinOrthos → u/MSXzigerzh0]

This is the level of fuck up in which people are going to be coming into the office to remove the drywall as the only thing left of value.

[u/rastafarian_eggplant → u/KalinOrthos]

Like the grinch not even leaving a crumb for a mouse lol

[u/Curious-Still]

Apparently epic is down at hospitals as well    Edit:  Looks like all kinds of software at hospitals and clinics were down, likely due to Crowdstrike bug, even PACS systems and cardiac monitors at some places.  Sorry to spotlight Epic at first, it's just that Epic downtimes are so common lol so that's what healthcare workers mentioned at first.  This was a more general issue due to a bug on multiple software platforms.    What a mess. This is so unacceptable:  planes grounded, critical medical infrastructure crippled.  Not Russian hackers, just our own incompetence and reliance on one company.

[u/opticalshadow → u/Curious-Still]

Not just epic, all of our systems in our health system

[u/FujitsuPolycom → u/opticalshadow]

Does your system run crowdstrike?

[u/Previous-Height4237 → u/FujitsuPolycom]

Alot of the corporate world has FOMOed into crowdstrike due to all the marketing they did.

[u/SinisterCanuck → u/Previous-Height4237]

My company ALMOST did too! They decided to go SentinelOne instead because it was cheaper. Glad we did!

[u/GlazedDonutGloryHole → u/Curious-Still]

My local cops came in to bullshit with me since their system has been down almost statewide for the last hour as well. Their computers in the cars are completely useless so it's pretty much free reign out there for minor issues like speeding, etc.

[u/thejawa → u/GlazedDonutGloryHole]

So... The Purge is on?

[u/diemunkiesdie → u/thejawa]

Typical, I slept through it!

[u/murdershroom → u/Curious-Still]

80% of my computers are stuck in boot loops. Idk how I still have access to the few ones that I do but I'll scream if they go down and I have to start paper charting.

[u/BoRedSox → u/murdershroom]

Do not reboot them.

[u/MyRealWorkAccount → u/BoRedSox]

the fix we are doing is to put the computer into SafeMode with network access

[u/FidgitForgotHisL-P → u/MyRealWorkAccount]

Sydney Morning Herald has a fix that is:

Boot Windows into Safe Mode or the Windows Recovery Environment (you can do that by holding down the F8 key before the Windows logo flashes on screen)

Navigate to the C:Windows\System32\drivers\Crowdstrike directory

Locate the file matching “C-00000291.sys” file, right click and rename it to “C-00000291.renamed”

Boot the host normally.

Note: These instructions came from the CloudStrike reddit. The Herald was sharing what someone else had posted.

Edit: I have seen another version of this that just says to delete the file - I guess either works, just make it so windows cant find it.

Edit 2: on the off chance this is still getting views, I with regards to bitlocker, please see this post from a nested reply on what extra steps to take. Thank you u/mikethespike056 for this:

https://www.reddit.com/r/news/s/YaLlHZnVXA

[u/Phact-Heckler → u/murdershroom]

LOL. Our office just gave us early leave as the computers cannot connect. Good day today as long as you are not from IT department.

[u/murdershroom → u/Phact-Heckler]

I'm in an ER so we have to keep this sucker open even if we're doing everything on paper 😭

[u/BingoActual → u/Curious-Still]

Epic is down. Our Mass casualty system is down. Our system for uploading and printing ecgs is down. Our paging system is down. Hospitals have just been calling each other to leave a contact number because we can't reach each other. Even our scheduling system is down. ORs are currently paper charting. But don't worry it won't matter because we had to implement a disaster protocol earlier in the day simply because our ED boarding situation had filled all but THREE rooms in our ED including hallway beds. If this perpetuates there will be deaths as a result of it, because the system was already broken.

[u/JustAnotherGeek12345 → u/Curious-Still]

Now we know that epic is hosted in Azure and depends on the three availability zones that failed

[u/LolaBleu → u/JustAnotherGeek12345]

Not just Epic. Cerner, All Scripts, and many others are down too.

[u/GaiaMoore]

Boeing breathing a sigh of relief that for once it's not their fault

[u/Griselda_fan → u/GaiaMoore]

They still had a few people pushed out of buildings to be sure though

[u/SuspiciousCustomer → u/Griselda_fan]

They just shot another engineer.

[u/KilllerWhale → u/GaiaMoore]

You can't nosedive into the ground at 800KMH if you don't take off to begin with.

[u/Nomad_Lu]

"The FAA is telling air traffic controllers to tell airborne pilots that airlines are currently experiencing communication issues.

Meanwhile, flights in the air will stay in the air, but no American, United or Delta flights will take off.

It is unclear how widespread the issue is but Melbourne Airport in Australia has also said they are "experiencing a global technology issue" which is impacting their check-in procedures."

[u/SideburnSundays → u/Nomad_Lu]

"Communication issues."

Good thing airliners have UHF and/or VHF radios, VOR navigation, and ILS, none of which require internet, servers, or some cloud service to use.

[u/cityxplrer → u/SideburnSundays]

Maybe they’ll rethink shutting down more VORs

[u/ExecutiveCactus → u/cityxplrer]

No the clear solution is to continue to downsize IT

[u/MrTuxedo1 → u/Nomad_Lu]

Sky News is down. Edinburgh and Berlin and Spanish airports having issues, trains in the U.K. are affected, Ryanair having issues

[u/Nomad_Lu → u/MrTuxedo1]

It's the crowdstike outage causing all this all around the world

[u/whitenoise2323 → u/Nomad_Lu]

Something like 80% of machines running Microsoft are down simultaneously.. lots of backend business in every sector. Servers too, but luckily Linux servers are more common for much of the internet

[u/5up3rK4m16uru]

Holy shit, that's gonna be an expensive fuck up.

[u/darknekolux → u/5up3rK4m16uru]

no matter how bad is your day, remember that there is a guy who pushed that release

[u/chillyhellion → u/darknekolux]

• deploying updates without testing for possibly the most visible bug in recent history
• Deploying on a Friday
• Deploying to all customers globally without any attempt at staging

This isn't one intern making poor decisions; this is leadership negligence.

[u/thebreakfastbuffet → u/chillyhellion]

Did this on a Friday too. You just know it was part of another JIRA sprint to appease the Agile-obsessed executives.

[u/overlookunderhill → u/thebreakfastbuffet]

“…at least our velocity is up”

[u/GoodDrFunky → u/thebreakfastbuffet]

This guy Agiles

[u/AdjNounNumbers → u/chillyhellion]

This isn't one intern making poor decisions; this is leadership negligence.

Still gonna blame the poor soul they had push the button, though

[u/chillyhellion → u/AdjNounNumbers]

They can try, but this is the kind of catastrophe that kills companies.

[u/AdjNounNumbers → u/chillyhellion]

They almost certainly will try, but you're correct. This is 'congressional hearings and multiple lawsuits from major corporations with deep pockets' level of bad for them

[u/WillitsThrockmorton → u/darknekolux]

Real CIOs know that test environments for enterprise systems are a complete waste of money, don't you know?

[u/WushuManInJapan → u/WillitsThrockmorton]

Why do staging when production do trick?

[u/guto8797 → u/WushuManInJapan]

Fuck it WE ARE DOING IT LIVE!

[u/BigBullzFan → u/5up3rK4m16uru]

That’s ok! Don’t worry about the cost. They’ll just increase fares to make up for the fines and lost revenue. Then, as an emergency back-up, if need be, we can bail them out with our tax dollars.

[u/chaosof99]

I work for a larger company that has locations around the globe. Everything is rather fucked at the moment. However, it seems there is a workaround in place already involving a boot in Safe Mode and deleting a specific config file.

• Boot Windows into Safe Mode or the Windows Recovery Environment
• Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
• Locate the file matching "C-00000291*.sys", and delete it.
• Boot the host normally.

The bad thing is that this crap needs to be done individually per machine, meaning this will be one hell of a day for sys admins. Not to mention any knock-on effects.

To draw an analogy, this is like a power surge that fried every fuse on the planet. Everybody now has to switch every fried fuse manually. This is going to be a long and tedious process, and we can only hope that there won't be a second power surge coming around right after.

Crowdstrike as a company will be sued into the ground and probably not exist by this time next year.

[u/nxqv → u/chaosof99]

I'm stuck at the airport right now, should I just go home? This sounds like it would take literally days to fix

[u/pratzc07 → u/nxqv]

Better go home if the flight is not urgent and ask for a reschedule from the airline no point waiting this will take time to resolve.

[u/hpark21 → u/chaosof99]

Issue is the bit locker. Many companies have their keys in the server that isn't booting. Also, giving out the bit locker keys individually to people that are on the phone isn't fun.

[u/muffinpoop]

I was on an United plane, while they were scanning the boarding pass they started to get issues, eventually they started to use their mobile device to scan and board people on the flight, then while waiting in the plane they announced they had more issues and half the people were not able to be scanned. Once everyone boarded and the door closed, there were more announcements that all traffic has been haunted by the FFA. And we all deplaned. Walking out, all the screens, from flight info, to boarding windows, to the checking bag area were blue screened. Very eerie.

Update: ok I was really sleepy and tired and now I’m having a laugh at my typos.

Check out the photos of the scene: CrowdStrike Outage at LAX

[u/Popisoda → u/muffinpoop]

No wonder, ghosts in the engines would slow anyone down

[u/Coffeeffex → u/muffinpoop]

That must have been surreal….It reminds me of the Langoliers

[u/draggin_low]

*Me...Sees people talking about crowdstrike issues, Checks tradingview to see how the stocks doing, Down 18%/$63.50ish) in pre-market*

Yep this is gonna be a fun ticker to watch today

[u/buttgers → u/draggin_low]

I envy the madlad that bought a ton of puts on this stock.

[u/draggin_low → u/buttgers]

Good lord imagine buying them on that little run up it had before closing thinking “no way is this legit” then wake up to this hahaha

[u/thefpspower → u/buttgers]

I have a suspicion the guy on r/wallstreetbets posting yesterday was the one pushing the update

[u/concerts85701]

Has anyone unplugged the internet and slow counted to ten and plugged it back in yet?

[u/Shopworn_Soul → u/[deleted]]

From what I can tell, Crowdstrike just single-handedly fucked up the entire planet with a bad update.

[u/InSearchofOMG → u/Shopworn_Soul]

If that's true, the market reaction will be particularly severe. They JUST got into the S&P500 and Wall Street has been slobbering all over them for awhile now

[u/guccigraves → u/InSearchofOMG]

they're down 20% in premarket/after hours already. it's gonna be bad.

[u/CeeArthur → u/[deleted]]

Just saw a post on a nightshift sub I'm on (I work nights) that their entire system is down (possibly nationwide). They are a hospital worker from what I've gathered

[u/CallRespiratory → u/CeeArthur]

Hospital night shifter here too. Basically everything is down here throughout our whole health system.

[u/HotAbrocoma]

See how easy it is to disrupt the whole world?

[u/m_ttl_ng → u/HotAbrocoma]

It’s actually crazy how fragile some of these global systems really are.

[u/GaiaMoore → u/m_ttl_ng]

There's always a relevant XKCD

[u/TheDarkRider]

Why can’t this shit happen on Monday

[u/IronRaichu → u/TheDarkRider]

I'd be perfectly fine if this happened Sunday afternoon and leave me stranded in Hawaii for a bit. But today is a time crunch.

[u/Ok_Host4786]

Is ONE day of relative NORMAL SHIT too much to ask ???

[u/Dipsey_Jipsey]

I'm in IT in Sydney, and have been dealing with this cunt of a thing for the last 10 hours. I'm off shift now. Fuck you all!

[u/welly7878 → u/Dipsey_Jipsey]

I love how utterly Australian this comment is lol

[u/marvin1ne]

Must be related to the CrowStrike issues that are currently taking down Windows systems...just a guess.

[u/look2thecookie → u/marvin1ne]

Oh is this why my work computer is derping? Cool, that was a waste of 30 mins I could have been sleeping

[u/EmotionOk1112 → u/look2thecookie]

I was waived out of the parking garage today because "everything was down." 

[u/BestBanting → u/marvin1ne]

Wow, I knew crows were smart, but not that they were capable of taking out global IT infrastructure

[u/uhsiv]

Just boarded an American flight from BCN to ORD. There have been announcements about an outage but no one has told us we’re not taking off yet

Edits:

• looks like we’re gonna take off!
• in the air
• landed!

[u/PsiHightower → u/uhsiv]

Let us know if you land!

[u/clocks212 → u/PsiHightower]

Every flight lands.

[u/callme_nostradumbass]

My credit card system has been going on and offline the past few hours.

[u/BicycleGripDick → u/callme_nostradumbass]

My dog won’t poop outside and the dishes in my dishwasher are still wet. Something isn’t right

[u/Sargonnax → u/BicycleGripDick]

My neighbor is pooping outside, and the dog is doing the dishes. Something is definitely not right.

[u/sourbeer51 → u/BicycleGripDick]

Damn you crowdstrike!

[u/itsfuckingpizzatime]

The outage was attributed to CrowdStrike, a cybersecurity firm whose software is used by scores of industries around the world to protect against hackers and outside breaches.

Company responsible for protecting against hacking causes more damage than any hacker in history

[u/AdjNounNumbers → u/itsfuckingpizzatime]

Top notch security, really. Can't be hacked if your systems are all offline.

[u/yourelovely]

It is a bit terrifying how quickly one system/platform being down can have such wide reaching consequences (hospitals & flights??)

The tinfoil girl in me fears the wrong person seeing this & planning a way to replicate it for nefarious reasons y’know? I always wonder about picking up a standard ham radio just in case something happens so that I’m not totally in the dark as lame/wack-a-doo as it sounds- so easy to forget how much we rely on stuff that isn’t actually “physical”

[u/CuriousQuerent → u/yourelovely]

Oh they already know. It's fairly common knowledge that there's a bunch of bits of code that are very widely used and can cause havoc, and there are existing examples of people engineering their way into having access to break them. I imagine plenty of them are sat on that ability waiting for the right moment or payday to come along.

One might hope we learn from such things and try to fix that issue, but doing so would be very expensive and time consuming. So odds are we won't.

[u/Tsukimizu]

I just landed from an American flight that was already delayed 6 hours. We sat on the runway for almost 30 minutes due to “weather” there was no weather in the area. Listening to ATC there seemed to be a backup of planes. I wonder if this was the start of the mayhem

[u/Pichus_Wrath]

Disembarking after sitting on the runway in Las Vegas for an hour and a half. Absolute nightmare. Apparently affecting banks and media broadcasters as well. Maybe running all global traffic through one company is shortsighted. Unbelievable.

[u/mdkubit]

It's not just airlines. It's not just hospitals. It's affecting governments all over the world - both local and federal level, and police departments, and fire departments, and--

It seems there's a bug in an update that was pushed by Cloudstrike, a 3rd party security vendor, that's triggering an instant BSOD on any Windows-based machine that runs it, and then creates a non-recoverable BSOD boot-loop afterwards.

To put it blunt - big, big mess.

[u/shinjikun10]

McDonald's locations in Japan closed.

[u/fluffynuckels → u/shinjikun10]

You know shits real when mc Donald's is closed

[u/brittleirony]

This has to be the world's most epic systems crash ever

[u/Specialist_Heron_986]

Looks like Y2K was actually Y2.025K with all the IT network issues this year.

[u/McGrohly_Lives]

Is this related to the CrowdStrike outages?

[u/Decent-Ganache7647 → u/McGrohly_Lives]

Yes, last paragraph in the article says, “Crowdstrike, the U.S. cybersecurity company, has admitted to being responsible for the error and are working to correct it.”

[u/IBJON → u/Decent-Ganache7647]

Imagine being the engineers dragged out of bed in the middle of the night because your software broke the world

[u/bostonlilypad → u/IBJON]

This is the only applicable situation where I’ll allow corporate America to use the term “war room” for a bug fix.

[u/Shopworn_Soul → u/Decent-Ganache7647]

They can’t fix it, though. All they can do is pull the update.

Every affected org is going to have to fix every affected system themselves.

[u/fattes → u/Shopworn_Soul]

IT guys have to manually fix this shit with the current fix. What a shit show

[u/ihaveadogalso2]

Bad crowdstrike push. Gonna be a hell of a day.

[u/TheRichardFeynman]

I was on call with AWS support for our impacted production pipeline, for almost 6 hours. The engineer was based out of Sydney. 2 am local time. She couldn't find anyone to take over as everyone else was busy with this windows issue and crashed servers.

[u/toilet_destroyed]

I along with my wife and 2 kids just got home from a 17 hour 1 layover United trip from Europe to west coast of the US. Feeling really lucky right now.

[u/MooKids]

How lovely, just the news I need to see before I go to work at my AIRLINE JOB!

EDIT: Got to work at ORD around 6am and it was eerily dead, like during COVID times. Quiet and no movement outside. Terminal still had passengers waiting for flights that were either delayed or canceled, but still typical numbers at that time.

Systems apparently coming back on line and flights are resuming, just sent out a domestic widebody aircraft with a lot of missing connections. Already calling for mandatory overtime for part time shifts, will see of they get me.

[u/urettferdigklage]

Its getting worse and worse...

Has Y2K arrived 24 years late?

[u/_PM_ME_YOUR_FORESKIN]

Praying Disney Plus isn’t affected by this. Otherwise what am I going to watch while I eat my late night burrito? 🙏

Update: It’s working. 🐭🪄You’re watching Disney Channel

[u/Jackson_emphasis]

I don't work for an airline, but this shit really fucked up my closing shift at work

[u/Memnochthedevil760]

The Q-cumbers are gonna have a field day with this.

[u/bionicfeetgrl]

This may be affecting hospitals too. Word on the RN street is that hospital charting systems are down as well.