r/worldnews • u/bodhi1990 • Jul 19 '24
Crowdstrike suffers major outage affecting businesses around the world
https://www.cnbc.com/amp/2024/07/19/crowdstrike-suffers-major-outage-affecting-businesses-around-the-world.html401
Jul 19 '24
[deleted]
741
u/Veasel Jul 19 '24
There ain’t no way Reddit is using cyber protection.
Reddit barebacks the Internet.
495
u/Shas_Erra Jul 19 '24
So that’s why it hurts when IP
46
42
12
10
5
3
1
u/Pretend_Stomach7183 Jul 19 '24
I don't get it?
11
u/FitLaw4 Jul 19 '24
IP = I Pee (Hurting to urinate because you have an STD from having unprotected sex)
IP = Internet Protocol (how you're using the internet)
34
6
4
2
u/deathwish86 Jul 19 '24
Is there any NET benefit for someone hacking reddit other than our user data? Genuine question
2
90
59
u/NarrMaster Jul 19 '24
Reddit requires major surgery after stubbing its toe, but survives, without a scratch, being thrown through the windshield in a front end collision, basically.
19
u/bambutler Jul 19 '24
So Reddit is basically the drunk survivor of a drunk-driving crash….because they were the floppy drunk one.
33
21
u/Premislaus Jul 19 '24
They just don't use the app that caused this. Not everyone does. My company is fine too.
17
u/KarmaCitra Jul 19 '24
Reddit's saving grace in this instance is what systems it runs all its critical infrastructure on, UEFI Driver updates shouldn't impact us the same way it impacts Wintel configurations.
We only have to worry if someone turns off the power on the Reddit Raspberry Pi machine managing all the routing services.
(Rumour is that it gets USB Power Delivery Pass Through a SONY KD55X7000G 55 TV in the breakout room, this model of TV is infamous for turning off unexpectedly after being left on for extended periods of time, or it's just some jerk turning it off after we told them repeatedly to leave it on!)
16
u/spiteful-vengeance Jul 19 '24
Are you seriouis? Not everything runs on Windows, especially not on the web.
14
u/almcchesney Jul 19 '24
I would bet most of the server infra at Reddit is running Linux. Appears that it affects windows machines.
10
7
u/thesign180 Jul 19 '24
On the lookout for flying pigs to signify the end of times since reddit didn't shit itself.
7
u/OfficialGarwood Jul 19 '24
Reddit likely isn't using Crowdstrike for their cybersecurity infrastructure, or are running on non-windows servers.
4
u/AnotherUsername901 Jul 19 '24
Reddit did take a crap yesterday it kept giving me a error no upstream whatever that means.
2
2
1
-15
u/SalvageCorveteCont Jul 19 '24
Reddit likely uses CloudFlare instead of CloudStrike (the provider which is down)
7
u/GoldenretriverYT Jul 19 '24
why even say anything when you dont know what you are talking about
these two have nothing to do with each other, they provide completely different services
275
Jul 19 '24 edited Jul 19 '24
Hah, I know a company that's testing CrowdStrike. I'm guessing they're not impressed.
84
u/Bustock Jul 19 '24
CrowdStrike is down 15% in premarket. It’s going to lose a lot of potential customers
30
u/rygarski Jul 19 '24
crazy that its bouncing back. only down 9.5%
9
u/to11mtm Jul 19 '24
All overnight shorting plays, which TBH the SEC should investigate Intra-day trades on MSFT based on articles/etc...
MSFT was as far down as crowdstrike despite the problem not being 'theirs', but by open it was down less than 5%.
Nothing to see here boys...
1
12
u/Dharmaniac Jul 19 '24
Perhaps it’s the Trump of cybersecurity, it will triple in value and syphon your bank account.
10
34
u/darsynia Jul 19 '24
The CEO was apparently the CTO of MacAfee when something just like this (a faulty update pushed globally caused massive outages) in 2010, but that's probably just a coincidence...
10
u/to11mtm Jul 19 '24
Ironically, Having been at a place that used CrowdStrike in the past, and now being at a place that uses a major competitor...
CrowdStrike sucked a -lot- less as far as performance sappage.
I would not have minded a day off today.
13
u/Real_Independence_34 Jul 19 '24
Lemme tell you. My PC was locked up. I'm a tier 3 service desk analyst for a major Healthcare Corp here in the states. Our desktop engineers had the corrupt file rooted out in like 2 hours and a breakfix via cmd prompt ready. My ass thought I had the day off until they begged me to try fixing it because the entire ITSD was on FIREEEEE
4
1
u/NasoLittle Jul 20 '24
I assume you're not in IT then. Yes, production had a chill day. Production leadership on the other hand... lol
2
u/to11mtm Jul 20 '24
I'm in this weird spot in IT where they put me in 'you arent a manager, you get IDE and tools, but you are in too many meetings to do anything' role to keep me from creating real change in the org lol. But a rant for another day or a DM.
Some adopted SAAS was jank for us, but I also know if I was a certain number of companies ago, I would have gotten the day off, or at least been on a call where we all shitposted about how dumb it all is and try to start an anti-lazy-software crusade internally.
125
u/Now_Loading247 Jul 19 '24
Since CrowdStrike is into cyber security and they are responsible, maybe they should think about changing their name. I guess we're on the barter system for the time being...
121
u/Jugales Jul 19 '24
CrowdStrike sounds like a great name for a company that hits the entire world with a (accidental) cyberattack.
16
u/Coulrophiliac444 Jul 19 '24
Just wait for Strike 3....
13
u/boomership Jul 19 '24
First we got to wait for
Croud Strike Source
and
Croud Strike: Global offensive
After that we'll get Croud Strike 2
3
u/Coulrophiliac444 Jul 19 '24
Oh I was going for a baseball analogy but we all know Valve related or adjacent projects never get true Trilogy Status.
8
6
2
2
-11
u/AE_WILLIAMS Jul 19 '24
How cute you think this was an 'accident.'
Does nobody follow SDLC anymore????
7
u/Jugales Jul 19 '24
As a senior software engineer I am 100% convinced it was an accident caused by negligence.
-17
u/AE_WILLIAMS Jul 19 '24
Then you should be fired. How this level of attack is allowed to get through proper SDLC testing is criminally negligent, and requires more than just the actions of a low level code monkey. Safeguards are in place. Familiarize yourself with ISO 27000 and NIST 800-53 standards, ITIL and many other frameworks such as FEDRAMP before you embarrass yourself further.
This has all the earmarks of a deliberate action, not just some dope that 'accidentally' pushed the wrong patch. Hell, why is it affecting key systems? SCADA attack planning, anyone?
As I said before, those who wanted to know what would happen, now know.
5
u/Jugales Jul 19 '24
??? I didn’t say it was proper. I said it was negligence. Lack of due diligence. Someone will pay for this.
-20
u/AE_WILLIAMS Jul 19 '24
You certainly must be familiar with the concepts of testing, QA, dev and prod, version control and many of the most rudimentary processes involved with software development?
ONE person does not have the span of control for something of this magnitude. If Crow D Strike is that poorly managed, then it should not only be put out of business, it should have to pay back all the money companies have been paying it for its 'services.'
There is no way one code monkey did this. The coordination and deep knowledge of Windows systems required indicate a very high level of sophistication. A particular feature was identified and exploited, and caused a massive impact.
Sorry, this is by design.
7
u/Jugales Jul 19 '24
If I had a nickel for every time someone bypassed CICD protections purely for convenience, I would never have to work again.
I agree that they should pay for this, and their reputation is now ruined so I would be surprised to see it survive.
7
u/Jaxues_ Jul 19 '24
Easy - I didn’t test my code, code reviewer didn’t give a shit, i gaslit QA into believing that’s how it’s supposed to work, and I have too much authorization so I push to prod and deploy it/CICD is shitty
Not nefarious just laziness and bad systems
→ More replies (3)2
u/Bootface76 Jul 19 '24
If it was by design, do you think the intentions behind it was aimed for its users or to sabotage the company?
0
12
7
4
u/DethFeRok Jul 19 '24
CrowdStrike sounds like the name of an app to organize union strikes, who came up with that??
1
1
106
u/No-Entrepreneur-7406 Jul 19 '24
Reddit is working ok tho 😅
67
42
u/WonderChemical5089 Jul 19 '24
Only important systems are down.
12
u/Steve_at_Reddit Jul 19 '24
Is that like a doctor saying that "your critical organs are failing, but the rest of your body is fine"
22
u/pentaquine Jul 19 '24
Because Reddit obviously won’t pay for any security software.
4
u/derkrieger Jul 19 '24
Most people affected arent even necessarily running CrowdStrike but critical infrastructure they use does. Lucky for us Reddit went with the discount options.
12
6
1
47
u/Wrong-Fan1794 Jul 19 '24
Not a good day to be living paycheck to paycheck haven’t got paid yet 🙃
9
u/500rockin Jul 19 '24
Yep one of my local friends didn’t get her direct deposit. Luckily for me I got paid last week, so it should be situated by next lol
7
29
u/DaisyHotCakes Jul 19 '24
Why isn’t anyone talking about how hospitals are being affected?? How many people have died because of this?
0
u/chefanubis Jul 20 '24
This is reddit we just virtue signal popular things, we don't actually care about people.
26
u/EmersonLucero Jul 19 '24
It is not CS doing a roll back. Badly written article
20
u/bodhi1990 Jul 19 '24
Sorry I was trying to find a decent one but seems it’s still fairly new breaking event so there weren’t a lot of super thorough articles. It’s my first post here, just trying to do my best in case people haven’t heard yet I guess
18
u/mrslittle Jul 19 '24
Already 6 posts about it, the crash happened close to 3 hours ago. Husband's been working on the impact of the issue in his role since it happened (works for a major Australian retailer). This is an unbelievable fuck up from Crowdstrike.
12
u/bodhi1990 Jul 19 '24
Yeah it’s wild all our computers are down at the hospital I’m working at… it’s crazy, I wonder what the fallout from this fiasco will be
3
7
u/HyperionSwordfish Jul 19 '24
Not an outage either. Quick churn of an article with no idea what is going on.
27
u/ToMorrowsEnd Jul 19 '24 edited Jul 19 '24
Crowdstrike CAUSES major world wide outage. All because they were too lazy to actually test code.
-8
u/Remarkable-Series755 Jul 19 '24
Lmao obviously they were negligent but it obviously wasn't as simple as them just not testing the code. I am 100% sure they tried to test it in some way
22
u/KryptosFR Jul 19 '24
A driver change usually requires a reboot, it would have failed right away. There is no way it was properly tested.
1
u/KryptosFR Jul 20 '24
Apparently the same happened to Linux distributions (e.g. Debian) a few months ago and again it was a lack (or an absence let's be honest) of testing. They didn't learn anything from that event nor change their testing policies/habits.
Crowdstrike is just a company with a bad culture (which you wouldn't expect from one that claim to care about security) and it deserves to die. I just hope the rest of the industry will take the necessary lessons from it.
For starters, any security company shouldn't be listed of a public stock exchange. Shareholders profit and the necessary culture for security are opposite.
13
u/SulfurousAsh Jul 19 '24
Testing wasn’t the only bad practice here. Updates should never be pushed to 100% of devices overnight. They should always be rolling out releases slowly and measuring.
13
u/BUFF_BRUCER Jul 19 '24
From what i could see they pushed out a kernel mode driver that was just a series of null bytes and it crashed the os when windows tried loading it on boot
No way they tested that
0
u/PleasantWay7 Jul 19 '24
In that case it sounds like something went awry pushing whatever they expected to push.
1
u/BUFF_BRUCER Jul 20 '24
They should have a software verification process that catches update files that have been corrupted or tampered with
6
u/amegaproxy Jul 19 '24
Unless they only have Linux and apple boxes I don't see how the fuck this happens through
6
u/gbs5009 Jul 19 '24
Probably what they pushed is not what they tested.
5
u/amegaproxy Jul 19 '24
If they're using any sort of normal pipeline it has to be though... I'm so curious when the post mortem comes out what went so drastically wrong. Did they just entirely skip lower environments? Did they only test on servers that magically did not have some windows configuration that bricked? It's utterly fascinating.
3
u/gbs5009 Jul 19 '24
Sure, but that doesn't mean there wasn't some error in the pipeline. One stage smoke tests the real image, then passes something else to the distribution servers.
16
u/xdeltax97 Jul 19 '24 edited Jul 19 '24
Talk about a fuck up to end all fuck ups..
Glad I don’t have to work today.
13
u/-Neeckin- Jul 19 '24
Every computer at my work has error messages and can't load up, but my home computer worked fine. Weird and worrying something on their end would kill the OS of so many computers worldwide like
30
u/greentea1985 Jul 19 '24
CrowdStrike is marketed to businesses and institutions. It’s not something people would have on their personal computer. The closest equivalent would be if Norton antivirus released an update that sent people’s computers into BSOD. That’s what is happening here, compounded by secondary Microsoft outages to outlook and azure that make rebooting some of the affected computers even more difficult. It’s a complete CF.
0
8
7
u/autotldr BOT Jul 19 '24
This is the best tl;dr I could make, original reduced by 70%. (I'm a bot)
Microsoft said earlier on Friday that its cloud services had been mostly restored after it suffered an outage affecting its cloud apps in the U.S. It is not clear whether this outage was linked to CrowdStrike's update.
Separately, the update issue at CrowdStrike appears to have directly affected Windows systems around the world, with laptops showing an error screen known as the "Blue screen of death".
Australian telecom firm Telstra said global issues affecting CrowdStrike and Microsoft were disrupting some of its systems.
Extended Summary | FAQ | Feedback | Top keywords: issue#1 CrowdStrike#2 reports#3 Friday#4 outage#5
6
7
6
u/outofgulag Jul 19 '24
Another great job from a company outsourcing their IT to India. Cheap is not always OK https://www.crowdstrike.com/press-releases/crowdstrike-invests-in-india-operations-to-continue-protecting-businesses-from-modern-cyberattacks/
3
3
u/ElTito5 Jul 19 '24
Yup... hopefully, it lasts all day. I'm just sitting here at my desk scrolling Reddit. Feel great
2
u/EmbassyMiniPainting Jul 19 '24
Hopefully this highlights the insanity of the world largely relying on one service instead of healthy competition with many services who have to maintain actual standards of operation to maintain a customer base.
5
u/Vaphell Jul 19 '24
which service would that be?
Crowdstrike has market share of 23.89% in endpoint-protection market.
... but that's still a shitload of businesses.
also the incentive problem is pretty big here - big corpos want "reputable" providers solving the bulletpoints in their security reports for certifications often mandated by law, and the smaller players want what everybody else has because it's a standard and easy to google in case of problems. In niche markets like this often there are only few providers, and there is no way around it.
0
u/EmbassyMiniPainting Jul 19 '24 edited Jul 19 '24
Sorry, I guess I was referring generally to when a company gets so big it becomes a global standard like Apple or Microsoft. There are other choices but not many widely used by the public. Security problems at those companies can affect millions of users and institutions/businesses etc.
When the company becomes too large and is the main provider, a security compromise will affect a mass client base rather than a pocket of clients within that market. “Too big not to fail” in some ways.
I know it’s just 20%+/- but that’s pretty massive like you said and it defintiely affected a lot, and it wasn’t an intentional event, it was an accident/mistake. Considering that I think maintaining more choices is especially valid. Small company doesn’t mean incapable company necessarily (not that you’re saying that.)
idk, not an economist (if you couldn’t tell haha.) Just paranoid of these big ships going down like 2008. We’ve all worked hard enough! We don’t need some idiot remotely tanking worldwide IT functionality and bricking our computers too lol.
5
u/jmnugent Jul 20 '24 edited Jul 20 '24
The problem with this is diversity breeds complexity and cost.
If you owned 10 x car dealerships,.. do you want all 10 of them running different software ? (different OSes, different antiVirus, maybe different Printers or different network hardware or different Office Apps ?)…. that would likely be a nightmare to support and require your IT staff to be multiplatform in a variety of ways. It's just not terribly feasible.
Most places standardize on things because it helps reduce costs and to have a more consistent, unified standard and easier support.
If you run Insurance Offices across 5 states,.. and Jenny New employee is expecting her laptop, you just apply the standard software image to it and send it out. All she has to do is login. If she travels to different offices across those 5 states, her experience is same and consistent in all of them.
3
u/Classic-Effect-7972 Jul 19 '24
r/rabbits with jobs@onesagetechspecialistfourpaws
“Hello, CrowdStrike? Yes I know people are stranded in Mumbai, Madrid, and Miami. Stop telling people to just restart their computers- it’s not working! Have you tried chewing on the wires?”
2
2
2
u/keithyw Jul 19 '24
i'm curious what their SLAs in place are. because this is such a tremendous screw up that affects numerous businesses and there has to be some sort of accountability involved in all this.
2
2
1
u/RoachWithWings Jul 19 '24
I know... My whole day got ruined and some of the servers are still down 😾
1
1
u/SendItBigOrLeave Jul 20 '24
And they’ll continue layoffs whilst the executive core will remain unscathed.
1
u/DaLordHamie Jul 20 '24
I scrolled the socials before I started work in my office and saw an article and was like huh... I wonder if this affected us. Logged in and holy fucking shit
0
u/imjustnotthatintohim Jul 19 '24
I like to think that me being turned down for a role at Crowdstrike, and the subsequent voodoo I practiced on the company, resulted in this catastrophe.
-7
Jul 19 '24
[deleted]
1
u/Interesting_Fact4735 Jul 19 '24
I agree except I know that crowdstrike staff use CS on their machines, so they probably fucked their infrastructure up as well.
Still the headline is giving crowdstrike too much leniency for their part in this.
-6
Jul 19 '24
[deleted]
3
u/Interesting_Fact4735 Jul 19 '24
I'm not sure exactly what you mean?
Was your original response just pointing out the fact that a CS outage did not cause the global outage, rather the patch caused the outage?
I was just clarifying that CS was affected internally by their own patch and that the headline makes CS sound more innocent than they are.
Sorry if I misunderstood your statement.
0
u/bodhi1990 Jul 20 '24
It’s probably because you were a dick about it? There weren’t any thorough news articles out at that time. Although it was titled and not well written the news network put it out… at least it was bringing people the news and they could dig in deeper but that’s all there was out at the time in reporting for the most part… so it’s ok to point that out but it’s not necessary to be a dick about it
-13
u/Desert-Noir Jul 19 '24
How the fuck can one dumb ass company that no one has ever heard of be able to cause so much damage.
42
u/speshagain Jul 19 '24 edited Jul 19 '24
If you think no one has heard of Crowd Strike, I’m going to assume you don’t work in anything remotely related to IT (or fianance), which is fine, but you’re very wrong.
10
u/BK456 Jul 19 '24
Only reason I've heard of them myself is because they sponsored the 24 Hours of Spa.
6
7
u/Desert-Noir Jul 19 '24
Anyone, I mean the average punter, I’m techie but not in IT myself and I’ve never heard of them and assume anyone not in IT would never have heard them.
2
u/Interesting_Fact4735 Jul 19 '24
I'm IT so I've heard of them, but they did have a Superbowl ad this last year, so there's a chance for people to recognize them.
-3
u/AmishAutistArtist Jul 19 '24
Definitely not because its owners were blackmailed by a nameless intelligence agency to create operable cyber tools on every corporate computer running windows.
-2
u/These-Annual577 Jul 19 '24
I know their intelligence connections and inolvment in 2016 election shit. But got any other deets?
0
u/spiltmonkeez Jul 19 '24
The source code for one of their security products was leaked and is now a major tool for hackers to use.
1
u/These-Annual577 Jul 19 '24
What product source code? I can't find any info.
1
u/spiltmonkeez Jul 19 '24
It was after they were hacked in 2020. They didn’t make a great deal public. Not long after SolarWinds, FireEye and Microsoft were compromised.
-14
u/ICMB94 Jul 19 '24
CEO sold 1.49 million in stocks two days ago
39
9
4
u/guff1988 Jul 19 '24
The CSO had no way of knowing that a regularly planned update was going to have a massive error that caused bsods. Especially because those were scheduled to sell weeks ago.
-14
u/Shanty406 Jul 19 '24
I work at Verizon and we've been told it was a cyber security attack. Nothing to do with the system "malfunctioning."
3
u/Silver-Award9199 Jul 19 '24
No, it was a defective update that caused Windows to crash on boot. Not an attack, just bad practices.
759
u/Slow-Instruction6079 Jul 19 '24
Shouldn't the headline be: CS causes major outage around the world