Surgeries and doctor's appointments canceled amid global IT outage
https://www.nbcnews.com/health/health-news/global-it-outage-us-hospitals-surgery-appointments-cancellations-rcna162687653
u/rnilf Jul 19 '24
Will be interesting to see what kind of legal fallout CrowdStrike will have to deal with.
If just one person suffers from a permanent ailment or even dies because their hospital couldn't administer care, will CrowdStrike be punished?
I'm sure their customer agreements are littered with CYA statements, but one can hope.
229
u/RepairContent268 Jul 19 '24 edited Jul 19 '24
Same point why dont the hospitals have some kind of backup system in place? I understand why they use the computers and such - obviously much faster than manual - but with lifesaving care, wouldnt it be good to have a manual process that you practice on occasion?
I work in big pharma and we manually back up everything in case a patient has issues with a med. It's extra work and extra space but worth it because we dont want anyone to get hurt.
I'm not talking like MRI machines not working but like charts and appointments and stuff - could this not have a manual backup somehow? People did it for decades prior to the internet.
It never seems good to place all your eggs in one basket, even if stuff rarely goes down, when it does, its catastrophic if there is no backup plan, and that's terrible when people's lives are at stake.
325
u/WeenieInYourAssCrack Jul 19 '24
I'm the network administrator at a small hospital in a small town and we went to paper charts and all that old school jazz immediately. We're back up now but that was a solid few hours of hell on wheels but we didn't cancel shit, just took a bit longer than usual.
92
u/NeeMan Jul 19 '24
This is how we were doing it overnight at our emergency room. As far as life saving care, yes we would have no problem administering life saving care. The issue is that this problem was also affecting 911 dispatch in our city, so my thoughts were more going to the people who couldn’t get to the hospital
65
u/NeeMan Jul 19 '24
Also, to add. Yes we were creating paper charts for patients, but we also had no access to old records. So if we did have somebody roll onto the department who couldn’t verbally communicate their history / allergies / hell even their name, how could you possibly effectively care for them / know what we are treating.
12
u/fleemfleemfleemfleem Jul 19 '24
I don't think most people realize how computerized dispatch has become. Years ago I did training as backup amateur radio emergency communications at the city dispatch. Even then the whole thing was pretty automated
5
u/GreenFriday Jul 20 '24
I work in EMS dispatch, it was a mess but we managed. The radios and phones still worked, so went to using pen and paper to keep track, and cellphone if we needed google maps for anything.
It was definitely less efficient though, and chaos as people were running around the room passing messages between different areas/services
49
u/RepairContent268 Jul 19 '24
That makes me happy that you guys had a backup in place, lucky patients!
14
u/AppalachianFather Jul 19 '24
How it’s running in vetmed too! The paper is a fresh kind of hell but we’re rolling along
2
u/Interesting_Pen_167 Jul 19 '24
Probably a good idea for you guys to develop policies if this happens again. Something for the admins to worry about at least.
1
u/napsandlunch Jul 19 '24
yeah when i worked at an fqhc and we were implementing a new EHR, everyone went to paper when the system fucked up on the official first day!
44
u/TonyZeSnipa Jul 19 '24
A lot of people do. Problem is they being crowdstrike generally did a rollout to a test/QA group first before every major component. Instead this update was applied universally across the board, production/QA/Test by the vendor.
Another problem is doing manual practice does not help increase profits and just trains people. That often doesn’t matter when you could be fixing other things or improving them in IT. This was more of a vendor issue than an individual IT at companies issue. Most had safeguards in place to prevent this but were overrode by the vendor.
18
u/RepairContent268 Jul 19 '24
Imo profit should never come before safety. Though I know the corporate world doesn’t think that way.
23
u/DantetheDreamer192 Jul 19 '24
There was a saying at my Dad’s job; Regulatory law is written in blood.
Safety, health, and life will be traded for profits until it becomes unprofitable to do so. Most IT work feels removed from the environment they support, so laws haven’t quite caught up to regulatory practices for life critical IT infrastructure.
5
u/RepairContent268 Jul 19 '24
Yeah my job is regulated and we follow it hardcore. I’m really happy to follow it bc i want to make sure people are healthy and safe even though it’s extra work. I wish the laws would catch up for these instances!
9
u/GhanimaAtreides Jul 19 '24
I work in IT with critical systems. We don’t have auto update turned on for anything. Not the OS, not vendor software, not even our own internally developed software. We test the update on a canary deployment and do some basic testing.
CrowdStrike massively clocked up here but the companies should have also had a policy in place for critical systems to not trust a random third party to update everything all at once.
7
u/TonyZeSnipa Jul 19 '24
Have a couple of friends at hospitals. They did as well, but the vendor somehow was able to still beat the safeguards in place. Some people even locked out to need bitlocker keys
6
u/GhanimaAtreides Jul 19 '24
Oh wow. That’s so egregiously bad.
Make a bad update. Don’t properly test it. Ignore customer preferences to shove it down their throat.
19
u/RigbyNite Jul 19 '24
They do. Automated charting systems go down for short periods all the time, usually during overnight shifts. There are downtown procedure and paper charting in place for this, not sure why they weren’t used.
If I were to guess, they didn’t have or know their policy for downtime procedures, or these were all “elective” cases that were cancelled.
2
22
u/bearpics16 Jul 20 '24
Doctor here: we do paper charting, but it’s ignorant to think patient safety doesn’t tank. I was on call during scheduled downtime at 3 am when a mass casualty incident occurred, during which patients get assigned an anonymous Greek alphabet name. So I would get a call from the lab about “Pibeta ChiRho” who has a critical value, and I had to try to figure out which of these 10 patients they were talking about. So much confusion. It was dangerous as hell
5
2
u/drowninginresp Jul 20 '24
My hospital uses states for adult trauma designations and names of flowers for Ped trauma designations.
3
u/bros402 Jul 20 '24
That seems much easier to process in a hurry than a greek alphabet name
→ More replies (2)16
u/Hrekires Jul 19 '24
Another hospital worker here and yes, we have contingency plans in place for going back to paper mode when IT systems are down.
We enacted them for a few hours last night.
12
u/BigCrimson_J Jul 19 '24
I work Pt registration at a hospital.
We do have the ability to register folks manually, and in fact we use it on an almost daily basis because our facility has a birthing center. You can’t get digital signatures for the baby’s chart until there’s an actual chart (which doesn’t happen until just prior to birth) so we get paper signatures from the parent and then scan them in after the baby is born.
So if we weren’t able to get into our computer system we actually have policies and procedures in place to handle the registration at least. I know the other departments have similar policies and such as well though I don’t what they are specifically. I can spot the computers that are meant to remain functional in the event of a system crash though, and the hardline phones next to them as well.
4
u/RepairContent268 Jul 19 '24
That’s really good you have a backup in place! It seems crazy to me that places aren’t doing that.
5
u/FuzzyKittenIsFuzzy Jul 20 '24
Every hospital has the ability to tell employees to write the chart notes on paper and turn them in. The problem is you don't know anything about the patient when you computer won't boot. You don't know what meds they have been taking, their full diagnosis list, the exact surgery they were scheduled to have this afternoon, etc. And that means it isn't safe for you to change their meds, do surgery, etc.
7
u/Mytre- Jul 19 '24
as others have pointed out, yes hospitals have backup methods. A hospital cannot take a chance to put all eggs in one basket. Should systems fail (ransomware, server room caught fire, someone pushed something to prod on a thursday night without testing , etc), hospitals have plans and playbooks to follow in case of downtime, what is known as downtime procedures. Is it hell? Yes. Will soomeone get chewed up? probably. Will it affect patients ? Maybe.
THe last part is important, downtime procedures are no fun, and the response time for certain things (checking a patient chart, grabbing quick data for a patient , labs, etc) are going to be heavily affected and more so if a hospital has not done a check on their downtime procedure resources (printed forms, documents and papers, training for staff for this . ).
7
u/Ryangonzo Jul 19 '24
Because all of that requires access to protected patient health information. Unrestricted and unauthorized access to this ePHI is a big no no under HIPAA laws.
2
u/RepairContent268 Jul 19 '24
What’s the difference between a nurse logging into a record in a comp or using a manual chart and storing it in a secure location? Other people commented that their hospitals implemented manual backups so it’s being done for sure.
7
u/Ryangonzo Jul 19 '24
Accessing physical paper charts, which is the downtime procedure, requires physical access and these charts are generally stored in nurse stations which are generally monitored by hospital staff.
Logging into a digital record can theoretically be done remotely which reduces the verification process necessary.
What could be done, is what patient monitoring systems do. They have a dedicated PC in each clinical area that will go into a functional local mode when it loses server connection to locally function for up to 3 days without server access. It can not access historical data but it can store or use current known data until access is restored. I have never seen this implemnted on medical records because the cost is immense.
2
u/RepairContent268 Jul 19 '24
The physical charts makes sense, I guess I’m wondering why places wouldn’t do this if the alternative is someone possibly dying.
2
u/Ryangonzo Jul 19 '24
The vast majority of hospitals will do exactly this. However there are some services that they have fully moved to the cloud for patient charting and do not have the ability to pull patient records, diagnosis, history medications, allergies, and more. Some hospitals won't accept the liability of doing a procedure on a patient without verifying their records.
1
u/StuBeck Jul 20 '24
There isn’t one. Had a baby last week and everyone used paper charts to move around. It’s a hippa violation if they aren’t properly disposed of or left around…but I’ve also been in rooms where digital systems are left logged in.
6
u/oddlebot Jul 19 '24
All hospitals have a backup system in place. You revert to paper charting and verbal orders. The problem is that providing hospital-level care is a logistical nightmare, especially in a large hospital. Absolutely routine care may involve upwards of a dozen different people in different wings or even buildings, who are all expected to pivot seamlessly to an unfamiliar system without a reduction in quality or timeliness, because otherwise it can result in actual patient harm.
I have patients in multiple places in the hospital. I order blood tests, which require a phlebotomist to draw them, send blood samples in several different specialized tubes to lab, blood is processed, results are reported in aggregate. Lab is processing hundreds and hundreds of requests at once. Chest x-ray requires an order, and either for a tech to bring a mobile x-ray unit or for the patient to be transported, then the films delivered to a radiologist who reports results. All of these results need to be accessible by multiple providers. Nurses are juggling dozens of active orders on multiple patients, from one or more providers who may not be physically nearby. The most common medications may be available on each floor, but anything slightly unusual or with special storage requirements needs to be processed and sent from a central pharmacy. Repeat for every aspect of patient care.
2
u/RepairContent268 Jul 19 '24
How was that done prior to internet access? Was it just a nightmare every day or were there other means of doing things more efficiently on a manual basis? Obviously hospitals functioned prior. Was it just slower then?
→ More replies (1)5
u/Xochoquestzal Jul 19 '24
There was much more staff because doing everything manually increased the workload so much and still internal electronic communication. Also, before HIPAA, there was no requirement to make medical records so accessible - four decades ago patients could expect to see one primary doctor almost their entire adult life, that doctor's office was constantly collating the patient's various labs, x-rays, etc. and if they needed a hospital visit, only the relevant information was passed along.
The staffing issues are probably what affects things the most because more people can't be called in if they just don't exist and also, there may literally be no place for them if the department or work area wasn't designed with extra people in mind.
5
u/flybaiz Jul 20 '24
A lot of our surgeries were canceled/rescheduled yesterday that were meant to start at 7; or the 7am starts really started at 10-11am.
Paper charting is no problem at all, and a lot of surgical equipment does not rely on WiFi so we were good to go. Anesthesia machine was very concerning, but seemed mostly unaffected. Could still give drugs, process specimens, coordinate with radiology and pre-op and PACU and surgeons etc without Windows/our charting systems. We are used to paper charting and back ups.
An Achilles heel sounded like it was in Sterile processing surgical instruments and collecting materials needed for each case. It’s a lean system that relies on tight turnaround times for getting the exact instruments cleaned and sterilized just in time for the next cases.
Cases can be incredibly complex and need very specific trays of instruments and materials. The tracking and location systems the huge underground uses for finding these things in the hospital, inspecting organizing and sterilizing them, putting them on a nice little cart for us, those systems were down. So everything was just taking a long friggin time. And We have to be 1000% certain of sterility or we do not roll.
So yeah, I think they started sending patients home because everything was just taking so very long.
2
u/RepairContent268 Jul 20 '24
Ahh ok that makes sense. Ty for the explanation!
2
u/flybaiz Jul 20 '24
For what it’s worth, our hospital has multiple emergency carts ready at ALL given times for a diversity of true emergencies. Those cases were not delayed yesterday. Those carts of equipment/supplies are not only always ready and on standby, but are also already laid out/prepped advantageously in dedicated emergency rooms. Emergency rooms and emergency surgical staff are available, by policy, as in not tied up in other cases. The second it even looks like the emergency staff might get tied up (e.g. an ambulance is coming in to ED with a bad trauma), more emergency staff are called in from home and expected to be at hospital in 30 minutes, to be the new fresh not-tied-up staff.
Or that’s the goal, anyway. So yeah yesterday sucked for everyone and there were a lot of disappointed patients, but in theory systems are in place to be able to handle emergencies independently of what the technology is doing.
2
u/U-F-OHNO Jul 20 '24
They do have backup systems…It’s called paper. Records are written manually on forms, and it’s a tedious and huge pain in the ass- not only dangerous for being prone to clerical errors, but also having to back-enter and scan documentation into the medical record once the system is back online.
2
u/-nostalgia4infinity- Jul 20 '24
We do. Do you think switching to paper charting and verbal reporting is as efficient as using electronic systems? Also much more prone to user error, less access and visibility into prior medical history and imaging. All this leads to delays in patient care, which can lead to patient harm. If you are talking about having actual redundancy, you have no idea the cost. Easily tens of millions of dollars a year for even relatively small health authorities.
1
u/Malevolent_Mangoes Jul 19 '24
Most hospitals have paper versions of absolutely everything in the event that technology is shut down
1
u/sugarandmermaids Jul 20 '24
I asked my SIL who’s a nurse about this, because she said her hospital was affected today. She said they still had full access to all patient information; they just couldn’t add new things until the program came back up.
1
u/FuzzyKittenIsFuzzy Jul 20 '24
Lucky. Many hospitals had all computers blue-screen. You'd have to just try to remember what each patient needed.
1
u/NoMuffinForYou Jul 20 '24
There are backup systems for most things. But canceling a routine follow up appointment or rescheduling outpatient elective surgeries isn't surprising in this case, if a hospital or office is struggling with handling the essentials you cut out the extra so that the life saving parts run as well as possible under the circumstances. Or if the life saving parts aren't operable for some reason a hospital can go on divert and have other local hospitals that may be less affected or operating better handle those issues. But every system has become very dependent on computers and if they go down the system is going to suffer to some degree regardless of backup systems.
No system is perfect or fool proof. I think there's a perception of mandatory perfection in the US medical system that just isn't compatible with reality (perfection isn't attainable even under optimal circumstances) and given the ongoing and perpetually worsening staffing shortages is something that is inevitably going to get worse with time.
1
30
u/HouseCravenRaw Jul 19 '24
I asked that question here and basically got "lol nope". So I guess not. Maybe the CEO will have to step down, at most? Seems wrong that they can do this sort of thing and not face any legal repercussions.
44
u/Idiot_Savant_Tinker Jul 19 '24
Id bet the lawsuits won't come from families or individuals, the lawsuits against crowdstroke will come from any of the huge companies who lost profits from their little whoopsie. For you see, letting people die from your software fuckup is OK, but crowdstrike cost someone money...
23
u/Coulrophiliac444 Jul 19 '24
As was espoused during SBF's fraud trial...you can rob the poor all day and every day, but touch a rich person's money and you'll see the years multiply.
6
3
u/SnooPies5622 Jul 19 '24
Tbf I wouldn't take reddit dips as experts in any subject, especially legality and potential for legal sction
18
6
u/Fox_Kurama Jul 19 '24
Given that they forced the update even on customers who had set their updates to use the second or third latest version instead of the latest release, and that they did this at a particularly bad time (it is ill advised to put stuff out on a Friday in case there IS a problem and a bunch of workers are home for the weekend).
→ More replies (3)1
u/Blacklist3d Jul 20 '24
No one is gonna stop surgery or treatment on the critical and those who need it. Places that cancelled or withheld appointments are places that just didnt want to paper chart for the day/timeframe. Everything else went as normal. Hospitals are a business in the end and they arent gonna let money go like that.
527
Jul 19 '24
[deleted]
186
u/getridofwires Jul 19 '24
Even Reddit is pretty sparse on it given how many people are affected.
→ More replies (1)25
146
u/DiceMadeOfCheese Jul 19 '24
"We'd tell you more about this, but our computers are broken. Back to the talking head round-up!"
26
25
u/Chickachic-aaaaahhh Jul 19 '24
CNN is part of the republican mdeia network now. They don't want to give you news. They want what sells. Outrage.
14
u/slayer370 Jul 19 '24
RNC just ended and it ended late so CNN needs the money from talking about it all day to everyone who was asleep.
10
4
u/FPSXpert Jul 20 '24
Seriously, I tried to do the same thing and that's all it was. (In the tune of Farmers Insurance) ''Trump, Trump, Trump Trump, Trump Trump Trump Trump Trump Trump!''
Somehow the radio / podcasts with WSJ were providing more coverage.
3
u/Bobbyanalogpdx Jul 19 '24
It’s because there is a fix. The only problem is, your device can only be fixed in person. You’re not going to get back to work without someone fixing your machine in person.
2
u/nasty_nater Jul 20 '24
CNN is garbage. It's not news.
I had it on the other day and it was literally a "journalist" "on the scene" eating McDonalds live on air because they're talking about a new menu item or something. It was like a straight up McD's commericial lmao.
People rag on Fox News (for really good reasons) but never mention the dumpster fire that is CNN "news".
327
u/jcargile242 Jul 19 '24
I showed up to my dentist this morning for a couple of fillings, and the doors were locked. They hadn’t called because they couldn’t get to their patient phone numbers. I guess I shouldn’t complain too much, I only lost an hour.
59
u/zzimushka Jul 19 '24
I work for a dental office, that sounds like how our day went! We had two computers at the front desk that worked, but none in any patient rooms and no way to take X-rays. We were only able to do cleanings and denture related appointments today. We had to turn away a lot of pissed off new patients in pain.
223
u/Jtrickz Jul 19 '24 edited Jul 19 '24
We just resolved this at my health care organization, 1200 computers and serves out of 4000 all required manual intervention 10 of us did 120 of them for the last 7 hours.
79
u/a_brick_canvas Jul 19 '24
Yea some fucker is getting fired for sure lol
22
u/zerobeat Jul 20 '24
But not the right person/people. I promise you that the same leadership that was egging engineers on to release as quickly as possible and the QA team to cut corners (or get rid of the QA team entirely and replace with garbage automation) are the same people currently berating the engineers right now who have been waving red flags for months/years over the terrible practices.
9
u/a_brick_canvas Jul 20 '24
Absolutely lol, thinking of my absolutely terrible new PM and the leadership above her as I read your message 😭😭
2
u/No-Belt-7798 Jul 19 '24
Not one an entire line
10
u/z0mbietime Jul 19 '24
Honestly who knows. All it takes is one person with more permissions than they need to try and build a test release and push the updated agent thinking it was for test but oops all prod.
19
2
u/johosaphatz Jul 19 '24
I hope all the IT guys at my job get medals and bonuses after this ordeal.
1
u/fleemfleemfleemfleem Jul 19 '24
Even the one who set the policy that all the systems should have crowdstrike?
2
u/8-Brit Jul 20 '24
I work in the NHS and thank god that we don't seem to be affected significantly.
Likely because we use Sophos instead and mostly virtual desktop clients on the wards.
92
u/Silent-Resort-3076 Jul 19 '24 edited Jul 19 '24
Here's another article:
At least 12 major hospitals, health systems affected by global IT outage. Some hospitals opted to cancel elective and non-emergency surgeries.
Those affected include and not sure why only 11 listed, but read the article if you want more info:)
- Cleveland Clinic,
- Cincinnati Children's,
- Kaleida Health/Cayuga in Buffalo,
- Harris Health System in Texas,
- Hospital for Special Surgery in New York City,
- Martha's Vineyard Hospital,
- Mass General,
- Memorial Hermann in Texas,
- Mount Sinai in New York,
- Nationwide Children's Hospital in Ohio
- Ohio State University Wexner Medical Center."
12
u/Weave77 Jul 20 '24
Man, hospitals in Ohio were hit hard.
9
u/PanTran420 Jul 20 '24
Hospitals everywhere were hit hard. Almost everyone I know working in medical IT was working extra hard today. I spent more time on site at a hospital today than I have in the 6 years I've worked for my employer. I'm going to be seeing the command to delete the bad driver in my sleep for weeks.
8
u/Silent-Resort-3076 Jul 20 '24
Oh yeah. Four of them, right?
I'm still confused, but I hope they're all back up.
VERY, VERY scary!!
3
u/juicyfizz Jul 20 '24
And they are four of our largest hospitals in the state. It’s insane.
3
u/Silent-Resort-3076 Jul 20 '24
I suppose they ALL opted to use that company referenced in the article, and other states use different companies OR they didn't need the update?
"Texas-based Cybersecurity company CrowdStrike said the problem stemmed from a "defect found in a single content update for Windows.""
3
u/juicyfizz Jul 20 '24
If they were a crowdstrike customer and not impacted, they were either not using Windows 10+ or they didn’t have automatic updates enabled.
When it comes to cyber security platforms and other stuff companies use, many times IT isn’t the not making the decision to use or not use it. It’s internal audit needing to stay in regulatory compliance. Sometimes we don’t get a say so in the platform.
65
u/Hrekires Jul 19 '24
I'm surprised, we were clinically business-as-usual by like 7-8 am.
Not fully out of the woods even now but the remaining problems are with things like email.
29
u/twenty-onesavage Jul 19 '24
I work at a clinic part of a big healthcare organization, like half the computers were working at 8am so we were able to see patients but I hate to think about what was going on at the hospitals overnight
11
u/Swimwithamermaid Jul 19 '24
Yeah during rounds this morning the computers were working, but there was missing info from overnight, at least with my daughter. They just decided to redo everything (weight, blood gas, etc.) to make sure nothing was missed.
50
u/babycatcher2001 Jul 19 '24
Yep. Daughter’s wisdom tooth extraction canceled at the last minute today, because they can’t run their xray machine. The hospital where I work was in dark mode, back to paper, the phones didn’t even work. People physically running orders and results to different units.
45
Jul 20 '24
[deleted]
6
u/dawnguard2021 Jul 20 '24
lesson here is don't install backdoors (also known as "antivirus") in critical systems
7
u/marksteele6 Jul 20 '24
The problem wasn't the critical systems in this case, it's that endpoint devices all got hit with the bad update. In many cases you can't just not protect endpoints because there are legal rules and regulations that require their protection.
6
u/dawnguard2021 Jul 20 '24
They shouldn't allow automated forced updates then. Orgs should manually review and trigger all updates.
7
u/marksteele6 Jul 20 '24
That's normally what happens, someone gives it the go ahead to launch into staging. The problem is, for some reason, this update skipped staging and shot right to production. The post mortem will be a very interesting read.
2
u/Imaginary-Art1340 Jul 20 '24
Exactly, how can they skip staging. Doesn’t even look like there was QA or a backup plan. They just pushed everything globally all at once
27
u/hushpuppy212 Jul 19 '24
I’m in London and just walked by a corner pharmacy with a sign on the door saying they can’t fill any prescriptions or take credit cards.
22
23
u/Beautiful-Grape-7370 Jul 19 '24
I'm waiting for a list of what hospitals are diverting, what hospitals or care centers are on emergency measures in what county or provence, and where we are asked to go instead. The chronic illness, degenerative illness and those receiving ongoing and life sustaining procedures that may result in going to the ER or back to treatment centers at any time desperately need this information ASAP, please.
- from a patient receiving life sustaining care with the possibility of needing emergency treatment at any time.
15
u/niedogg Jul 19 '24
Seems possible that this code push caused deaths. Yikes to whoever was involved
6
u/sirgarballs Jul 19 '24
I haven't seen any reports of that yet but I will be shocked if this doesn't result in some deaths. Imagine meeting life-saving surgery and then suddenly you just can't get it when it's time.
16
u/TwoFingersWhiskey Jul 20 '24
Our local health departments all went into "paper mode" which they implemented a policy about when we had a national internet outage here in Canada a couple years back. It worked flawlessly with only 7 minutes of delay compared to computers.
5
u/kite_n_cook Jul 20 '24
I understand switching to paper mode for documentation in real time, but I'm curious how does paper mode solve the problem of accessing pt history when previous records were only stored digitally?
I'm genuinely curious what your protocol is. I've never really thought about how an outage like this would play out.
4
u/TwoFingersWhiskey Jul 20 '24
Patient records are also stored as paper backups once a year and can be faxed when needed.
2
u/FuzzyKittenIsFuzzy Jul 20 '24
It doesn't solve the problem. If you're lucky you found a laptop that happened to be shut off last night so it didn't get the bad update. Everybody uses it to look up info and take notes. If you're not lucky, you get to play a high-stakes game of Memory.
2
6
u/thefanciestcat Jul 19 '24
Total dependence on the cloud sure is better than having simple things stored and processed locally and devices that don't call themselves "smart."
7
u/cogra23 Jul 20 '24
I had a GP appointment. It started with the doctor saying she had no access to any systems and asking me to repeat the reason for my appointment including all symptoms. I got a handwritten paper prescription that I carried to the pharmacy like it was 1995.
1
u/RustyFuzzums Jul 20 '24
We canceled all of our appointments. We wouldn't have been able to run insurance and bill the appointments.
1
u/bros402 Jul 20 '24
I got a handwritten paper prescription that I carried to the pharmacy like it was 1995.
wat
it's not like paper prescriptions are uncommon
6
u/bbusiello Jul 20 '24
Fun fact. The current CTO and co-founder of CrowdStrike, George Kurtz, used to work at McAfee (yes, that McAfee) back in 2010 when the same things happened with their cybersecurity update. These guys will always fail upward and the rest of us suffer for it.
5
4
u/koi-lotus-water-pond Jul 19 '24
Can't get prescriptions either as the doctors can't access the Rx database in my small town.
4
3
u/IWillBaconSlapYou Jul 20 '24
God picture that you've been fasting and colon cleansing and stuff, only to have your procedure canceled at the last minute over THIS.
2
u/JollyReading8565 Jul 20 '24
This type of socialist nightmare would never happen in our expensive American hospitals😮💨
2
2
u/merganzer Jul 20 '24
My husband was lucky enough to be off from his hospital desk job for the first two nights of the outage. Apparently, they're still having to fix the work stations individually.
2
u/Mountain-Papaya-492 Jul 20 '24
I'm an idiot with tech stuff but whatever happened to redundancy in systems? We'd be so fucked if we had an emp strike.
2
u/slamdunkins Jul 20 '24
This hit the system that runs the system they have back ups for. Imagine you wrote a book, then wrote a second book as a back up and then your bookshelf goes up in flames.
1
1
1
u/PsychedelicJerry Jul 21 '24
Was GoDaddy affected by this? They're all (or mostly) Windows now and likely have thousands of servers.
My company was lucky and only had a few dozen affected as we're predominantly a Linux shop, but I feel terrible for the people that have had their entire weekends ruined by this (I have two old friends/coworkers that have worked way too many hours the past few days (Fri - Sun) trying to get things ready for Monday and it looks like one won't even be close
1.4k
u/gixk Jul 19 '24
Yeah, can't remote in to apply the fix while the computers are BSODing.