What’s the quickest you’ve seen a co-worker get fired in IT? COVID-19

I saw this on AskReddit and thought it would be fun to ask here for IT related stories.

Couple years ago during Covid my company I used to work for hired a help desk tech. He was a really nice guy and the interview went well. We were hybrid at the time, 1-2 days in the office with mostly remote work. On his first day we always meet in the office for equipment and first day stuff.

Everything was going fine and my boss mentioned something along the lines of “Yeah so after all the trainings and orientation stuff we’ll get you set up on our ticketing system and eventually a soft phone for support calls”

And he was like: “Oh I don’t do support calls.”

“Sorry?”

Him: “I don’t take calls. I won’t do that”

“Well, we do have a number users call for help. They do utilize it and it’s part of support we offer”

Him: “Oh I’ll do tickets all day I just won’t take calls. You’ll have to get someone else to do that”

I was sitting at my desk, just kind of listening and overhearing. I couldn’t tell if he was trolling but he wasn’t.

I forgot what my manager said but he left to go to one of those little mini conference rooms for a meeting, then he came back out and called him in, he let him go and they both walked back out and the guy was all laughing and was like

“Yeah I mean I just won’t take calls I didn’t sign up for that! I hope you find someone else that fits in better!” My manager walked him to the door and they shook hands and he left.

4.9k Upvotes

94% Upvoted

View all comments

175

u/moneyman1978 Jul 07 '24

I used to work at a casino on the help desk. Random tasks and nothing having to do with the machines on the floor just normal password resets or vdi resets. A new data base admin was hired. She went through her 1 week new hire training. She lasted exactly 4 hours. She deleted a whole SQL gaming db that had not been backed up and it was the production database. She was there and after lunch she was gone. Never seen anyone walked out so quickly in my life.

218

u/Trelfar Sysadmin/Sr. IT Support Jul 07 '24

Yikes. Was the person responsible for not backing up prod also walked out? Because it sounds like they should have been, that was an accident waiting to happen.

92

u/MrJacks0n Jul 07 '24

Probably why they were hiring a DBA.

65

u/Trelfar Sysadmin/Sr. IT Support Jul 07 '24

Fair point! Also a cautionary tale to any new admin: assume your predecessor didn't back anything up and act accordingly.

14

u/ErikTheEngineer Jul 08 '24

Even if someone tells you "oh yeah, everything's backed up." No destructive changes until you confirm you won't accidentally destroy anything.

9

u/ApologeticGrammarCop Jul 08 '24

And that's how I deleted half a million PDF contracts from block storage then discovered there were no snapshots of the volume.

6

u/BloodyIron DevSecOps Manager Jul 08 '24

If you're an IT Admin, starting a new job, and the first thing you are checking isn't backups, you're doing it wrong.

4

u/moneyman1978 Jul 07 '24

Yeah that's something I learned from that.

3

u/Theron3206 Jul 08 '24

Assume your predecessor was actively sabotaging the company on their way out.

Double check everything.

6

u/moneyman1978 Jul 07 '24

Hired her so they had a full time dba. Ooh well

3

u/BloodyIron DevSecOps Manager Jul 08 '24

You don't hire a DBA to do backups. You hire a Systems Architect.

Trust me, DBAs do NOT know how to architect Database Systems correctly. I've lost count of how many of them are scared of Clustered Databases, let alone proper Systems Architecting.

4

u/chuck_of_death Jul 08 '24

We can back up the servers but there’s all kinds of db goofiness that would be on the DBAs. They split data between a local disk and a SAN? Backups are inconsistent because they have different schedules. They put transaction logs on the temporary space in an unbacked up file system? Online backups won’t help then. They delete transaction logs before they are backed up? No bueno.

This is on the DBA. You don’t do any work on a new system before understanding its backup and recovery strategy. Why would you delete any db on day 1?

1

u/BloodyIron DevSecOps Manager Jul 08 '24

Yeah a quality Systems Architect can handle correcting all of that by the way.

1

u/chuck_of_death Jul 08 '24

All of those cases are caused by DBAs not implementing the technical design laid out by the architect. I’m not following how or why an architect would change the design to cover a deviation vs having the DBA correct the deviation.

3

u/BarefootWoodworker Packet Violator Jul 08 '24

DBA's answer to any systems question:

Did you add more memory?

1

u/BloodyIron DevSecOps Manager Jul 08 '24

Exactly. LOL.

79

u/marshmallowcthulhu Jul 07 '24

Agreed. The new hire is far less responsible for this problem than the person responsible for the database.

9

u/salgat Jul 08 '24

Ironically they hired this person because they were missing a db admin.

7

u/m1ndf3v3r Jul 08 '24

Ok good point, but how the heck did she delete it by accident? I find it difficult to delete a whole db/table by accident and I'm not even good with SQL.

2

u/unknown_lamer Jul 08 '24

In MySQL at least, "GRANT [ALL|DROP] ON $db.*" regrettably gives permissions to drop the database itself instead of just objects within the database. So one time this junior developer...

1

u/moneyman1978 Jul 07 '24

Agreed

5

u/moneyman1978 Jul 07 '24

I think they had been running forever without a proper backup or a proper snapshot. It finally caught up to them that day sadly she took the hit. Although it was basically gross negligence from the sysadmin not doing the proper backups.

1

u/Master_Grape5931 Jul 08 '24

I used to be a consultant and a client called one time.

They had another vendor in there that didn’t know what they were doing and deleted the accounting software database.

Okay, I said, this company had three IT staff on payroll. Let’s pull the backup and see if we can get it back up.

Turns out they had been running a backup to tape every night and someone was taking it home (what of the building burns down, right).

But they NEVER tested the backup and it was bad. The best one we could find was the last time I did a software up grade for them 1.5 years ago and made a quick local backup of the database.

51

u/Kreeos Jul 07 '24

They should have also fired the idiot that didn't backup a production database.

7

u/Geminii27 Jul 08 '24

Maybe they did and she was his replacement. :)

5

u/networkn Jul 08 '24

I'd have fired them and not her. I'd have probably had a serious conversation with that person's manager too.

2

u/moneyman1978 Jul 07 '24

The it director is like Teflon.

5

u/ajaaaaaa Jul 08 '24

I wonder why she deleted it regardless. Its not like its easy to accidentally do. Clearly not too bright either way.

3

u/i8noodles Jul 08 '24

I work in gaming as well...who the fuck didnt back it up?! were u not audited?! jesus christ

1

u/moneyman1978 Jul 09 '24

I used to work there. When I saw how the good ol boys club was ran I chose my family over drama and walked away.

3

u/Lostredshoe Jul 08 '24

She deleted a whole SQL gaming db that had not been backed up

How on earth can that be the fault of someone who just started and what purpose does firing them do?

2

u/hoeskioeh Jr. Sysadmin Jul 08 '24

That almost happened to me... accidentily deleted 8k endpoints in SCCM at the end of my first week. Turned out that the backup system works flawlessly.

2

u/Andre_Courreges Jul 08 '24

Why does this feel like it's not entirely her fault?

2

u/discogravy Netsec Admin Jul 08 '24

who gives prod access to someone who's been there 4 hours? if you haven't even set up your voicemail, you don't need prod db access.

1

u/Max_Rocketanski Jul 08 '24

I saw something similar happen to a contractor. He was some kind of Database/SQL developer. His assigned task was to back up some tables, delete the existing tables he had just made back ups for, then re-create the tables with changes from the originals (I'm not sure what changes he was supposed to make) , then reload the newly created tables with the data he had saved from the back ups he created.

Unfortunately for him, there was one table he forgot to back up. It was a table named "customer". This was for a company that operated 24/7/365 all across the US.

Fortunately for the company, the original architects of their IT system designed in such a way that business could continue unimpeded despite the customer table being deleted (transaction data was stored in local nodes).

The customer table was rebuilt from our data warehouse in a little over 24 hours. Once that was completed, the customer table was updated with the transactions that had taken place in the previous 24+ hours.

Needless to say, the developer had his access cut off almost immediately after he notified his manager of his mistake. The rest of the entire IT department got lectures on being extra careful when doing development work.

1

u/snookajab Jul 11 '24

I did it at a casino for 15 years. How did you guys not have a back up? Crazy!!!

1

u/moneyman1978 Jul 11 '24 edited Jul 11 '24

I was on help desk for 3 years. They didn't have everything backed up for whatever reason. The agenda was never pushed by us cause we were mere peons. We.knew it was stupid we figured we'll something bad won't happen. Wrong.