Only time will tell whether July 19, 2024 will be a day that will “live on in infamy,” so to speak.
But anyone who works at Microsoft or CrowdStrike will likely remember that day with horror for years to come.
On the day mentioned above, a software update from previously respected cybersecurity giant CrowdStrike caused outages of Microsoft systems worldwide and even affected air traffic.
It’s still unclear whether the blame for this debacle can be placed solely on one person, but it’s safe to assume that many Microsoft or CrowdStrike employees were shaking with fear, wondering if they should go back to work the next day.
The incident piqued the curiosity of Redditor RATTLECORPSE, who wanted to know what massive, potentially career-damaging mistakes others might have made at work. He asked:
“In honor of CrowdStrike: What was YOUR biggest mistake at work?”
This is a great boss…
“In 2008, at my new job at an investment bank, I spent over $400 catering an all-day meeting including lunch for 12 people.”
“~$33 per person.”
“The food arrived and the office manager asked me which meeting the food was for because it wasn’t recorded in our meeting setup system.”
“I realized the meeting was the next day and I had tons of food for no one.”
“I asked them to bring all the food and put it under my desk.”
“She looked at me sideways and did it.”
“I had just come from a government environment where such spending was exorbitant and a problem.”
“I looked at the food and thought about it for about 20 minutes.”
“What to do???”
“I finally decided to go to my boss and tell him what I had done.”
“‘You have the food under your desk,’ he said?”
“‘Yes,’ I replied.”
“I expected a lot of suffering.”
“Show me,” he said, walking over to my desk, looking and laughing.”
“Put it in the kitchen and send an email to everyone in the company to come and help themselves, he said.”
“Order it again for tomorrow,” he said, giggled and walked away.” – Plus attachment
There is so much to be thankful for!
“A bug was reported that caused our sales software to silently refuse to process credit cards while simultaneously reporting that the credit card was processed correctly… on Black Friday.”– flakey_biscuit
Why it’s always worth checking three times…
“Allowed a contractor to pour a slab on a huge building before the plumbing was in the ground.”
“$200,000.” – PippyLongSausage
Concrete Cement GIF by JC Property ProfessionalsGiphy
It’s easier than you think…
“The entire inventory database has been deleted.”
“I didn’t know how to get reinforcements.”
“Thank God there are nice IT people.” – Teasing girlfriend
Honesty ALWAYS lasts the longest
“I broke $10,000 worth of PA speakers after I made a hip-hop artist’s sound engineer run the system too loud for an entire show.”
“When my bosses told me it was $10,000, I thought they were exaggerating.”
“But months later, I broke my ankle and had to do desk work.”
“I had access to the accounting software and found the repair invoice for these speakers.”
“Legitimate $10,000.”
“I was honest and open about what I had done and they respected that, so I wasn’t fired.” – EightOhms
If in doubt, keep politics to yourself
“I used a picture of George W. Bush with his face half rotted away as a sample image for some HTML work I did in 2006.”
“The developers were supposed to replace it with the real image, but they simply put an image from the database in front of it, and instead the bush image was still there as a background image in the CSS styles.”
“For quite a while everything was running fine and nobody noticed until the database server went down and the entire homepage of the website was covered wall to wall with pictures of the president who, according to my boss, looked ‘like his face had been shot out with a shotgun.'”
“Higher-ups in the company wanted me fired, but fortunately the people on my team protected me.”
“I had my first real job after college as a website developer.” – Subscribe
George Bush dancing GIFGiphy
A small mistake in the midst of a lot of bigger ones, it seems…
“I can’t confirm that this is really my fault because we were never blamed for it.”
“The equipment was old, but what I did could well have caused it to fail.”
“We planned a power outage to do electrical work at a facility owned by a multinational company you’ve probably heard of. We’ve been planning this for weeks.”
“The data center at this facility was backed up by a large UPS battery storage and we had to bypass it.”
“There was no practical way to provide backup power, so that’s why we shut it down and installed a more robust redundant backup power supply.”
“So they planned a soft shutdown of their data center for eight hours on a Saturday.”
“The data center is always powered by the batteries and the regular power just charges the batteries, so if there is a power outage there is literally no interruption.”
“The first thing I did was turn off the power to the data center, which was powered by the batteries.”
“Then I turned off the normal power that charges the batteries.”
“No power in, no power out.”
“We do the work, we want to turn it back on, but it won’t turn back on.”
“The emergency number will be called and a technician will be dispatched immediately.”
“Bad motherboard.”
“That same evening, they have an employee from the manufacturer board a plane with the replacement part to fly across the country and deliver it in person several hours later on Sunday.”
“We learned that the motherboard failed due to overheating.”
“We also learned that the cooling fans are powered by the same batteries that power the data center.”
“So if you (hypothetically) turn off the power and the battery power at the same time, there’s nothing left to power the fans that cool the CPU, which stays very, very hot after it’s turned off.”
“At the beginning of the project, we were told that the penalty for a one-hour interruption in service would be approximately $200,000.”
“On Sunday, they were about an hour away from telling their employees to stay home on Monday.”
“This mistake would definitely have caused a six-figure or even a seven-figure sum.” – RogerPackinrod
It’s amazing how much trouble a single finger slip can cause…
“Charged someone $2,000 and tried to refund them a much smaller amount (I’d say around $400).”
“The manager was out of town for a holiday weekend and the problem took about four days to resolve.” – DismalTree4161
Messed up or reflexes?
“It wasn’t necessarily a failure, but when I worked at McDonald’s almost 20 years ago, a customer in the drive-thru threw a chocolate shake at me and screamed that he had ordered strawberry.”
“I instinctively caught him and threw him back, hitting him in the face and splattering the interior of his BMW with chocolate shake.”
“If my GM hadn’t been standing damn close to me when this happened, I probably would have been fired.” – FlirtyFairy
happy Ronald McDonald GIF from McDonald’s CZ/SKGiphy
The opposite of mass communication?
“I deleted the email app from all phones across the company and the issue had to be resolved manually.”
“There were about 2000 people.” – fiasco
Even the projector knew the film was bad…
“I worked as a projectionist in a cinema in the early 2000s and we had to check everything in advance before it was shown to the public the next day.”
“We were supposed to stay in the projection booth to supervise the reel (it was still real film back then), but often we would just go into the cinema itself to watch the films in advance.”
“Halfway through The Hulk (Ang Lee’s film), the film froze in the projector and ruined about 20 minutes of the film.”
“Each roll of film was worth $150,000 (maybe not? I was 16, that’s what my manager said, lol).”
“Of course, those were bad times.” – Arcane Psyche
Whatever happened to “What’s mine is yours”?
“I made an unsolicited call to a bank about home loans.”
“I suggested to my wife that she could pay off the $10,000 balance on her credit card at a much lower interest rate and thereby save a lot of money and possibly get a tax break.”
“I had no idea she wasn’t on the credit card, and had no idea her husband was hiding it from her.”
“The man left me a very nice voicemail.” – BathroomInterior2036
Embarrassing Jay Z GIF from ComplexGiphy
Always choose your words carefully
“The application service provider I used to work for called me very early one morning.”
“Customers reported a total loss of service and temperatures went through the roof.”
“The pager was constantly ringing.”
“We have to deduct downtime.”
“Fifteen minutes later I called the CTO and woke him up.”
“I said, ‘Did you happen to forget to renew our DNS registration?'”
“It was the loudest scream I’ve ever heard.”
“That was peanuts compared to the intercontinental chaos that Crowdstrike caused.” – phil_mckraken
Well, SOMEONE screwed up…
“I replaced a switch in a data center with a spare switch (which they said was to be used via serial number) following the instructions and documented instructions, which resulted in about two million users being offline for a few hours before they fixed the issue.”
“The network diagrams were amazing.”
“I never heard from this admin again.”- S_SubZero
A happy ending for everyone?
“I was the one who suffered.”
“My wife and I bought a hot tub at the State Fair.”
“One year later, I see a credit on my card for the full price of the tub from the manufacturer.”
“Because we are good people, we do the right thing and call them.”
“I explained exactly what happened.”
“We specifically stated that we had the tub and were happy with our purchase.”
“A few days later we received an invoice with a full refund and a zero balance.” – Jedly1
Happy Lisa Simpson GIF from The SimpsonsGiphy
Who hasn’t made one or two mistakes at work, so a certain degree of leniency is to be expected.
However, if your mistake is known not only to your colleagues but to the whole world, it is probably a good idea to start cleaning up your desk.
