From out of control on-premise fires to misplaced tequila bottles that delete customer data [NSFW—language!], there’s a lot that can go nightmarishly wrong in the tech world. While we don’t often want to ruminate on what’s beyond our control, Halloween is the perfect time to face our fears, and entertain ourselves in the process.
In celebration of the spooky season, below are IT horror stories to give you the chills, or maybe just a laugh.
Dropped Database
Our first story comes from Paweł Kmieć, a senior managing developer, who shared with ThinkIT an all-important lesson about knowing your environment. Kmieć was developing a reporting application. The deadline was drawing near and everything was going well. It was time to move the application from the development environment into production.
As Kmieć tells it, “I happily launched all necessary data imports and spent quite a lot of time manually verifying and cleaning data afterwards. The day before [the] demo to our CFO, I mistook the dev database with the newly created production database and issued a statement which still haunts me ten years later: DROP DATABASE.”
Kmieć reports that he hasn’t developed in production since this incident.
Beer Goggles
Easy jobs can sometimes end up being harder than expected. Egor Shamin, System Administrator at ScienceSoft, shared with us a story from 2012 at a previous job. He and a coworker traveled to do what he describes as a “cushy job”—building a network for 30 PCs.
The first day of work went smoothly, and they only had to pin out network sockets and install patch panels, so they chose to spend the rest of the night relaxing. After sharing a few drinks together, Shamin turned in for the night, but his coworker kept on drinking.
“We were both late to work the next day, which was already a bad start,” Shamin recounts. “But what made it worse was that we needed to run one more wire, and the line we had physically didn’t allow it. My partner decided that he’d be able to neatly widen the line with the drill. Thanks to the nine beers he’d had the previous evening, all the guy managed to do using the drill was cut down all of the wires.”
The pair ended up working late into the night making wire twists since they didn’t have any other cable. Shamin says that, to his credit, his coworker did most of the job himself. And in the end, the network worked perfectly fine.
Getting Squirrely
For all of our efforts to control our environment, nature often has other plans. Did you know that squirrels are among the reasons that data centers sometimes go down? Reddit user JRHelgeson had a brush with a squirrel squatter himself while moving customer equipment into a new facility.
There were a number of things he saw that could go wrong. The furry creature might burrow under the raised flooring and build a nest in the fiber bundle. Or he might have the run of the data canter by getting up on an overhead ladder. JRHelgeson knew his team had to act fast to evict the gate crasher.
“We had guys with packing blankets moving in from three sides to get him cornered and he scurries up to the top of a rack filled with customer equipment. As they are closing in, the squirrel starts growling and—preparing for a fight—empties his bladder, raining down on the servers in the rack.”
The team finally caught the squirrel and got him out. Fortunately, no real damage was done since the top of the servers were sealed. From that day forward, if the outside door is open, the interior door is closed, or an alarm will go off.
The Little Patching Server That Could
The web has a treasure trove of nightmare stories if you know where to look. The next story, featured on Global Knowledge, was shared by Derrick B., and serves as a reminder to never do patching work in the middle of the business day, even if the patching isn’t expected to have a major impact.
Derrick’s team was all set to patch Windows servers. Change Management approved their patching and timing, which was intended to occur after hours to minimize impact. Everything started off well, until the patch management server crashed and wouldn’t reboot. Calls were put out to hardware support for troubleshooting, but the field tech didn’t arrive until the next day. The tech ran diagnostics and solved the problem, which only led to bigger issues.
“The monitoring console lit up like a Christmas tree and pagers started going off all around the server and application admins’ aisles,” Derrick says. “Servers were going down all across the enterprise. Senior managers started calling to find out what the heck was going on. Customers were being impacted as applications became unavailable.”
It turns out the patching server just wanted to do its job. It has set a flag to run the patching jobs before crashing and picked right up where it had left off as soon as it was repaired.
Not-So-Experimental
Cleaning up databases should always be done with great care. GlowingStart founder Alex Genadinik shared his horror story with Business News Daily, recounting a time when he accidentally deleted around 26,000 business plans on his apps.
“I was cleaning up some old tables in a database and noticed a table with a weird table name. I thought that it was something experimental from a long time ago and deleted it,” Genadinik says. “Then, literally five minutes later, I started getting emails from my app users that their business plans were deleted.”
With the number one function of his apps wiped out, Genadinik had a big problem on his hands. He went to work with his hosing provider and was able to have the database restored within a day after paying a fee. Talk about a scare!
Gone Phishing
Unfortunately, the next story is becoming an all too common occurrence for IT professionals. Our own Victor Frausto, security engineer, shared a past incident with a phishing email that was spammed to employees from one user’s account. Even though the attempt was caught early and minimized, the resulting work to reset the account and ensure the malicious email didn’t spread made for an eventful day at work.
“We had to disable the compromised account, scan the laptop, re-enable the account and change the password,” Frausto said. “And then we had to that for anybody who opened the spam email and clicked on the malicious link. Scary!”
Sometimes, the scariest thing in tech can be the feeling that you have to go it alone. Check out INAP’s managed services and disaster recovery options to get a handle on your IT nightmares!
Happy Halloween!
