Ivanti Insiders Reveal Their Scariest Moments on the Job
When you think of scary moments on the job, you might picture physical job hazards such as those associated with police officers, construction workers, firefighters, or electricians. But job hazards come in all shapes and sizes.
Just because Ryan, the IT guy, works in a comfortable office with a nearby vending machine does not mean his job is stress-free.
Imagine it's summertime and you're an IT professional at a hospital. Suddenly the AC goes out in the server room. Not your problem, right? Actually, it is your problem, because the servers can't run in a 98-degree environment.
One by one, servers start shutting down. More than 40 servers are now offline, including the phone system. It also happens to be 4 a.m. on a Sunday, and even though you feel like going home and crawling back into bed, you're the person everyone is relying on to get the computer systems back up and running as soon as possible.
The pressure to "fix it" is compounded by the knowledge that if doctors and nurses can't access patient records, lives could be at risk.
In the spirit of Halloween, we asked our Ivanti Insiders what their scariest, most adrenaline-charged moments have been on the job. The above scenario is a true story from one of the responses, and below are a few more.
We've also included what each Insider learned from their bone-chilling experience.
Issue: "UPS (uninterruptible power supply) caught fire in the network closet."
Fix: "Fire was contained to just that stack, but all switches, routers, and other peripherals had to be replaced."
Takeaway: "Automated management and status for all UPS going forward."
— Joshua, Systems Management Tech
Issue: "We had built new Citrix PVS servers, and it looked liked everything was configured as should be, so we started to do a failover from the old to the new servers. But all the servers went down because of an old IP address in the configuration."
Fix: "With nerves of steel!"
Takeaway: "Always double check the configuration."
— System Engineer
Issue: "One of our user's accounts had been compromised and a hacker was sending malicious emails through the web mail portal."
Fix: "User's password was changed and all connections on the web mail portal were bounced in order to force all users currently logged in to have to log in again, cutting off the hacker's access."
Takeaway: "How to kill all the active sessions in the web mail portal by bouncing it."
— Josh, Cyber Security Administrator
Issue: "Office 365 email going down."
Fix: "Called Microsoft and requested an escalation on the issue."
Takeaway: "External support contracts can be great, but they have to have the same urgency as you would if you were prioritizing the issue. Also, everyone loves to complain about receiving emails, but take it away and you cripple all of them."
— Kelly, Technical Support Specialist
Issue: "Cryptovirus breach on network share."
Fix: "Disconnect from network and revert to older version."
Takeaway: "How quickly it can take effect."
— Ross, System Engineer
Issue: "Mayor's account constantly locking him out."
Fix: "Wiping the Apple stored passwords on all six of his Apple devices and disabling the Apple account sync."
Takeaway: "Apples suck."
— Desktop Support Analyst
Issue: "Launched a command that started removing the users' local accounts from their computers."
Fix: "We caught it before a large number of machines executed the task, restored/recovered data for the rest. Fortunately, coworkers pitched in to help."
Takeaway: "Measure twice, cut once for everything. Rushing through things leads to mistakes."
— System Engineer
The issue: "Explosion/fire in data center."
The fix: "Sprinkler systems put out the fire, but caused further damage and longer outages."
Takeaway: "Find a real service provider with better facilities and implemented redundancy and DR strategies."
— IT Analyst
The issue: "Deleted information by mistake that affected several thousand devices."
The fix: "With patience!"
Takeaway: "Measure twice, cut once."
— Avalanche Admin
And a couple frightmares we found particularly amusing...
The issue: "Touching elementary school keyboards... those things are mad scary!"
The fix: "Gloves and lots of Purell."
Takeaway: "Keyboards in public schools contain the plague."
— Team Leader
The issue: "People."
The fix: "Avoiding people."
Takeaway: "Avoiding people is not a problem."
— Ivanti Insider
We hope all of the above examples give you the opportunity to learn from the mistakes of others and keep your frightmares to a minimum.
Happy Halloween from all of us at Ivanti!