Ever since I got into working in the IT field, I have always been part of a team. I like working in a team environment and have turned down positions where I would be the sole IT person. When I started my current job, the staff was small and fairly inexperienced. They were shocked to find someone with my resume available in deep east Texas. Since I started, we have grown by one, plus had a retirement that resulted in another very experienced hire for a replacement. So now four of the six non-management people are very experienced, and the other two are very experienced at the specialized systems they used in other departments before transitioning to IT. All in all, I am very comfortable with where we are.
Then I have weekends like this one. I have the late shift this week, so I am coming in when everyone else is leaving. Everything starts good, progress on several projects is made without disturbing the business. I'm happy. Then I get back to my desk and my director calls. My director who is out on leave due to a death in the family. Alerts on the VMWare environment are hitting her inbox. While we are going over the notes, half the system goes down. Crap.
Now, I have experience with virtual machines, but not with our software. There are similarities, but this is not only different, its also been five years since I touched it. So, call to the vendor, then two escalations later I am on the phone with a Level 3 tech who was just about to clock out. We spend two hours fixing things and doing detective work to find out what caused the server to crash. Turns out that a network switch that was put online for testing a month ago has been arguing with the VM server over the IP. After a month, the server lost the argument. What made it worse was that due to failover partially working, the business was not down, but fixing it would cause an outage. That part sucked. Despite warnings to management, the phone still rang off the hook.
Once the network admin showed up, traced the MAC to the switch and fixed the issue on his end, we got everything back up and running and made some other changes to better support the load. Two and a half hours of operating in areas of the environment that I was intimidated by. Stress that no amount of coffee can fix.
The real disappointing part was finding out that all of the documentation that I am working on daily to fix/organize/update was no use because those who set up the virtual environment before I arrived had not documented this part. So, the new network administrator relied on the documentation that I was assured was correct and caused me to have to stretch my IT skills. It was uncomfortable, but made me realize how much a do recall and where I need to improve. It also put two other projects well behind and made today a very busy shift. Still, I got it done and I feel good about that.
No comments:
Post a Comment