The Evil That Men Do

7;55 AM. I have been at work 25 minutes. I have sent out my morning reports and checked the ticket queues, responding as needed.

"You have mail"

El Gee:

Severity 1 in Mexico regarding WAN. Customer wants us to take the lead and coordinate an escalation, arrange meetings, etc. I want you to take this because I am busy. Contact NOC (Network Operations Center) for more info. Customer expects meetings every two hours.

Hugs and Kisses,

Dotted Line Mgr in Europe

OK, so I took some creative liberties with the "hugs and kisses" part, but the rest of the e-mail is what he wrote. Now when I first get this, I think it is a joke. This is the same man who yesterday told me that we should never work like this and today, he does it to me. Rather than call our NOC (which is 5000 miles away), I call the account mgr for Mexico, because according to procedure, he should have all the info, right? WRONG. "Joe" knows what is going on but since it is not an issue that we work on, HE IGNORES IT. I have to politely tell him that it is our contractual obligation to handle all customer IT escalations no matter who is working on them and I should be getting all my info from the account manager (him), not some bloody NOC 5000 miles away. He senses my disdain at his ambivalence (I wonder how?) and offers to call the IT manager at the site. While I am talking to him, I e-mail the NOC to get an update. I feel foolish because I do not have a ticket number, but I pray for the best.

Just as I get off the phone with the account manager, our customer's escalation mgr calls me. She is not happy that I have not called a meeting yet regarding this issue. After she vents, I politely tell her that I was just handed this less than 30 minutes ago and I have not had time to research it yet. To her credit, she was mature enough to stop complaining and actually listen to me while I told her what I know and what I am going to do. I told her who I had talked to and who I had not gotten in touch with yet, and when the first meeting was being conducted. She then said that she did not have to be called back unless I needed her for some help from her side. I thanked her and hung up.

First meeting: I tend not to invite too many people because I am a firm believer in "too many cooks spoil the soup". The people in attendance are the right people, but I needed buy in from at least one person from the WAN team (who just happens to also be the customer). Since they are not available, I gather info from the customer, the account mgr, and the NOC. I thank the people for their time and agree to meet again in two hours, per customer request.

90 minutes later: Our customer's escalation manager calls me back...she is rather concerned that the issue has not been resolved. I inform her that it has been narrowed down to a local telco issue and a tech in in route to troubleshoot equipment. The manager begins to press me to open a formal escalation (so far it has been informal but documented) but I reason with her to give me 2 more hours before we call in upper mgmt. She agrees.

Noon: I hold another meeting. Attendants are late to arrive, worrying me to some degree. Mexico has been without WAN connectivity for 18 hours...way to long. This time one of the key people is able to attend and real discussions open up. After scrutinizing the path the "escalation" took, we deem that the NOC and the local tech had a break in communication as well as a tool problem at the NOC (the circuit in question was not listed on the telco's trouble reporting page). Then the shift manager at the NOC says, "The circuit is back up!" My heart returns to its regularly scheduled blood pumping.

I issue action items and type up my notes. The notes are sent out to all those who need to know and then I make the call to the customer escalation manager. She is at dinner (but she has not received her order yet, so she agrees to talk) and I tell her the good news. She is very happy.

I explain the situation and the processes to her, telling her that an escalation manager really did not need to be involved since she never requested a formal escalation. She asks me how this could have been handled better and I told her about the breakdown in communication at the NOC and she seemed to understand.

She offers a very warm and sincere "thank you" to me for being willing to assist coordinate this. I tell her that I did not mind helping, but if she really wanted make things go smoothly she could have her site managers call the ADM if they were having problems, BEFORE they are escalated. She tells me that she is very happy that I am working on the $customer account. That, and 99 cents will get me a cup of coffee at Dunkin' Donuts.

After all that I finally ate lunch and began working on this little gem before I forgot the details. Always a dull moment.