Does Your Service Desk Know What Changed Last Night

Written by Mark Hillyard

In my career I have held almost every position in IT Service Operations one might imagine.  I have been a 1st level tech, operations manager, system engineer, even acted as a data center operations manager.  I have also seen the consequences of poor communication within IT Operations.  Service Transition without the hand-off is a major contributor to unplanned outages and confusion, which leads both IT and Business Management to lose faith in the efforts put forth by their technical staff.

It is likely that a good IT Organization holds weekly, if not daily, CAB meetings to discuss upcoming changes, and this is a very good thing.  The advent of the Change Advisory Board has been extremely beneficial for IT Service Management as a whole. However, it is also likely that these same organizations are not effectively communicating changes that occurred in the recent past.  That is to say, if an approved change is rolled back, what is the communication plan?  If the Service Desk is only aware that a Change was scheduled, but not informed of its outcome, what process do they follow?

Chances are, if you have spent any time in IT Service Management, this has been a reality at some point.  I recall an all-hands meeting called by our COO to complain to the employees about failed changes.  The most interesting thing about the meeting was that I think a lot of people were shocked by the numbers of failed changes and incidents caused by changes.  In retrospect, I think the executive was missing a key point as he railed against both Operations and Development for 20 minutes or so in an open forum in plain view of other support staff.  What we all took away from this tirade was that we were doing a terrible job of executing changes; but the reality was that we were being failed by the absence of a good communication plan.

The organization held daily CAB meetings by phone.  This gave upper management great visibility into anticipated changes; however, instead of providing quality hand-offs from development to production operations, changes were being implemented and rolled back multiple times in a single window, with no explanation as to what was functionally changing.  All management could see was failed changes and customer impacting outages as a result of poorly executed changes, and they made it very clear that everyone was on the hook going forward.  But they provided absolutely no guidance as to how we could mitigate these issues.  They, for all their bluster and good intentions, spent no time seeking a root cause for the rapid increase in failures.  It was all about the numbers.

Middle management clamped down hard, and we spent several months making as few changes as possible to keep the lights on.  This was not the best solution.  Change is a big part of Service Management, and if no one is confident in allowing changes to take place, the Service Provider is going to suffer, as will the wider business.  It happened to us in this case as well.  We began to get incidents unrelated to change but that could have been avoided completely if an appropriate changes had been implemented.  Performance issues began to crop up as systems were unable to scale due to change freezes and the overall timidity when it came to requesting a change at all.

Our organization was large enough, and successful enough, that we managed to weather this unfortunate setback for the Service Desk.  But it could have been, and should have been avoided by a clear plan by management to improve communication during the Service Transition phase, especially as Services were handed back to operations after a Change.

For those of us that have held positions of authority within the IT Service Management sphere, we would do well to consider this when Change Management appears to be faltering, or when an increase in Change related incidents creeps into our weekly status reports.  We must remind ourselves that the first place to look for solutions is not always with heavy-handed punitive action.  Often, simply improving communication–especially through a concerted, well-managed communication plan–can have a dramatic effect on our Service Desk and its effectiveness.  Operations staff need the tools to do their jobs well, and if we don’t provide a conduit from development to production for this kind of communication, we can rest assured that our success rate will dwindle as change frequency and overall numbers increase.

Now that you’ve got Change Management on the brain…
Enjoy this quick overview of how to run a successful CAB meeting.

Originally published August 08 2014, updated December 12 2019