If you missed part 1 of this two part series on Maximizing Your Investment: Finding ROI Through a Configuration Management Database (CMDB) click here. In the second part of this series we will explore the CMDB and problem management, change management, and how to properly implement your CMDB.
THE CMDB AND PROBLEM MANAGEMENT
The process we all love to hate. We all want to do it, but we are all so buried in firefighting and break-fix that, well, who has the time? Some shops, and I include some of my own previous employers among them, tend to treat problem management somewhere between major incident management and finger-pointing 101. I cannot really explain why this is, but it definitely is not what is intended by ITIL best practice. Problem Management is meant to be a very proactive, enhancing, and ultimately cost-saving process that should be ongoing for the entire environment. Unfortunately, for reasons stated, it gets relegated to the dustbin of, “Well, we don’t know why it broke, but someone is to blame for it!”
So, how do we reverse this trend, and why can a CMDB help? Well, the answer to the first question is very dependent on an organization’s particular situation, but I can say that if you start approaching Problems more as research exercises, rather than trying to figure out who blew it on that last deployment, you may find yourself fixing things before there are incidents—and that is how it should work.
As for your CMDB and Problem Management, think of it this way: when a service goes down, especially a tier 1 or tier 2 type service (say, email, just to keep my examples consistent), there is a flurry of activity. Incident Management swings into action, getting things back up with duct tape and bubble gum if necessary. Then, when everything quiets down, management wants some sort of After-Action report, perhaps a meeting on what happened, why it all went plaid. So, we call up our Problem Manager (probably the only member of the Problem Management team) and make him or her write up metrics and monitoring reports, get it together so we can sit down 14 hours after the outage to discuss what went wrong. And maybe we get it right. Maybe we can say, with great certainty, why the service failed. But often, this is just an educated guess, based on previous outages, and related changes, and a healthy dose of wordsmithing.
Now, enter the CMDB, with its ridiculous amount of data at our fingertips. If we are keeping it up to date, maintaining system baselines, updating CIs after changes, etc., we suddenly have, in a completely relational and meaningful package, all of the components that make up the service (hmm…traceability, anyone?). Now your overworked Problem Manager can see, quite easily, where things went poorly. And, not only that, we can use this as a predictor of future performance. So, not only can the Problem Manager appease management with cold, hard facts about the systems involved, he or she can suggest a real solution based on the available data. And all of this because we maintained our CMDB properly. Instead of promising the Marketing VP that it will never happen again in the middle of one of her amazing demos, we can say, with great certainty, that we know exactly where things went badly, and precisely what will ensure it doesn’t go that way again.
Now is the CMDB some sort of wizard that can conjure solutions for you? Of course not, but having well-documented, highly integrated data at your disposal can go a long way to creating a positive approach to root cause analysis.
Additionally, with the proper tools in place, we can even create visual references of how CIs interact with one another, including changes and requests associated with each component. A good visualization tool can even show components that are currently down, and how that is impacting components and services around them.
So, really, building and maintaining a great CMDB not only tells you what you have, but how to better manage it to avoid incidents down the road.
THE CMDB AND CHANGE MANAGEMENT
The final process we want to discuss is Change Management. This is, without a doubt, one of the most important processes we perform in IT. Additionally, it is the most responsible for maintaining a quality Configuration Management System.
It is widely held that nearly 80% of all IT outages can be traced to human error. A Forrester study a few years back supported this theory. And, those human errors exist largely in one very visible, very critical process within the ITIL lifecycle. We fear change. Systems that don’t change don’t break.
As a system admin, when I got a frantic call from a development team demanding we make an emergency change because they had promised the business some feature by the end of the week (and the call usually came in on Friday afternoon), my stomach twisted into a little ball. My head began pounding. These are not exaggerations. I went through so many ‘emergencies’ that were manufactured by over-promising, that I began to manifest physical symptoms for a psychological malady. I hated making changes. It meant I was going to get exactly no sleep for the next 3 days while we rolled back, rolled forward, rolled sideways, trying to get this cool new feature to not bring down an entire service.
Interestingly, a lot of the changes that end up causing outages do not manifest as failures. The new feature, or changed configuration works exactly as it was intended. But it broke something else. Something we had missed. Some system that was dependent on the old configuration to run properly. And those sorts of things don’t often get caught in regression testing and QA. Use cases for the changed service are tested thoroughly. We prove out every possible scenario we can think of in the development, test and staging environments (if we are lucky enough to have all 3), and the awesome stamp comes down, and we push the change to production. The new cornflower blue bar chart showing how amazingly awesome our sales team is shows up right where it is supposed to; it is clickable; you can drill down into infinite levels of detail. Everyone cheers. And then network printing stops working in the Seattle Office. How is that possible?
Well, it turns out that the database server being used for the new feature also housed the print server that everyone in Seattle uses to create daily reports, some of which are hundreds of pages in length (I will not go into how environmentally unfriendly that is, but—well, you know). So, when 50 marketing folks started clicking all over this neat new feature, they bogged down the network connection on that database server, and it caused the print server to crash. Incident Management logged into the DB server and restarted the print server, and five minutes later, it happens again. If only we had known before the change that this server was multi-purposed. How, oh how, could we ever know such esoteric information?
TA-DA! Configuration Management Database. It is perhaps an extreme example of how a seemingly unrelated service could be affected by a production change so thoroughly tested, but it is not far-fetched, especially in smaller to mid-sized firms. We don’t have budget to have a separate database server and print server. And that system is not doing a whole lot—most of the time. CPU usage is low, memory utilization is near nothing. Network activity is pretty flat (except for those 2 hours every afternoon when Seattle prints the New York City phone directory). This seems like the perfect place to drop a database. And it is free. We do not have to buy infrastructure. This is a super idea. And that is not sarcasm. We want to utilize our hardware, especially if it is just sitting idle most of the time, wasting money. If your CMDB is well-maintained, you will know, well in advance of any change to that server, that there are two very disparate services being provided. And you can predict based on your capacity plan—you do have a capacity plan, right? (Again, a discussion for another day)—how much more utilization you can expect when that new feature gets added. And if you can say that the server cannot handle it with a high level of confidence, maybe Marketing will be willing to shell out a couple thousand bucks to buy their own database server.
Figure 1: ServiceNow Change Management Dashboard
Now, here’s where we find ourselves with a cart and a horse, and very little direction as to which goes in front. The fact is, Change Management must be pretty mature within the organization (some say CMMI Level 3 minimum), and well-controlled. No ad hoc “executive” exceptions, a good Change Manager, all the trimmings, before your CMDB can be this successful. Why? Well, if you do not have awesome change control, you are not going to keep your CMDB up-to-date with any real accuracy. And an inaccurate CMDB is probably less effective than no CMDB at all.
Configuration Management is tedious. It is hard. It takes a small army of people to make sure it works within the organization. But with high quality change control, it can be achieved. And it CAN bring great value to the organization.
COMMON CMDB CHALLENGES
This segues quite nicely into the challenges you face as you begin your journey toward good configuration management. As I mentioned before, immature Change Management is a sure-fire path to mediocre configuration management. The two are inextricably linked. To finish my analogy…Change Management is the horse.
The second big obstacle you will face, once you make the decision to dive into building a quality CMDB is the dreaded rogue database. Gee, what’s a rogue database, you ask? Well, if you have ever worked in a small IT shop, especially one in a rapid-growth organization, chances are you had a spreadsheet of all of the systems you were responsible for. You probably didn’t share it with anyone, just updated it when things changed, so you could quickly reference it when something went wrong, or someone needed some data. You basically ran your own CMS in Excel. And, that was probably sufficient to avoid the pitfalls of no configuration management. But then the company grew. Assets increased five, 10, maybe 100-fold. And there were a lot of other people keeping similar spreadsheets to themselves regarding their systems. When you start, at an organizational level to begin true configuration management, you have to snuff out these one-off solutions. Not in a bad way. They can go a long way to federating your data, to be perfectly honest. And, if you approach the task in a positive way, “Hey, I really like what you
have there; that data is going to be really useful to us as we build our central configuration management database,” it will help with buy-in from those admins and engineers who will complain about updating two systems. HINT: They only need to update yours, once the data is gathered.
It can be a tough road getting everyone to jump on the CMDB bandwagon, and as with many initiatives, inclusion of the stakeholders, especially those that will be responsible for maintaining the CMDB, is key to winning them over. Give them that one feature that they just cannot live without. Turn your biggest critics into your biggest allies. As with so many things, converts are the best evangelists.
AUTOMATION TOOLS FOR CMDB
There is a—truism—in ITIL. Something our trainers stress in every Foundations class. While ITIL, itself, is not prescriptive when it comes to technology and tools, it does tell us something very useful, and we spell it out as explicitly as possible. Tools help with everything. Tools are useless without process, to be sure. If you just go out and install the first slick-looking ITSM tool you find on the market, but you have no processes in place to support it, you will be much poorer, with the exact same problems that led you to buy the tool in the first place. And everyone will hate the tool. So, process 1st, tool 12th. (The intervening 10 steps are also process, just to be clear).
But, now you have some great processes in place. They are working well, but boy is it tedious. It takes too many people too much time to do what should be very little work. Effort levels through the roof, productivity seemingly at a standstill. Now you have the perfect opportunity to look at automating some stuff. And that is where…Tools help with everything.
Something a lot of you are probably familiar with on at least some level is the idea of automated discovery. Altiris, LANDesk, and myriad other tools on the market all claim to find everything on the wire, and store, classify, and polish it all to a high gloss sheen in a super-easy-to-use database that will integrate with everything including the toaster in your break room (which was probably manufactured in 1964, and still had bread crumbs from that year stuck in the bottom). A lot of these tools are insanely cool. I’ve worked with several of them, and it is kind of astonishing how effective they have become at gathering unsettlingly accurate and thorough information regarding every system you have attached to your network. Which tool should you use for automated discovery? That is a question for you and your organization to answer. Most have great strengths. They all have some weaknesses. It all depends on your budget and threshold for implementation effort.
So, how does an automated discovery tool really work? Well, mostly SNMP, to be honest. Some require a special agent be installed on the target systems, and that can be a bit clunky. But most work within a pretty standard set of parameters and give you a really good peek inside your infrastructure. And, once you have a real baseline of your environment, you can use that as a touch point down the road. You can see baseline changes to existing components, and, perhaps more importantly, you can audit your environment with a high degree of accuracy. This
helps weed out changes that maybe didn’t quite follow the processes you painstakingly built to help you maintain your pristine CMDB.
The next step is finding an ITSM tool that can help you with a lot of linking of components within the CMS. A good tool will be able to show you that when you are going to change that print server and add a high traffic database, what services are going to be affected. It can also help with maintaining license information. And, if you have ever been the target of a software license audit, you are keenly aware of just how unpleasant vendors can be when they find you’ve over-allocated your seat licenses by 28%.
Additionally, a good ITSM tool can help you to visually understand the interactions between components. I touched briefly on this while discussing Problem Management, but it is a key feature that a quality ITSM tool will provide.
It can be as simple or as complex as you require for your environment. And there are hierarchical ways to establish these links. Each CI can be linked further up or down the chain until you have a complete picture of how your environment interacts with itself. So, when you are about to run Change 10181, you know that it could impact CI ATX9023, even though that is not the component that is being acted upon in the change. This can go a long way toward planning, justification, timing, and budgeting for your environment. And, since the majority of us are, in fact, visual learners, pretty pictures can be worth even more when you head to your daily CAB meeting.
FINAL THOUGHTS
So, there is a lot of value that you can glean from a quality, well-maintained CMDB. It is by no means the fix-all for what might ail you, but if you are heading down the path of best practice, a good configuration management system, anchored by a great CMDB, will really make all of your process improvement efforts more effective.