Following on from my previous post about surviving implementation, here’s some advise on running your Change Management process aka The Day Job.
In Change Management your key areas to focus on are:
- Recording and processing the Change
- Change assessment
- Change Advisory Board (CAB)
- Build and Test
Raising the Change
So first up; recording the Change. Ensure your Change Management policy covers who can request a Change i.e. can anyone raise a Change? Just IT? What about the business? Each organisation will be different but one thing to keep in mind is making sure your policy gives clear guidelines on the difference between a Change and a Service Request i.e. with Request Fulfilment ends and where Change Management begins.
Create a Change form so that Changes can be raised in a standard way. It’s really important to have consistent information in your Change requests so when reviewing and approving Changes, you have all the facts needed to make the right decision for your customers. If you have a sparkly, all powerful toolset to do it for you then happy days. If not, and there are lots of organisations just getting started with Change Management, then it’s time to get creative. In the past, I’ve set up Change request forms using Word or Excel (tweet me if you would like to see some examples).
Things to consider for your form:
- Service affected
- Impacted Cis
- Will the CMDB (if you have one) need to be updated afterwards
- Implementation windows
- Implementation teams
- Pre implementation testing
- Implementation plan
- Post implementation verification
- Back out plan
- Impact to other environments
- Will the change be replicated to your DR environment?
You need to have a lifecycle approach for raising Changes for example; the process for a standard Change will be very different to an emergency, sorting something out in the middle of the night type Change. Ensure this ties back in to your policy with criteria and examples so there’s no confusion.
Look at how Changes are classified and prioritised. If every Change is urgent, high priority, which one do you implement first? Classification is also really important – make your Change owners accountable for risk and impact assessments. If your company or Change Management tool has a Risk calculator use it as it enables Change requestors to assign a tangible risk to the Change (it removes the “if in doubt click medium” behaviour type) if not, I have some templates that I can share.
Next up is our old friend, the Change Assessment stage. This is the initial check that the Change is reasonable and makes sense, a sanity check if you will. Sounds obvious but unfortunately you can’t teach common sense. On one memorable occasion, I was reviewing a list of Changes on my pre-CAB FSC and one jumped out at me. The Change was to move a business critical server from a secure Data Centre environment to the Server technician’s desk. The Change justification? “My little legs are getting tired from walking to the server room all the time”. Needless to say words were had and the Change was removed from the schedule (I just wish I’d taken a screen shot).
Things to bear in mind when assessing a Change are benefits, both technical and to the business. There’s no point in having the newest most gadgetastic server is the world if your end users don’t see any benefit from it. Really think about the risk involved with the proposed activity for example:
- Number of people affected
- Loss of productivity
- Seasonal considerations
Make sure you have clear assessment criteria for managing Changes. Some examples could include:
- Pre implementation testing – how do we know this Change will go as planned?
- Deployment plan – does it make sense, if other teams are involved are they aware and do we have contact details for them? Are there any dodgy areas where we might need check point calls or additional support to mitigate risk?
- Post implementation verification – ok we’ve done the Change, how do we make sure everything is as it should be?
- Back out plan – hope for the best but prepare for the worst. What happens if something goes wrong on the night? Do we fix on fail or roll back? Are the Change implementers empowered to make a decision or is escalation needed?
- Impact to other environments – “who cares about other environments?” I hear you ask. Let Aunty Vawns tell you why it matters from personal experience. Once upon a time in a galaxy far, far away I worked for a large investment bank in the city. A code Change to one of the most business critical systems (the market data feed to our trade floors) took longer than expected so instead of updating both the production and DR environments, only the production environment was updated. The implementation team planned on updating the DR environment but got distracted with other operational priorities. Fast forward to 6 weeks later, a crisis hits the trading floor, the call is made to invoke DR but we couldn’t because our market data services were out of sync. Cue a hugely stressful 2 hours where the whole IT organisation and its mum desperately scrambled to find a fix and an estimated cost to the business of over $8 million. Lesson more definitely learned that day.
So we’ve raised our Change and sanity checked it to make sure what we’re planning on implementing is sensible and won’t, you know, set anything on fire. The next step is the Change Advisory Board or CAB. When setting up your CAB, make sure you have a clear Terms of Reference statement which will give attendees a steer on how to prepare, good meeting behaviours and how to represent Changes effectively.
Not every Change has to go to CAB. In fact, I’d say you should use CAB for your big, complicated Changes that would have a major impact on the business. Monthly, BAU server patching? That’s a candidate for a standard Change. Keep CAB simple and uncluttered with an agenda that deals with Changes that are high risk, major impacting or have lots of complicated detail. Make sure the right people turn up and that all areas are represented (don’t forget Security if you have any ISO 27001 or NGN regulatory requirements).
An effective CAB agenda could look something like this:
- Review of implemented Changes
- Incidents (or if time is short Major Incidents) caused by Change
- Lessons Learned
- Forward schedule of Change
- Candidates for templates / standard Changes
- Improvements / CSI
- Good news stories
Remember, CABs don’t have to be a 2 hour meeting where everyone is locked in a conference room. Look at Agile or Lean on how you can make efficiency savings, consider virtual Cabs where you can approve most things via your toolset and make full use of technology such as conference calls, WebEx’s or Skype.
The next stage of the process is Build & Test. The first thing to look at is developing standard build methods. Use automation where appropriate, it saves duplication work, reduces the probability of human area and the economies of scale can be huge. If automation is too expensive, ensure build methods are well documented and templated where possible.
Look at your testing environments and ensure they are fit for purpose. Every organisation will have different needs but when looking at this – do you have enough to cover your operational work eg a live environment for production work, a DR environment in case the worst happens and an environment dedicated to testing and training? If you are limited do you have a booking system in place so that testing / training / dev time can be allocated fairly?
What about Changes to non production environments? Are they covered by your Change process? Is there a monthly environment refresh just in case?
How do you test Changes before go live. Is it Bob from the Server team saying – “this’ll do” or so you have something more formal in place. Are the business willing to help support with Change testing? Getting the business to support testing can have huge benefits. One client I worked with has huge challenges with keeping the Marketing department happy with Changes that they had requested to the company website. The Marketing team didn’t provide any post Change validation and the amount of backed out Changes to the website or emergency Changes to fix errors such as typos were common place. By engaging with the Marketing team and asking for someone to be available for a set 20 minute period of the Change window (we called it the smoke testing phase) our effectiveness increased and so did our customer engagement.
When implementing Changes it’s not just a case of hitting a bit red button and shouting “fly my pretties, fly” to an imaginary army of flying monkeys (although that would be so cool). A professional approach and great communication are key to making this stage a success.
First up, your Change Schedule (aka your Forward Schedule of Change or FSC). Make sure it’s easily accessible to your stakeholders so stored somewhere central that’s easy to navigate. Make sure it’s dynamic and relevant – otherwise you risk your data being obsolete within minutes of hitting the print button. Make sure it’s fit for purpose, states what Changes are being carried out, when and by which Service. You don’t need expensive toolsets for this; you can do it in Excel.
Something that goes hand in hand with your FSC is your Projected Service Availability or PSA. Yes, I know ITIL 3 insists on calling it a PSO (Projected Service Outage) but lets not frighten the horses. Planned Service Availability implies we’ve got this. Everything is planned, tested and safe; you have nothing to worry about. In contrast PSO just says downtime. What do our customers not like? Downtime – no matter how well planned out. Again, Excel can be your friend here, keep it simple, a list of your Services – green for those that are up and running, amber for those due some maintenance time.
When looking at Change windows ensure they have been agreed in advance with the business and that they are codified as maintenance windows on any SLAs. Try and negotiate pre-approved Change slots where possible eg the third Thursday of every month between 22:00 and 01:00 we will be doing security patching. If you know in advance that a Change needs to take place outside the usual Change window – ask nicely! If you’re really good you might be able to get some SLAs relaxed so that as an organisation, you’re not penalised for carrying out break / fix work.
Finally, we have the Review stage. We’ve done our Change and everything has (hopefully) gone to plan. So let’s carry out a review of our implemented Changes. When things go well, brilliant! You have new candidates for standard Changes, work instructions or templates. If things go badly or you really do end up setting something on fire then let’s look at how we can do better next time. Carry out a Change review to look at what happened, what went wrong, what was the root cause, how did we fix it and how do we stop it from happening again? Involve Incident & Problem Management here as they have super powers in these areas. Above all you want your lessons learned to be captured, discussed and acted on. This could be a regular agenda at CAB meetings or form part of a Service Improvement Plan or SIP.
When reviewing Changes look at the benefits. In terms of business benefits, did we achieve the expected results? Have we got any customer feedback? We also need to look at the technical benefits i.e. have we increased stability, added resilience or fixed any Known Errors. If we have – brilliant – tell the Service desk so they can let our customers know.
I’d like to finish by saying that Change Management really is a critical process for managing controlling and protecting your live environment. As Change Managers we get to ask all the hard or awkward questions but on the bright side, we’re usually the first to know about cool new projects or sparkly new gadgets. Keep smiling, it will be fine. And if it isn’t, tweet me, I’m always happy to help!