A practical look at why some metrics programs fail while others are successful, along with some tips you can use to kick your metrics up a notch.
I was math-challenged as a child and hatred of anything having to do with numbers followed me into adulthood. This hatred remained with me until I became a manager and needed to begin proving the work my team was doing or understanding where we were failing. Actually, the turning point may have been the now-overused adage “you can’t move what you don’t measure,” a powerful concept that has a lot to do with the metrics programs I’ve created over the years. I’ve worked hard at this, mainly because of my math aversion. While Excel certainly helps, it’s all still “funny math” and through practice I’ve learned how to justify any story I want to tell using the numbers available from the IT Management tools my organizations have used.
Ultimately, if you can a story with metrics, how do you decide what is the right story? That’s the focus of this blog: determining the story to tell and to whom.
Building a Business Oriented Metrics Framework
Ultimately, if you turn to ITIL for help with metrics, you can be led astray pretty easily unless you read all of the books (or at least Service Strategy (SS) and Continual Service Improvement (CSI)). This is because at the end of each process described, there is a list of Critical Success Factors (CSF’s) and Key Performance Indicators (KPI’s). These are great sample metrics for the process you might be implementing and are critical for measuring that process’ success, however providing them to business partners will have you producing the same type of metrics IT’s been producing for years, the type that are not of real interest to the business. You’ll also be led into a false set of security because they came from ITIL, didn’t they? YES!
While these process metrics are one of the three types of metrics ITIL recommends you produce (Process, Service and Technology) and while they are important metrics to produce, they’re of little or no interest to business partners outside of IT because they don’t tell you how well IT is doing at delivering on the key strategic initiatives of the business.
To craft a metrics program that is of interest to the business, you need to start with the business. To help you get started, you can use the informal framework for building business-based dashboards and scorecards presented here (If it seems familiar, it is. It’s based on ITIL’s Continual Service Improvement approach):
This framework is very simple:
- Know the vision of the organization or line of business
- Document the goals that support this vision
- Discover those Critical Success Factors (CSF’s) the organization feels are needed to be successful
- Create Key Performance Indicators (KPI’s) or measurable indicators of the Critical Success Factors. Include target levels for these, so success is clearly shown.
- Organize them into dashboard views for each audience that may be viewed live (on-line).
- Develop scorecards that may be used for trending, historical reporting.
Three Steps to Using this Framework
This framework can be delivered using five basic sets of activities or steps, which are described below.
In addition to these steps however, some of this can only be demonstrating using examples. For these, let’s use a sample organization that is expanding into web-based sales to demonstrate the concepts. In this organization, the new Web Sales department and the Audit/Control group are tasked with delivering on three goals that support the organization’s vision of “providing the best shopping experience on the web.”
These goals include:
- Providing Customers with an Excellent Web Shopping Experience
- Giving Customers the ability to do shop any time of day (or night)
- Guaranteeing credit card security
With this in mind, let’s look at the five steps:
Step 1: Create a Focus Group
To ensure alignment, create a focus group consisting of key stakeholders from several lines of business and a few IT Managers. For the organization in the example, this would include managers from Web Sales, Audit/Control and the IT teams tasked to develop and deliver the website.
Step 2: Understand the vision, goals of the organization
With the focus group, take a look at the organization’s strategic plan. Typically the strategic plan includes a set of initiatives designed to support the organization’s vision, similar to the web sales initiative. These are often stated as goals so review the business goals associated with the initiatives and define the ways in which IT supports these goals. Think of the goals as the pillars that support the organization. This will ensure your program aligns with these goals and the strategic initiatives.
To move to the next step you will need the vision and goals, similar to the ones provided for the sample organization.
Step 3: Identify your audiences and their contribution
Next, working with the focus group, create a matrix to document the goals and critical success factors for each of the organizations to which you’ll be reporting. This matrix will be used to plan the dashboards and scorecard measures you need. Using the sample organization, the matrix would look like the one that follows.
|Web Sales department||(1) Excellent Web Experience(2) Ability to do shop anytime|
|Audit/Control||(1) Confidence when using credit cards|
|IT||(1) Service Operations Excellence(2) “Fort Knox” security|
Step 4: Make the goals measurable
To quantify the goals, you’ll need to work with your focus group to determine the Critical Success Factors that will demonstrate the fulfillment of their goals. The best Critical Success Factors (CSF’s) will be: “SMART”: Specific, Measurable, Attainable, Realistic and Timely.
Once your and the focus group have agreed on the CSF’s, you’ll be able to develop Key Performance Indicators, or measures that support the CSF. It’s extremely beneficial to develop KPI’s along with targets, so you and your business partners are clear on whether you’re successful in delivering on each of the goals. The best part about this approach is that when IT and the business agree on measures and targets, it’s easy to tell when IT has delivered or when IT is not meeting the needs identified by the business.
The ITIL books demonstrate this process clearly at the end of each process documented. The last section of the process description includes a list of Critical Success Factors for the process and Key Performance Indicators that support them.
For example, the Incident Management process (ITIL Service Operation 2011, p. 109) has a Critical Success Factor to “minimize the impact to the business of incidents that cannot be prevented.”
This is not measurable by itself, but four Key Performance Indicators follow it:
- The number of known errors added to the Known Error Data Base (KEDB)
- The percentage of accuracy of the KEDB
- Percentage of incidents closed by the service desk without reference to other levels of support and
- Average incident resolution time for those incidents linked to problem records
At the end of this stage, your matrix will be complete, similar to the one which follows for the sample organization:
|Web Sales department||(1) Excellent Web Experience(2) Ability to do shop anytime||(1) Customers are satisfied with the website design and functionality(2) Web site is available 24×7||(1) 85% of customers give the site a 5-star rating on exit(2) Web site is 100% available|
|Audit/Control||(1) Confidence when using credit cards||(1) Web site is PCI compliant(2) Security patches are up to date||(1) 100% PCI Audit pass rate(2) 90% of patches applied within 24 hours|
|IT||(3) Service Operations Excellence(4) “Fort Knox” security||(1) Web site is available 24×7(2) Web site is PCI compliant(3) Security patches are up to date||(1) 100% site availability SLA(2) 99% performance SLA(3) 100% PCI Audit SLA(4) No Security Breach SLA(5) 90% on-time patch SLA|
You might notice several things when reading this list:
- A qualitative measure (5-star rating by customers) is used to determine the customer’s view of the website. This is a critical measure as the CSF points to the customer’s experience.
- The quantitative measures that sound like IT performance measures are translated to SLA’s for reporting purposes under the IT list of KPI’s. When creating the dashboards and scorecards in the IT Service Management tool, these SLA’s may be configured to demonstrate IT’s achievements against the business KPI’s.
- Most of them sound like technology metrics. While this is true, these are a short list of technology metrics that these audiences care about. Notice some frequently reported, but missing measures: average speed of answer at the service desk, mean time to restore service etc. These would be IT metrics that support teams would need, but not IT management or the business, unless IT is failing to deliver on the metrics listed in the matrix and management wants to dig down to discover the reasons.
Step 5: Build the dashboards and scorecards
Once the matrix is agreed on and the method of measuring each KPI is defined, documented and agreed on by the focus group, the final step is to design dashboards and scorecards that represent these KPI’s. These are both graphical views of the Key Performance Indicators listed above, showing the result in comparison to the target. The main difference between the two is in the delivery:
- Dashboards are dynamic: live representations of the data, often provided via a web portal that is integrated either to a measurement tool or directly into an ITSM tool.
- Scorecards are static: they provide a historical look at the data including trending over a period of time.
There are two final aspects of using this framework:
- Continual improvement
- Measurement retirement
As these dashboards and scorecards are used by the business, it’s important to come back to the focus group to evaluate the results. This may lead to creating new KPI’s or tweaking the ways in which they are measured, depending upon the focus group’s satisfaction with performance. In the case of the sample organization, it’s possible that the business is not meeting their objectives and may initiate changes to their critical success factors that will drive a need to change the measures. The point here is that you should not build the dashboards and scorecards then forget about them. Rather, you should meet with the focus group quarterly to review the metrics programs and IT’s achievements. This is a great opportunity to talk about service improvements that the business might need to support the initiatives as well.
Knowing when to stop delivering a dashboard or scorecard report is the last critical piece to a successful program. Once IT is reliably meeting the targets set by the business for a particular goal, it’s a good idea to discuss this result with the focus group during the quarterly review. In this case, you’re not looking at changing CSF’s and KPI’s to address a business need, but rather you’re reviewing the KPI’s to see if the business still needs to see them continually and if any of the targets need adjustment.
Bear in mind that once you are achieving targets reliably, the business might want to work with IT to “up the game.” So in the sample organization, once the security patches and PCI audit result SLA’s are being met consistently, the business might want an shorter SLA for deployments of new features to the website. Thus, the matrix would be adjusted and the appropriate changes made to the dashboards and scorecards.
Benefits of the program
Providing metrics that are responsive to your business’ needs rather than the same stale set of IT metrics they don’t really care about will have a significant impact on the relationship between you and the rest of the business. Looking back at the reasons to measure, you can expect the following results:
- Direct: Live dashboards also provide the ability to determine the activities needed to drive success of an initiative and whether these activities are providing the expected result,
- Validate: You and your stakeholders are able to use the metrics you provide to validate whether IT’s performance is contributing to the business’ ability to meet their goals and objectives,
- Justify: IT is able to produce metrics that support a business case for infrastructure or development projects related to the delivery of a service,
- Intervene: Live dashboards provide IT and the Business to know when there is a performance issue and they can intervene immediately to turn the problem around.
This helps an organization move from a purely reactive mode to a more proactive approach that is integrated with the success of the business’ initiatives in mind.