Insights

Framing Metrics for Success

April 9, 2019

Phil Dempster

Be careful what you measure, how you measure and look out for unintended behaviours of where a metrics gets perceived as a target.

Note: This article was originally posted on our legacy Medium blog, it hs bee moved over and small editorial tweaks made.

‍

“Measure what matters” — John Doerr

‍

For many Organisations there is a tendency to see value in measuring every and anything that is measurable, without investing time evaluating each metric to understand the why, the value it provides and the potential consequences (see "Cobra Effect" in the recommended reading section below).

‍

‍

Metrics are regularly used as ways to control outcomes and behaviours, the more metrics the better! It is common to see metrics chosen without consideration of their possible benefits, negative side effects, or their relevance to their target audience.

Some examples of metrics and measures that encourage or drive particular outcomes and behaviours:

Incentivising individuals on revenue goal — with the desired intent of driving sales to improving the company’s available cash.
Goal developers on lines of code produced — with the belief this will increase productivity and speed of value to customers.
Reward (or punish) teams by focusing on error rate — to ensure quality.

These may seem simple, sensible and logical metrics to measure and they WILL control behaviours but maybe not with the intended results.

Let’s have a quick look again at each of the above

People started making less profitable and higher maintenance sales which eroded margins, hence reducing available cash
Developers started cutting and pasting large amounts of code without thinking about the longer term maintenance, testability, readability, or quality of the code, reducing the overall value delivered.
Measuring errors resulted in a slowdown of work done and a reduction in outcomes released from teams as new inbound work was rejected on the basis it required more discovery or elaboration in order to reduce errors;

There is a lot of information about metrics, and observation systems that let us collect metrics and measure just about anything: people, teams, leaders, sales, e-commerce performance, business performance, finances and the list goes on.

The challenge is not that there aren’t enough things to measure, it’s ensuring we are using and measuring the right metrics to accomplish what we are trying to achieve. How can we be consistent on the type or value of the metric, how do we attempt to detect a metric that cause bad behaviour and how do we create a culture where metrics are used for continuous improvement and achieving.

When thinking about a metric, have an upfront discussion clarifying the language used. Ensure you collectively have a common understanding of measures, the why and the expectation. Share this information throughout the organisation to ensure everyone is starting from the same solid base.

This is not about the actual metric but its definition, intent and scope. Start with framing up metrics with the template below.

‍

"When a measure becomes a target, it ceases to be a good measure" - Goodharts Law (attributed to Charles Goodhart)

‍

Framing a metric

When thinking about a metric consider these seven dimensions, this will help to make them:

Valuable

There has to be a clear and directly associated relationship between the metric and how it will achieve a better outcome. Without this a metric should not be measured as it becomes just because something is measurable metric.

Achievable

If a metric is going to be used to help drive goals, then ensure the goal is honest and achievable. This doesn’t mean that they need to be easy. Setting up unrealistic targets will create defeatist attitudes, while too easy only the minimum effort to reach them will be made.

Realistic

If the metric is going to be created and measured, then it has to be either within a person’ s or team’s control or influence. Metrics outside this zone provide no or negative value as they will shift independently of an individual or team’s actions and deflate morale.

Real Time

The data needs to be as real time as possible, with easy accessibility combined with high visibility. Timely metrics allow for effective learning, ability to quickly change course and minimise the distraction of reporting. Metrics that are slow to collect and disseminate, e.g. are shared monthly/quarterly/yearly provide data that’s often too late to act on.

Opposite

Care and thought needs to be applied to the potential outcomes or behaviours of the metric. The metric may create opposition to the goal trying to be achieved. As in example 1, increase revenue could lead to selling that reduces margins than potentially profitability.

Observer

For many reasons it is important to understand who the observer of the metric will be, the metrics may all be visually available but must directly speak to the metric’s audience, e.g. even though a team’s test time metrics may be seen by anyone, they provide the highest value to the team who has an understanding of their pipeline and the work, they could be useful to others consuming the services from an engineering space, however potentially significantly less value or understood for someone in logistics or finance.

Measurable

Metrics have to be both measurable and as low cost (ideally zero cost!) to measure as possible. This implies automated collection, reducing or eliminating the burdens to collect and administrative overheads. Manually collected metrics quickly become out of date, and often include errors or introduced biases by the collector.

‍

Once there is agreement and alignment on the framing of a metric, it’s time to decide on which metrics to use. It is best to start by creating two or three categories, even if there is only one or two metrics in each category, then collect some data on these metrics.

Make sure you have captured your framing outcome so that others may quickly understand the metric and intent. An example of categories could be Delivery, Customer and Business. Your organisation will have its preferred language so consider what works best in your context.

You can add more metrics and information about your metric as you learn, as well as remove metrics that fade in relevance, are not realistic, drive unwanted outcomes or no longer measurable.

Collecting data is important as it helps to make the metrics real, which in turn evolves the thinking of the metrics purpose and collection process early during their creation. A common trap for teams or an entire business units is to get stuck in excessive planning that gets wasted once data gets collected and fails to meet the expectations of the plan.

‍

Our goal is to also share some examples of framed metrics, the need to pair metrics, displaying metrics as well as some ideas and metrics I have historically used.

We hope that the time you have invested has returned value to you and encourage you to provide real-time and measurable feedback‍