DORA Metrics

While most organizations have embraced DevOps developing the right culture, establishing communication channels, and implementing advanced tools, they are still unable to assess the success of their DevOps initiatives. DevOps and engineering leaders recognize DORA metrics as a standard framework for measuring DevOps success. DORA metrics help them measure software delivery throughput (speed) and reliability (quality) accurately. By analyzing the DORA KPIs, teams can easily gain insights into performance trends, detect issues across different stages of DevOps, and take remedial actions to deliver better software, faster. For instance, by tracking deployment frequency, organizations can observe if their teams have improved over a period, remained consistent, or experienced extreme deviations. Similarly, a higher change failure rate can indicate issues with change management, quality testing, and more. With a DORA metrics dashboard, organizations can easily visualize DORA metrics, detect process bottlenecks, perform root cause analysis, and take action for continuous improvements in DevOps.

DORA metrics are a marker of software delivery throughput (speed) and reliability (quality), which eventually correlate with the delivery of higher end-user satisfaction and business value. However, organizations can track business value over a period based on certain focused and custom KPIs. For instance, the metric 'Innovation' can be defined as a function of every feature enhancement delivered to customers, excluding bugs. Tracking this metric can help businesses determine how much they have actually worked towards improving their product or service against removing its existing, lingering flaws. Similarly, Mean Time to Recover (MTTR), which is one of the 4 key DORA metrics, can provide insights into customer satisfaction. Such metrics and their trends over a period can offer organizations data-driven predictability and can be used as a business driver.

Teams tend to misunderstand the purpose of their metrics and start using them as their goals, which is an anti-pattern as cited by Goodhart's Law - "When a measure becomes a target, it ceases to be a good measure." The metrics are the outcome of a team's performance; however, teams tend to game the system and shift their focus to improving their scores instead of real-world performance. Another major challenge with metrics is related to benchmarking; teams often lack awareness of what's a good deployment frequency in their context. Many times, the DORA dashboards are rigid and offer no clues as to what the metrics mean. For example, the DORA dashboard might indicate that the team deployed 200 times in a month but would not offer any insights into past trends or assess the lead time from request to delivery. Such dashboards offer limited flexibility in assessing the flow of deliveries and business value.

Selection of the right metrics can be a complex task. While the DORA metrics offer insights into the end performance of your DevOps practices, they can go only so far. For instance, DORA metrics alone cannot help you identify the bottlenecks in CI or answer why the software isn't always in a releasable state. You might need to track additional metrics to understand how long the branches exist, or how frequently the changes are made to the trunk, how many things are in progress, and so on. Additionally, organizations can measure defect volume and escape rate, failed deployments, change volume, unplanned work, and SLA compliance to get a more holistic view of their DevOps. Gathr can help you create a custom DevOps monitoring tool to meet these requirements.

The selection of the right metrics is the most crucial step in the implementation and analysis of KPIs. For example, improvement in metrics like ‘lines of code’ and ‘number of defects fixed’ may give a false sense of comfort as they don’t necessarily translate to improvements in quality, productivity, and reliability. At times, it is seen that teams tend to focus on improving their scores and may lose sight of the business goals. For example, measuring the ratio of committed vs. completed features can force teams to prioritize schedules over quality. That’s why organizations need to find metrics that link directly to customer satisfaction and not just delivery goals.