Agile requires a change in mindset from traditional methods to more quick and agile ways of working. In an Agile project we want  to provide business value early and often to customers. But how do shorter and quicker cycles impact quality? How does Agile handle and measure quality?

Agile recommends using velocity and burn-downs. But they both predict the progress of the project not the quality.

This post will try to answer some of these questions- 

1. How does Agile ensure quality is built in the software?

2. How can we measure software quality in agile projects?

3. How can we make sure metrics are used for good and not evil?

How Agile ensures Quality is in-built into the system?

The goal at the end of any sprint should be to provide a working application with minimal or no defects. We do this by adopting the following best practices.

• Each user story has a list of acceptance criteria and all user stories meet the criteria

• Each sprint team member has the same understanding of the user stories

• We use the 3-amigos practice to make sure we have clear requirements and design

• We use code reviews and have good code coverage

However no process is ever perfect and even if all of the above are in place we are still going to have defects. So let’s look at some of the ways we can measure how good or bad we’re doing and also if it’s getting better or getting worse.

How to measure software quality in Agile projects? 

Quality of software can be measured by the number of defects that are pushed to the client or found in production and the number of users it impacts. The key to good quality software is to ensure that critical defects are not released to production

Identifying the right metric is not an easy job. There are many to choose from but depending on your circumstances you need to decide which one is relevant and will best help you obtain the overall objective for your project. Some example agile quality metrics are as follows:

• User Story Acceptance= No of user story accepted by the customer/number of stories *100

• Review Effectiveness = (No. Of Defects found in Review)/ Total No. Of Defects found before Delivery (both Reviews and Testing) * 100

• Defect Leakage= (E/ I+E) *100

• Defect Removal Efficiency = (I/ I+E) *100
  where

I = Pre-delivery (Internal) errors including review defects as well as testing defects and
E = Post-delivery defects (External) or defects found after the release.

• Defect Density =Defects found/Size (actual in Story Points)

• Testing Effectiveness =(No. of Testing defects detected internally / No. of Testing defects detected internally + No. of Testing defects detected externally) * 100

• Test Coverage = % of code covered in automated testing

• And many more

These metrics, when compared with velocity, can give you important insight into the project. You need to choose the right metric or combination of metrics to capture based on your project complexity and type. 

You can also include non-functional metrics that indicate the quality of the product. Some examples are:

• Portability(usability of the same software in different environments)

• Performance(software responsiveness and stability under a particular workload)

• Functionality(how well the software meets the user’s needs)

• Usability(degree of ease of use and learnability, even for people with disabilities)

• Reliability(the software design and functionality can be depended on to be accurate)

• Compatibility(the ability of a piece of software or system to work with another)

• Maintainability(ease with which code can be amended or maintained)

• Security(software is secure from malicious attacks and hackers)

Good vs. Evil?

Agile metrics work best when they are used to serve as a lead indicator to any future problems. The best way to select a good metric is a 3-step process:

1. Consider the goals that you are trying to achieve

2. Develop metrics that would help indicate that you are moving in the right direction

3. Decide how best to capture these metrics

4. 
   

For example, if the goal of the project is to reduce the number of defects in production, use defect leakage to show if you are moving in the right direction or not. This is pretty simple to capture and gives you important insights in to the quality of your deliverable.

From the definition above we can see defect leakage is the number of pre-delivery defects divided by the number of pre-delivery defects and post-delivery defects. You should analyze any downward trends, identify the root cause behind the drop in the metrics and implement the right process improvements to get you back on track. Good metrics should always – give you an opportunity to improve.

Metrics can become an important indicator of your project’s progress and can help you identify process improvements, but at the same time, they can be easily misused or manipulated.

To ensure that you use the metrics for good:

1. Just enough metrics. You would not like your teams to spend a substantial time capturing data leaving development work aside and metrics becoming an overhead for the team.

2. Never measure an individual. Metrics are there to indicate patterns in your project. They should be defined carefully to measure your goals and should not focus on measuring individual’s performance.

3. Never measure something because it is easily available. Just because a measure is easy to generate should not be criteria for capturing these metrics. The criteria should be based on what we described how it aligns with your project’s goals.

4. Use metrics to track the progress of your project and improve the team’s results.  Metrics serve no purpose if they simply exist on your dashboard and are never analyzed to gain insights into the project. Analyze trends to help improve  future deliveries

Let’s take a scenario where a manager saw drop in defect removal efficiency from 95% to 89% while velocity increased from 10 to 12 story points. There were less defects reported by the sprint team before delivery than the last sprint. Higher velocity at the same time suggests that the team has spent more time in ensuring that more stories are delivered which could be compromising quality. It is likely that the effort that the team should have spent on review and testing was instead used for delivering more story points. The Manager needs to correct course at this time otherwise defects will just accumulate and need to be dealt with at the end of the project

Metrics are used for evil when:

1. You start using it for intra-team comparisons. Many factors define team dynamics and if you start using metrics for comparison, this may lead to low team motivation and further performance deterioration.

2. You manipulate data to show better metrics: Many teams know the goals they need to achieve and in fear of comparison or poor performance, tries to show better results by manipulating the data. This defeats the whole purpose of having metrics in place. This can be avoided simply by having just enough metrics, not treating it as a mode of comparisons between the teams and not tagging individuals with these measurements

It is important to have metrics in a project to understand if we are meeting the goals set and take corrective actions if goals are not being met. Metrics give us an opportunity to analyze our performance and help to reduce defect leakage to the customer. While they are vital for a project, it is important that we don’t get obsessed with metrics and they become a burden for the team. Plan just enough metrics which gives you the right information to make  sound planning decisions.