Software Testing Metrics

What are Software Testing Metrics?

A software testing metric indicates the degree to which a process, component, or tool is efficient. Here we have a quantitative measure of the effectiveness of an approach. Let's say we automated certain test cases and we have 50% test coverage using tool A. Next time the tool was changed and manual testing was also used so test coverage was 66%. Now we have solid data to justify why manual testing in combination with automation would be better to test the product.

Why Test Metrics are Important?

1. Software testing metrics are used to increase the overall productivity of the development process.
2. It helps to make more informed choices about the tools and technologies being used.
3. It helps to identify unique ways and techniques that are beneficial for their system, hence increasing performance.
4. Software testing metrics determine the health of a process, tool, and approach used.

Types of Software Testing Metrics

There are three types of software testing metrics:

1. Process Metrics: The quantitative measures that define the efficiency of a process based on parameters like speed, time, utilization of resources, etc. Managers can look at these numbers and set new guidelines for the phase.
2. Product Metrics: Measures to determine the quality, size, performance, and efficiency of the product come under product metrics.
3. Project Metrics: Quality and productivity of the project, utilization of resources, cost, and time come under the project metrics.

Manual Test Metrics

Since manual testing is a step-by-step testing process carried out by a quality analyst, finding quantitative measures in the process are little different. These metrics are manual test metrics. There are two types of manual test metrics:

1. Base Metrics: The essential data taken out via the carrying testing process comes under base metrics. It comprises test cases and test cases completed.
2. Calculated Metrics: The base metrics data is further taken out to carry differential results that provide more information about the process or product. It is more useful for tracking project progress.

Some Other Metrics

• Defect metrics: The measures telling about defect ratio, speed taken to fix a defect, and complexity of a defect come under defect metrics.
• Schedule Adherence: Schedule Adherence tells the expected time given for a task vs the time taken to complete it.
• Defect Severity: Defect severity talks about how much the impact that defect/bug has on the product.
• Test case efficiency: Test case efficiency covers how well a test case can determine the impact of a test case.
• Defects finding rate: It tells what is the pattern of flaws over some time.
• Defect Fixing Time: The time difference between when the defect was assigned vs when it got fixed.
• Test Coverage: How much of the requirements and functionalities are covered in the testing process comes under test coverage.
• Defect cause: The modules or components causing the defect.

Metrics Life Cycle

Let's look at various stages of the test metrics lifecycle:

1. Analysis: The QA team identifies the metrics like time, effort, efficiency, etc.
2. Communicate: Communication among the testing team as to how to capture the metrics, the process of it, and what all data needs to be extracted.
3. Evaluation: All the calculation of the data happens here. Preparation of these metrics is done.
4. Report: These metrics are reported. Data is compared, and loopholes and analysis of the report are done. Measures to improve are discussed so that next time the process would be more effective.

Formula for Test Metrics

1. Test Case Effectiveness: Test Case Effectiveness = (Total defects detected / Total number of test cases) x 100

2. Passed Test Cases Percentage: Passed Test Cases Percentage = (Total number of tests passed / Total number of test cases) x 100

3. Failed Test Cases Percentage: Failed Test Cases Percentage = (Total number of test cases failed / Total number of tests executed) x 100

4. Blocked Test Cases Percentage: Blocked Test Cases Percentage = (Number of blocked/skipped tests / Total number of test cases) x 100

5. Fixed Defects Percentage: Fixed Defects Percentage = (Total number of defects fixed / Total number of defects) x 100

6. Rework Effort Ratio: Rework Effort Ratio = (Rework efforts spent in that phase/ Total efforts spent in that phase) x 100

7. Accepted Defects Percentage: Accepted Defects Percentage = (Defects Accepted by Development Team / Total number of Defects Reported) x 100

8. Defects Deferred Percentage: Deferred means that we know that there is a bug but would fix that bug in the next release Defects Deferred Percentage = (Defects deferred / Total number of Defects Reported) x 100

Examples of Software Testing Metrics

Let’s take an example to calculate test metrics:

1. Percentage of test cases executed = (No of test cases executed / Total no of test cases written) x 100 = (180 / 200) x 100 = 90%

2. Test Case Effectiveness = (Number of defects detected / Number of test cases run) x 100 = (10 / 180) x 100 = 5.55%

3. Failed Test Cases Percentage = (Total number of failed test cases / Total number of tests executed) x 100 = (80 / 180) * 100 = 44.44%

4. Blocked Test Cases Percentage = (Total number of blocked tests / Total number of tests executed) x 100 = (0 / 180) * 100 = 0%

5. Fixed Defects Percentage = (Total number of flaws fixed / Number of defects reported) x 100 = (10 / 10) * 100 = 100%

6. Accepted Defects Percentage = (Defects Accepted as Valid by Dev Team / Total Defects Reported) x 100 = (10/ 10) * 100 = 100%

7. Defects Deferred Percentage = (Defects deferred for future releases / Total Defects Reported) x 100 = (0 / 10) * 100 = 0%