Diminishing Returns on Software Quality

Software production, like any other production, obeys the law of diminishing returns. Reducing flaws is part of the software production process so it also falls under this umbrella of diminishing returns. The law of diminishing returns states that when we add additional input into a production process we will see diminished output relative to the inputs at a some point. We can restate this in reference to quality analysis, the amount of time spent QA testing, writing unit tests, or other quality control measures will become less and less effective at reducing bugs and errors the more time is spent after a threshold of time spent.

An example of this would be a QA team is testing a section and they work 4 hours. They spent the 1st hour working and they found 8 bugs, they spend another hour working and only find 4 bugs, after the third hour of testing they found 2 bugs, and in the fourth hour they only find one more bug. We could say that the output diminished each hour they worked.

In the graph below you can see how the number of bugs starts to trail off. This would be an example of how after a certain time the output is decreasing even though more time is spent testing. In this example over 50% of the total bugs found would have been the first hour 80% by the second hour.

Bugs Per Hour Chart

Most often the initial QA testing is the most valuable, and then less and less bugs are found the more testing is done. This shapes the way we approach QA, and why many times QA is cut short, because reduction of flaws becomes more and more expensive, especially with larger products. This information also can shape how a testing process is developed, and how deep or shallow we choose to test each section.

Putting this altogether helps us as developers understand that flawless software is very difficult, and expensive to produce, because QA time becomes less and less effective after a certain threshold. The difficult part is figuring out when QA returns are diminished, and is there a point where it starts to drop off drastically? Every project, and every organization are different, and so this could very greatly depending on a number of factors, QA testers ability, software complexity, domain complexity, organizational factors, etc.

The law of diminishing returns as relates to quality and QA ties into the bigger picture of the project triangle (Scope/Quality, Cost, and Time). If we understand how much time it will take to reasonably test our software we can better balance the quality with deadlines and costs. We know that we may be able to shave 20% of a QA testing budget and not cut into the most valuable QA testing time and can more accurately forecast the risk associated with such a decision.

Post a Comment

Your email is never shared. Required fields are marked *

*
*
Profile Picture

About Ian Lintner


I am a software developer, mostly web,  in Des Moines, Iowa. I take a very opinionated stand concerning development, you will never regret a simple design or architecture. My education was at Drake University in Biology and Computer Science. Offline I am recently married to my wife Heather. I try my hand at many hobbies currently I am gardening till the snow comes in.



My Current Projects


Des Moines Twitter Trends