I just read a great Dr. Dobb’s article on testing. Here are a few of my thoughts on the subject.
In Breaking Away From The Unit Test Group Think, Cedric Beust made some excellent points. He states that today you are unlikely to find a developer who does not think that writing tests for their code is part of his/her job description.
In general this is true, but I think the degree of testing thoroughness exhibited by developers covers a large range. We have all worked with developers that commit code that has only been tested for the nominal cases and not for the edge cases. I have always thought that what separated a developer from a great developer are two things: the ability to decompose a problem into a coherent design and the ability to identify the edge case for that design. Let’s be honest, all software people are not skilled at both of these disciplines.
“A unit test is just a convenience that allows you to track down bugs faster. At the end of the day, the reason you write tests is to make sure that your users will be able to be productive with your application, not to make sure that you can debug faster.”
“If there is one thing you should keep in mind about unit tests, it’s this: Unit tests are a luxury.”
I disagree with Cedric’s basic premise on the value of unit testing. I believe that there are two major benefits provided by unit testing. First, it provides a structured way to develop code and it results in more consistent and thorough testing than if it is not done. Secondly, when used as a regression test, it gives an organization the flexibility to re-factor code over time. If edge cases exposed by each identified bug are added to the unit tests, then the unit can be re-factored with confidence.
That being said, for applications that are not safety critical, 100% unit testing could be considered a luxury with an uncertain return on investment. In these cases I would advocate implementing unit testing for those modules that are the most critical to the application, the most complex, and have a history of containing the most bugs that get reported from the field.
“Running code coverage tools is a good practice, but you should exert caution when analyzing the reports. I recommend reviewing which pieces of your code base are not being covered and deciding for yourself whether not testing a particular area is reasonable or too risky.”
This is a great point. I consistently get asked by customers that are not building safety or life critical applications:
What is a good goal for code-coverage percentage? Is 80% good enough? 90%?
Cedric’s point here is exactly right. Code coverage is a great metric for determining the adequacy of your testing, but it should be interpreted within the context of return on investment.
For organizations that have not historically done code coverage analysis in conjunction with their functional testing, the initial results can be a great help for them to understand what they are not testing.
Many organizations with legacy applications, and legacy test suites, will find that they are not testing significant functional areas of their application. I strongly believe that the only way to adequately assess the completeness of testing is via tool-based code coverage analysis.
Recent Comments