Jeffrey Fortin

February 21, 2018

Continuous testing (or CT) is often talked about in the context of a modern DevOps software development environment. The DevOps approach has been used very successfully in product domains that involve web and mobile applications. For these applications, it is expected to have frequent updates as well as constant quality reports from the users of the applications. This environment allows for and demands that quality be quick, efficient and effective.

Typically, software updates are received by thousands (perhaps even millions) of users and the market reaction is swift. A software update that does not meet quality expectations can have a dramatic impact on the market share and brand perception of a product or a company.

For software developers in highly regulated, safety-critical industries such as aerospace/defense, automotive, industrial controls, automotive, and more, applications either did not traditionally contain a high degree of software content, or that software content wouldn’t be required to be updated often. However, as we enter the software-defined world, that way of business is significantly changing. The amount of software-controlled electronics in every industry is increasing exponentially and the market must adapt.

There is an increasing amount of safety-and business-critical equipment in these industries that are leveraging the flexibility and interoperability of a software-based implementation, and this requires regular software updates. Software also now provides a great opportunity for these companies to differentiate themselves. However, unlike their web and mobile application counterparts that may have a lifecycle of a week or less, these devices are typically subject to stringent safety regulations and have much longer product lifecycles. For example, it is not uncommon for a railroad engine to have a lifecycle of 20-30 -- or maybe even more than 100 years. Not only are these machines expected to be in operation for an extended time period, but they must provide value for that extended period of time. Users of these products are also highly critical of poor quality, where the impact can be downright disastrous.

Why should you test continuously?

A key benefit of automated and continuous testing is that it enables developers to find mistakes as soon as they are made – before it is too late. Additionally, this type of testing provides the ability for everyone to participate in the solution. Continuous testing also enables users to build metrics to assess release readiness, so there is information available at any instance to determine if the software is ready to be released at that point.

The path to continuous testing

Measurement is an important consideration in designing any effective test process. A good starting point is to assess what percentage of your tests are currently automated. If an organization is performing a lot of manual testing for example, continuous testing might not be the right path at this time. Another part of this equation is to look at what percentage of the testing could be automated to implement a continuous test process. If there isn’t a lot of testing required, it would be a good idea to evaluate what can be done to make it automated. It is also key to evaluate how much overhead (in terms of time) testing takes. If it takes an engineer half a day to run a test, the chances of them doing it repeatedly are going to be pretty low. And finally, it is always important to assess testing completeness as it drives release readiness. For an in-depth discussion on this process, see my previous blog post.

What criteria can you use to assess if continuous testing is right for you?

In your organization, who or what skills are needed to run any test? Is there a QA group or test group that has very technical knowledge typically needed to run a test? A move toward automation will help enable anyone to run tests, which is a tremendous benefit. Additionally, how often are test results published? If you are running all of these tests and no one finds out the result until the product is ready to ship (or unfortunately, not ready) - continuous testing can help move this process from very rarely to frequently. Most importantly, is the organization meeting its quality aspirations? If all the tests have been run and everything passes but customers are still disappointed, then either the tests or requirement are ultimately wrong. Can you continuously improve the quality by adding back customer assessments of your product into your quality program?

What to do about it
In the embedded space, there is no one simple software development environment, so first and foremost, when designing your test process, use the right tool for the job. There are a lot of options available from static analysis, tools that help build automation such as Jenkins, requirements management systems, and our own complete test automation solution, VectorCAST. Do your research and find the one that fits best for your environment.

Next, start small and make incremental improvements over time. If you have a focused plan, try something and see if it works. If it performs well, move on to the next step. During the process, establish a goal and a timeline for your objectives. Higher quality alone isn’t a goal- you need to understand why you want improve quality. As you do these stepwise improvements, measure where you are and see if you are moving towards your goals. Set up a quality assessment such as the one shown below. Establish metrics for each one of the sliders. Keep making the process better and watch to the sliders move towards your goals. Once you reach them, set more aggressive targets or thing of new things you want to improve.

Software quality will be the number one product differentiator in the software defined world we are living in. The projects that can master quality with an efficient and repeatable process will be the winners.