Code Correctness

Jeffrey Fortin

November 02, 2017

Start with the requirements
The first place to start in determining code correctness is to look at your requirements. If you are in one of the regulated industries, the practice of writing clear testable requirements is very familiar to you. If not, it’s worth taking some time to understand the best practices. I’d suggest taking a look at a blog post on the IBM website on “Managing Your Requirements 101 – A Refresher”[1] by Vijay Sankar. The second part of his blog is about writing good requirements. In it, he writes that requirements should, “Establish a common understanding between the project sponsor, customers, developers and other stakeholders.”[2] This common understanding is the basis for code correctness. The code is right when it does all the things expressed in the common understanding and no more. The last part, “and no more” is critical for safe and trusted systems. It may sound good that the code does more than it should, but in fact, that is the definition of a bug. Once the code leaves the enclave of common understanding defined by the requirements, it enters the dark world of defects, crashes and ultimately leads to adoption resistance, high support costs and brand tarnishing.

Having the code do “no more” than what the requirements specify is fundamental to the discipline of integrity levels that we see in the Avionics (DO-178C), Railway (CENELEC 50128) and Automotive (ISO 26262) industries. Each of the respective safety standards calls out the need to perform code coverage analysis from running requirements-based tests to show there is no unintended functionality.

Why is code correctness important?
It may seem obvious that the code should be correct, but actually, it rarely is. The impacts of code being incorrect, meaning not meeting the requirements, include loss of productivity, high maintenance costs and loss of brand value. If the code is being used in a safety-related application, the cost can be loss of life, injury or property.

How can we measure code correctness?
The best way is to test it with requirement-based tests. Having test vectors that can validate the code behavior is consistent with the requirements allocated to the code is the best way to measure code correctness.  Manually creating test vectors and generating the test harness is time-consuming and error-prone. It normally doubles the amount of code you need to write and can easily be as much as ten times more inefficient. It is far better to use a proven test automation platform. VectorCAST has been widely used in high-integrity and safety-critical applications for decades.

What criteria can we use to measure code correctness?
Metrics need to be a part of your software development plan. Using metrics, you can create analytics to inform your project activities. Metrics allow you to make decisions based on facts. It’s too easy for ego and emotions to play a role in determining release readiness. Establish and measure your software using metrics that are important for your project. Examples are number of tests, percent of tests passed, number of requirements and percent of requirements tested. Code coverage metrics do not give information about the correctness of the code directly but can point to areas of the code that have not been tested. These are problem areas that may contain incorrect code and need to be analyzed. You may need to add more tests, more requirements or remove the code.

What steps can we take to improve code correctness?
Follow best practices for coding
In the embedded world that we live in, code is expected to work correctly under all conditions. Making the assumption, “oh that will never happen” is not wise. Embedded systems usually run for long periods of time without restarting and encounter a wide variety of operational conditions. 

Accounting for these long intervals of time and for all operational conditions is one of the things that makes embedded programming difficult to get right. Conditions such as clock rollover, register overflow and register underflow need to be accounted for in the software design. They should be included within the requirements baseline and tests added to see that the software performs correctly, meaning according to the requirements.

Resource dependencies and constraints are the root of some of the most difficult bugs to fix in an embedded system.  They can provide hours of entertainment as you are trying to pass your integration or system testing while the software is locking up or crashing in unpredictable ways. Enforcing access ordering policies can eliminate deadlock conditions. These mechanisms should also be tested according to the requirements. Creating system tests can ensure the implementation is robust. With VectorCAST/QA these tests can be run automatically and by any member of the team, giving you the best chance of finding errors quickly.

Coding standards and conventions such as MISRA C and the JPL Coding standard [3] can help to eliminate undesirable side effects in the code.  With a static analysis tool like VectorCAST/Lint, an automated process can be established to enforce coding standards.

Test-Driven Development (TDD)
If you want your code to be correct, don’t write bad code to begin with. OK, I know that’s an overstatement, but it’s been shown on many software projects that following TDD helps to dramatically reduce the number defects entering the code base, and when they do, they are found quickly at the lowest cost point to recover. TDD helps in two ways. First, it gives you the best chance at writing a good test. Since tests are written right after the requirements, the intent of the requirement is fresh in your mind. The temporal proximity helps to ensure you’re testing for the right things. Secondly, the tests are not biased by the implementation. It’s only natural to look at the code to try and extract additional requirements from the implementation and then codify them into the test base. These then become de facto requirements but are never added to the formal requirements baseline and, in fact, may contradict the stated requirements. This can lead to a contradictory situation where all tests pass implying high quality, but the code doesn’t meet expectations, meaning it actually lacks quality. Establishing requirements-based tests early on with a clear process for change requests gives you the best chance your code will have the best quality possible.

Code correctness starts with the requirements. It’s the requirements that allow you to judge if the code is correct. Put tests in place early on to measure if the code is meeting the requirements. Establish tracking metrics and determine your release readiness criteria from the start. When you are ready to begin coding, use best practices and automatic tools for checking the code for known vulnerabilities. Have an automatic process for checking the code against the test cases. Once the tests pass, stop coding! Writing correct code especially in embedded systems is not easy, but by using these tools and practices, you can greatly improve your chances of success.