In pursuit of code quality: Don’t be fooled by the coverage report

I know this is not a new paper. It’s from 2006. But my inbox is really deep!

In any event, the points made in this paper are applicable to software development . . . → Read More: Don’t be fooled by the coverage report]]>
Last week, someone sent me a link to this paper:

In pursuit of code quality: Don’t be fooled by the coverage report

I know this is not a new paper. It’s from 2006. But my inbox is really deep!

In any event, the points made in this paper are applicable to software development and testing regardless of the industry or the language used. the paper reinforces the message that we use when introducing our VectorCAST/Cover product to potential clients. Specifically the following three points:

Code Coverage Tools are an “easy addition” to a developer’s tool kit
This is really a key point. Unlike many tools that require a change in process, code coverage tools can run silently behind the scenes while you perform all of your existing test activities. The setup of the code coverage tool can all be built into the build environment. If this is done properly testers will not even realize that they are testing a build with code coverage enabled.

There is a disconnect between coverage percentage and test quality
This is the central point of the paper. That is, that relying on the fact that you have achieve a high level of code coverage does not mean that your application has been thoroughly tested. The author uses may examples to make his point and I won’t repeat them here.

This point causes confusion with many organizations as they look to adopt code coverage for their projects. We often get asked: “what percentage of code coverage should we achieve”. As the author correctly points out, this looks at the question from the wrong side. Code Coverage should not be the goal of testing. Testing should occur in the context of proving that the application, sub-system or unit are implemented correctly. Once the correctness of the application is established then code coverage can be examined to determine if the testing is complete

What does missing code coverage tell us? Well the obvious thing that it tells us, is that part of the application has not been tested. Analysis is required to answer: “why has it not been tested”. There are three general reasons for the lack of code coverage: oversight by the testers, inadequate requirements, and dead code.

Different Code Coverage Analysis Catches Different Problems
The author uses several examples to show the deficiencies of statement coverage and branch coverage; specific poorly constructed statements and conditions that will not be exercised if you only get statement coverage or if you only get branch coverage.

This point is exactly why industries such as avionics have adopted different coverage mandates for different levels of criticality in the software being tested. For example, the MC/DC (Modified Condition / Decision Coverage) catches problems like short circuit evaluation and function calls nested in conditionals and is required for the most safety critical systems in Avionics, Railway, Medical, Automotive, and Industrial Controls.

I encourage you to read this paper, especially the hypothetical situations regarding estimating the time to modify existing code. Anyone who has been in this industry for a few years will get a chuckle out of that!

The message of any organization being good at everything is so unreasonable that it’s funny. Any organization that is truly exceptional understands their core competencies and they work . . . → Read More: “We Specialize in Everything”]]> Driving to work this morning I passed a billboard for a local hospital. The tag line was really funny to me:
“Specializing in everything.”

The message of any organization being good at everything is so unreasonable that it’s funny. Any organization that is truly exceptional understands their core competencies and they work to build their products around those capabilities. The concept of being really good at a few things applies to businesses of all sizes. When Jack Welch was at GE he used to say that he did not want to be in any business where GE could not be in the top three.

This made me think about the software tools business and VectorCAST. We have a really simple goal; constantly make our products better and focus on what we know a lot about: embedded software testing.

There was a time in the software tools business when customers wanted a single unified solution from a single vendor — kind of the IBM model. Today customers are smarter. They want best of breed point solutions for each tool with the ability of the tool to inter-operate and share data. Organizations have come to realize that no one vendor has the best technology for every aspect of software development tools.

Raymond Kiefer, General Motors Active Safety Technical Fellow: “Vibrating alerts may help drivers . . . → Read More: News: 2013 Cadillac to feature vibrating safety seat]]> GM has announced that their 2013 Cadillac XTS, ATS and SRX luxury sedans, will feature a vibrating driver seat to help prevent crashes. GM’s Directional Tactile Sensation feature is designed to warn the driver of crash threats while driving and parking.

Raymond Kiefer, General Motors Active Safety Technical Fellow: “Vibrating alerts may help drivers who do not hear beeping alerts due to hearing loss or competing noises, and may be preferred by drivers and passengers who might be annoyed by beeps and shut crash-avoidance features off.”

SOURCE

General Motors is recalling the 2012 Buick Regal for, according to the National Highway Traffic Safety Administration, “failing to comply with the requirements of federal motor vehicle safety standard number 108: lamps, reflective devices and associated equipment.”

The NHTSA states a software error is responsible for the defect; causing the parking lamps to . . . → Read More: News: Software defect to blame for 2012 Buick Regal Recall]]>

General Motors is recalling the 2012 Buick Regal for, according to the National Highway Traffic Safety Administration, “failing to comply with the requirements of federal motor vehicle safety standard number 108: lamps, reflective devices and associated equipment.”

The NHTSA states a software error is responsible for the defect; causing the parking lamps to fail to activate when either the headlamps or parking lamps are turned on.

While this defect is not a major issue, it could reduce the visibility of the vehicle and increase the potential for a crash.

According to Consumer Reports, “this flaw violates U.S. safety standards and may cause an increased risk of vehicle collisions due to reduced visibility of the Buick car.”

SOURCE

Today, with RSC approval, the LynxOS-178 Version 2.2.2 . . . → Read More: News: FAA Reaffirms Approval of LynuxWorks LynxOS-178 Reusable Software Components RTOS for Safety-Critical Applications]]>
San Jose, CA — [via] – LynuxWorks, Inc., the first and only embedded operating system vendor to receive Advisory Circular AC 20-148 approval from the Federal Aviation Administration (FAA) for reusable software components (RSC), has received its second RSC approval for the LynxOS-178 product family.

Today, with RSC approval, the LynxOS-178 Version 2.2.2 RTOS (real-time operating system) can be more extensively used by avionics systems integrators and embedded developers for the operating system portion of the safety-critical software code and supporting DO-178B artifacts for reuse in other system designs with other software components without the need for full recertification.

“More than ever, software component use in safety-critical software for embedded military applications or avionics flight systems is critical,” said Stephen Balacco, director, Embedded Software and Tools Practice, VDC Research Group, Inc. “With RSC approval, the software components identified as reusable are already approved by the FAA.” The OS provider chooses software components that are target hardware-independent, such as the kernel and libraries, which can constitute as much as 65% of the code. From that perspective, RSC approval provides system integrators and developers a tremendous head start when a significant amount of the operating system is already approved.

According to the VP of Engineering for LynuxWorks, Arun Subbarao, “LynxOS-178 was designed solely for safety-critical applications. The standards-based approach for software reuse in airborne systems and equipment offers a ‘software black box’ solution that can significantly reduce the time and cost of achieving FAA certification and further reduce the risk involved in redevelopment efforts across multiple safety-critical systems.”

In 2006, the LynxOS-178 RSC RTOS was initially approved for the Rockwell Collins Adaptive Flight Display Runtime, Common Computing Runtime, and Data Concentration Module Runtime for Pro Line Fusion. Rockwell Collins offers a family of flexible avionics system solutions designed to address a wide range of aircraft and missions — from light turboprops to long-range business jets, from commercial helicopters to special missions aircraft.

A Standards-Based Approach for Reuse LynxOS-178 addresses system developers’ need to meet FAA certification by providing a time, space and resource partitioned commercial off-the-shelf (COTS) operating system that fully satisfies the objectives of the DO-178B, level A and the ARINC 653 standard as well as full conformance with the IEEE POSIX Standard. Currently, any time a systems integrator or developer wants to integrate components of any other RTOS, or any previously certified DO-178B software in multiple hardware environments, each new hardware/software configuration or system must be recertified. In other words, integrators and developers cannot take full advantage of the software’s existing certification across other multiple platforms, but instead must go through the entire certification process for each new system they build. The RSC acceptance letter from the FAA for LynxOS-178 can be used to support virtually any FAA Technical Standard Order.

SOURCE

Using VectorCAST with Real Time Embedded Linux Operating Systems

The 40-feet long railgun launches the projectile out of the barrel at 4,500 to 5,000 miles-per-hour — over . . . → Read More: US Navy Tests Futuristic Railgun]]> After six years of development the U.S. Navy’s futuristic electromagnetic railgun is one step closer to reality. Vector Software customer BAE Systems, has developed a prototype of a new-generation, long-range weapon for surface ships.

The 40-feet long railgun launches the projectile out of the barrel at 4,500 to 5,000 miles-per-hour — over six-times the speed of sound. The 40-pound projectile travels upwards of 110 nautical miles in about 5 minutes. That is roughly the distance from Washington, DC to Philadelphia. This new weapon consists of parallel rails and uses an electromagnetic pulse to create magnetic fields that propel the projectile.

The Navy’s most advanced shipboard gun in operation today, the 5-inch, 54-caliber lightweight gun, has a range of about 13 nautical miles.

Mark Signorelli, BAE Systems Vice President and General Manager, Weapon Systems: “This event marks a monumental achievement for our company and the Office of Naval Research. We are excited to work with the U.S. Navy to bring this revolutionary and game-changing capability to our Sailors and Marines.”

SOURCE

The Striker Integrated Display Helmet is fitted with an optical head tracker, with targets popping up in the . . . → Read More: New RAF helmet allows pilots to shoot down enemy jets using eyesight]]> Marking one of the biggest leaps forward in attack capabilities in military history, Vector Software customer BAE Systems, has developed a headset that allows RAF pilots to shoot down planes simply by looking at them.

The Striker Integrated Display Helmet is fitted with an optical head tracker, with targets popping up in the visor, which provides a supposedly highly accurate missile fire with low latency, at any altitude.

Mark Bowman, Chief Test Pilot, BAE Systems: We’ve moved on enormously from where we were 20 or 30 years ago. The helmet display is most probably as significant as almost the jet engine in what it is going to do for combat capability and what the pilot can do.

A heads-up display (HUD) has been around for several years but this HUD is inside the visor and moves with the pilot’s head; effectively making the pilot part of the weapons system. The pilot can simply look at a target even if it is behind or below, lock onto it, and then fire – regardless of where the aircraft is pointing.

Mark Bowman, Chief Test Pilot, BAE Systems: Traditionally the helmet has been seen very much as a crash helmet and a walkie-talkie. What we have got now is actually a piece of not only optical equipment but something that is really technically advanced that most probably I, certainly as a young boy, only ever dreamt of but now it is very much a reality.

SOURCE