Skip to main content

Challenges of Using Artificial Intelligence in Safety-Critical Systems

Artificial Intelligence (AI) has transformed the world of technology, enabling systems to learn, adapt, and make decisions without explicit programming. From autonomous vehicles to medical diagnostics and flight control systems, AI promises unprecedented efficiency and capability. However, when it comes to safety-critical systems—where failure could result in injury, loss of life, or significant damage—the use of AI introduces profound challenges that go far beyond traditional software engineering. Unlike conventional software, which behaves predictably according to its programmed logic, AI is built on learning and training. Its decisions and outputs depend heavily on the data it has been trained on and the patterns it recognizes during runtime. This adaptive, data-driven behavior means that an AI system’s responses may vary with changing inputs or environments, often in ways that are not explicitly defined or foreseen by developers. While this flexibility is a strength in many applica...

How Traceability Helps Uncover Bugs in Unused Code in Safety-Critical Software

How Traceability Helps Uncover Bugs in Unused Code in Safety-Critical Software

In safety-critical software—whether in avionics, automotive systems, medical devices, or industrial automation—the margin for error is essentially zero. Every line of code must exist for a clearly defined purpose, and that purpose must be rooted in an approved requirement. This strict discipline is vital not only for certification, but also for ensuring that the system behaves predictably under all operating conditions.

One of the most overlooked sources of defects in such systems is unused or dead code—software elements that do not correspond to any requirement and are not executed during normal operation. While such code may appear harmless, it can introduce significant risks. This is where end-to-end traceability plays a powerful role.

What is Traceability in Safety-Critical Software?

Traceability refers to the ability to map each software artifact—requirements, design elements, source code, object code, test cases, and verification results—back to one another in a bidirectional and auditable manner.

In other words, traceability answers crucial questions such as:

  • Which requirement does this code implement?

  • Which test case validates this requirement?

  • Is every requirement fully implemented and verified?

  • Is there any code that does not map to a requirement?

In the context of safety-critical development, traceability from requirements to object code and test cases is not a documentation formality—it is a mandate. It ensures that:

  • every requirement is implemented;

  • nothing extra is implemented;

  • every implementation is properly tested.

This discipline is fundamental to preventing unintended functionality, which is one of the most dangerous failure modes in high-integrity systems.

Why Traceability Is Crucial: Eliminating Unused Code

Unused or unreferenced code—sometimes left over from debugging, experimental features, future upgrades or legacy modules—poses a non-obvious but serious risk. Even if the code is not actively invoked, it still consumes memory space, contributes to system complexity, and may unintentionally execute under unforeseen conditions.

For example:

Unused code added for debugging or testing purposes may accidentally remain in the final build. This code might consume excessive CPU or memory resources, preventing the actual safety-critical software from executing properly. Such issues may surface in safety-critical environments, where resource constraints and deterministic execution are essential.

Even when unused code does not directly execute, it can lead to:

  • unexpected control flow

  • compiler optimization anomalies

  • security vulnerabilities

  • faulty integration behavior

  • misleading coverage results

Traceability helps teams identify such fragments early in development—long before they propagate into integration or certification phases.

How Standards Mandate Traceability

Most international safety standards explicitly require strong traceability as part of the certification process:

  • DO-178C (avionics)

  • ISO 26262 (automotive)

  • IEC 62304 (medical software)

  • IEC 61508 (industrial systems)

These standards insist on:

  • complete and correct traceability from requirements → design → code → object code → test cases

  • verification that no extraneous code exists

  • evidence that all code is linked to a requirement

  • justification and removal of dead or deactivated code

  • structural coverage measurement to identify untested or unreachable code

In DO-178C, for example, the concept of "extraneous code" is explicitly defined, and its presence is considered a certification-blocking non-compliance issue.

What Issues and Bugs Can Traceability Help Identify?

Traceability data reveals numerous kinds of issues, including:

1. Unused / Dead Code

Code segments not mapped to any requirement or test case. Such code may:

  • remain from prototyping

  • represent outdated functionality

  • include leftover debug routines

  • contain resource-consuming loops or handlers

2. Unreachable or Untested Code

Detected through structural coverage analysis (e.g., statement, branch, MC/DC). These reveal:

  • impossible conditions

  • logical oversights

  • missing test scenarios

  • unnecessary branches

3. Incorrect or Over-Implemented Features

When code performs more than the requirement intended—leading to unintended functionality.

4. Missing Implementation

Traceability also reveals gaps where a requirement has no corresponding code.

5. Misaligned or Outdated Test Cases

Test cases that do not map to any requirement often signal inconsistencies in the verification process.

Together, these findings significantly strengthen the safety assurance argument and contribute to more predictable system behavior.

How Traceability Is Measured and Practiced in Real Projects

In practice, traceability is managed using a combination of tools, processes, and verification workflows:

1. Requirements Management Tools

Tools such as DOORS, Jama, Polarion, and JIRA provide bidirectional traceability between requirements and downstream artifacts.

2. Static Analysis Tools

Static analyzers help identify unused or unreachable code automatically, contributing to the traceability matrix.

3. Structural Coverage Analysis

For safety-critical systems—especially in avionics—coverage analysis at the object code level is mandatory. Tools such as:

  • VectorCAST

  • LDRA

  • Rapita RVS

  • Cantata

highlight which sections of code were executed during testing and which remain untested.

4. Traceability Matrices

A traceability matrix consolidates all links between:

  • requirements

  • design elements

  • source code

  • object code

  • verification procedures

  • test results

This matrix becomes central to demonstrating compliance and identifying discrepancies.

5. Manual Reviews and Code Walkthroughs

Even with automation, human oversight remains essential—particularly for identifying dead code that may not be evident in automated tools.

Conclusion

Traceability is more than a certification checkbox; it is a fundamental engineering practice that ensures safety, correctness, and predictability in critical systems. By enforcing a direct relationship between requirements, implementation, and verification, traceability reveals hidden issues—particularly unused code—that can introduce subtle and potentially catastrophic defects.

Unused code may appear harmless, but in tightly controlled, resource-constrained environments, it can:

  • disrupt memory or CPU usage

  • create unexpected control paths

  • generate misleading coverage metrics

  • undermine system reliability

Through disciplined traceability supported by static analysis and structural coverage tools, development teams can detect, analyze, and eliminate such risks early in the lifecycle.

In the realm of safety-critical software, every line of code must be justified—and traceability is the mechanism that ensures it.

Comments