Artificial Intelligence (AI) has transformed the world of technology, enabling systems to learn, adapt, and make decisions without explicit programming. From autonomous vehicles to medical diagnostics and flight control systems, AI promises unprecedented efficiency and capability. However, when it comes to safety-critical systems—where failure could result in injury, loss of life, or significant damage—the use of AI introduces profound challenges that go far beyond traditional software engineering. Unlike conventional software, which behaves predictably according to its programmed logic, AI is built on learning and training. Its decisions and outputs depend heavily on the data it has been trained on and the patterns it recognizes during runtime. This adaptive, data-driven behavior means that an AI system’s responses may vary with changing inputs or environments, often in ways that are not explicitly defined or foreseen by developers. While this flexibility is a strength in many applica...
In folklore, a bullet cast from silver is often the only weapon that is effective against a werewolf, witch, or other monsters. Software product, just like a werewolf, transforms into something disastrous and unfamiliar. It is a monster, disguised as something working but revealing its true face at other times, requiring a silver bullet to be laid to rest. Do we have such a silver bullet for software bugs? The answer is not yet. This analogy has been taken from popular research paper "No Silver Bullet — Essence and Accidents of Software Engineering" by Fred Brooks.
Software testing is the most crucial and fundamental part of software development. Organizations allocate maximum resources and budget on careful testing of the software. A famous saying by Bill Gates is a clear manifestation of the importance of software testing in Microsoft: "50% of my company employees are testers, and the rest spends 50% of their time testing". A small dormant bug, which is not revealed in testing, can produce catastrophic results during active usage in terms of human loss and financial damages. Researchers in software industry have spent great deal of effort to make software testing as efficient and thorough as possible. But the fact is, software testing is incomplete. Only exhaustive testing is exhausted, which is nearly impossible to achieve.
Software testing can reveal the presence of bugs but not guarantee their absence. We cannot have a test suite which strictly guarantees that the software product is bug free. Even the test suite which achieves 100 percent of strongest coverage criteria i.e. path coverage may fail to find certain bugs. We can simply say that testing is best effort optimistic approximation of the correct behavior. This is because software testing covers only a small subset of input domain which may not be enough to reveal a certain bug. Consider the following simple code:
int i;
read (i);
print (10 / (i - 3));
For the example code above, the test suite consisting of inputs other than i = 3 will achieve 100 % path coverage but not reveal the bug. Note that this is a simple 3 lines program. It becomes nearly impossible to cover substantial part of the input domain and achieve high coverage for a million lines program. 100 percent coverage for large software programs is generally not possible due to the presence of unreachable code. It is important to remember that code coverage is just an objective measure of the likely effectiveness of the test suite. 100 percent coverage does not mean all the bugs have been found in the program. Safety critical software applications are often required to demonstrate that testing achieves 100% of some form of code coverage.
Figure 2: MC/DC Code Coverage Required by DO-178B
All techniques of Black Box functional testing are focused to achieve maximum coverage of the program code under test. Straw man based approach of exhaustive testing is very inefficient and nearly impossible. Consider that it will take approximately 600 years to exhaustively test a simple function which prints a sum of two integer values. Random testing approach is based on selecting the input randomly without any designer bias. Both the Random and exhaustive testing techniques consider a lot of redundant and isomorphic test inputs. Furthermore, it has been observed that bugs are not randomly distributed across the entire program input domain. Therefore, Systematic testing is found to be the most effective approach to achieve maximum code coverage and find bugs, generally located at the boundaries of input domain partitions. Systematic testing can be automated to simplify the test data generation and execution process and greatly minimize the cost of catching regression errors. But the truth is that the automated systematic testing approach may also not be sufficient to catch certain functional, logic, structural and programming errors.
Testing is generally complemented with other software quality techniques for completeness. Static analysis, dynamic analysis, code reviews, walkthroughs and formal proofs of correctness are some of the techniques used to complement software testing. These techniques are helpful to catch certain bugs like memory leaks, array bounds, pointer errors, exceptions, buffer overflows and coding errors which are often missed by functional testing . Static and dynamic code analysis tools are an integral part of modern compilers and they are pretty complete in catching the kind of bugs they are designed to catch. The problem with code analysis is that it generates a lot of false positives which may never occur in real world scenarios. Researchers in the field have been trying to improve software analysis to reduce the number of false positives. Some research is also focused on bringing a leap forward in technology by designing analysis tools which can prove the absence of bugs. But, as of today, there is no such single software quality solution which may be considered a silver bullet for software bugs.
Comments
Post a Comment