In today’s fiscal environment, efficient and effective testing is essential. Often, military system requirements are defined using probability of success as the primary measure of effectiveness – for example, a system must complete its mission 80 percent of the time; or the system must detect 90 percent of targets. The traditional approach to testing these probability-based requirements is to execute a series of trials and then total the number of successes; the ratio of successes to number of trails provides an intuitive measure of the probability of success. However, this method of testing has proven to be cost prohibitive, especially at high levels of statistical confidence and power. Often, one or more continuous metrics empirically related to the probability based metric provide more information about system performance than the pass/fail construct. Using these metrics in lieu of the probability-based metrics to plan testing both reduces test costs and provides a better understanding of system performance. In this talk the authors discusses the cost of using binary test metrics (e.g., success or failure, hit or miss). They present several common T&E examples, translating the original probability based requirement to a related continuous metric, and show potential cost savings and information gain achieved by the conversion.
Suggested Citation
Freeman, Laura J, and V Bram Lillard. Continuous Metrics for Efficient and Effective Testing. IDA Document NS D-4571. Alexandria, VA: Institute for Defense Analyses, 2012.