2012 on Test Science Research Document Library https://research.testscience.org/year/2012/ Recent content in 2012 on Test Science Research Document Library Hugo -- 0.129.0 en-us Institute for Defense Analyses Sun, 01 Jan 2012 00:00:00 +0000 A Bayesian Approach to Evaluation of Land Warfare Systems https://research.testscience.org/post/2012-a-bayesian-approach-to-evaluation-of-land-warfare-systems/ Sun, 01 Jan 2012 00:00:00 +0000 https://research.testscience.org/post/2012-a-bayesian-approach-to-evaluation-of-land-warfare-systems/ This presentation is a presentation for the Army Conference on Applied Statistics. The presentation covers a brief introduction to land warfare problems, and devises a methodology using Bayes Theorem to estimate parameters of interest. Two examples are given, a simple one using independent Bernoulli Trials, and a more complex one using correlated Red and Blue casualty data in a Loss Exchange Ratio and a hierarchical model. The presentation demonstrates that the Bayesian approach is successful in both examples at reducing the variance of the estimated parameters, potentially reducing the cost of devising a complex test program. This presentation is a presentation for the Army Conference on Applied Statistics. The presentation covers a brief introduction to land warfare problems, and devises a methodology using Bayes Theorem to estimate parameters of interest. Two examples are given, a simple one using independent Bernoulli Trials, and a more complex one using correlated Red and Blue casualty data in a Loss Exchange Ratio and a hierarchical model. The presentation demonstrates that the Bayesian approach is successful in both examples at reducing the variance of the estimated parameters, potentially reducing the cost of devising a complex test program. The presentation concludes with suggested next steps applicable to the Army Ground Combat Vehicle program.

Suggested Citation

Wilson, Alyson, Lee Dewald, Robert Holcomb, and Samuel Parry. A Bayesian Approach to Evaluation  of Land Warfare Systems. IDA Document NS D-4711. Alexandria, VA: Institute for Defense Analyses, 2012.

Slides:

]]> Continuous Metrics for Efficient and Effective Testing https://research.testscience.org/post/2012-continuous-metrics-for-efficient-and-effective-testing/ Sun, 01 Jan 2012 00:00:00 +0000 https://research.testscience.org/post/2012-continuous-metrics-for-efficient-and-effective-testing/ In today’s fiscal environment, efficient and effective testing is essential. Often, military system requirements are defined using probability of success as the primary measure of effectiveness – for example, a system must complete its mission 80 percent of the time; or the system must detect 90 percent of targets. The traditional approach to testing these probability-based requirements is to execute a series of trials and then total the number of successes; the ratio of successes to number of trails provides an intuitive measure of the probability of success. In today’s fiscal environment, efficient and effective testing is essential. Often, military system requirements are defined using probability of success as the primary measure of effectiveness – for example, a system must complete its mission 80 percent of the time; or the system must detect 90 percent of targets. The traditional approach to testing these probability-based requirements is to execute a series of trials and then total the number of successes; the ratio of successes to number of trails provides an intuitive measure of the probability of success. However, this method of testing has proven to be cost prohibitive, especially at high levels of statistical confidence and power. Often, one or more continuous metrics empirically related to the probability based metric provide more information about system performance than the pass/fail construct. Using these metrics in lieu of the probability-based metrics to plan testing both reduces test costs and provides a better understanding of system performance. In this talk the authors discusses the cost of using binary test metrics (e.g., success or failure, hit or miss). They present several common T&E examples, translating the original probability based requirement to a related continuous metric, and show potential cost savings and information gain achieved by the conversion.

Suggested Citation

Freeman, Laura J, and V Bram Lillard. Continuous Metrics for Efficient and Effective Testing. IDA Document NS D-4571. Alexandria, VA: Institute for Defense Analyses, 2012.

Slides:

]]> Designed Experiments for the Defense Community https://research.testscience.org/post/2012-designed-experiments-for-the-defense-community/ Sun, 01 Jan 2012 00:00:00 +0000 https://research.testscience.org/post/2012-designed-experiments-for-the-defense-community/ The areas of application for design of experiments principles have evolved, mimicking the growth of U.S. industries over the last century, from agriculture to manufacturing to chemical and process industries to the services and government sectors. In addition, statistically based quality programs adopted by businesses morphed from total quality management to Six Sigma and, most recently, statistical engineering (see Hoerl and Snee 2010). The good news about these transformations is that each evolution contains more technical substance, embedding the methodologies as core competencies, and is less of a ‘‘program. The areas of application for design of experiments principles have evolved, mimicking the growth of U.S. industries over the last century, from agriculture to manufacturing to chemical and process industries to the services and government sectors. In addition, statistically based quality programs adopted by businesses morphed from total quality management to Six Sigma and, most recently, statistical engineering (see Hoerl and Snee 2010). The good news about these transformations is that each evolution contains more technical substance, embedding the methodologies as core competencies, and is less of a ‘‘program.’’ Design of experiments is fundamental to statistical engineering and is receiving increased attention within large government agencies such as the National Aeronautics and Space Administration (NASA) and the Department of Defense. Because test policy is intended to shape test programs, numerous test agencies have experimented with policy wording since about 2001. The Director of Operational Test & Evaluation has recently (2010) published guidelines to mold test programs into a sequence of well-designed and statistically defensible experiments. Specifically, the guidelines require, for the first time, that test programs report statistical power as one proof of sound test design. This article presents the underlying tenets of design of experiments, as applied in the Department of Defense, focusing on factorial, fractional factorial, and response surface design and analyses. The concepts of statistical modeling and sequential experimentation are also emphasized. Military applications are presented for testing and evaluation of weapon system acquisition, including force-on-force tactics, weapons employment and maritime search, identification, and intercept.

Suggested Citation

Johnson, Rachel T., Gregory T. Hutto, James R. Simpson, and Douglas C. Montgomery. “Designed Experiments for the Defense Community.” Quality Engineering 24, no. 1 (January 2012): 60–79. https://doi.org/10.1080/08982112.2012.627288.

Paper:

]]> Statistically Based T&E Using Design of Experiments https://research.testscience.org/post/2012-statistically-based-t-e-using-design-of-experiments/ Sun, 01 Jan 2012 00:00:00 +0000 https://research.testscience.org/post/2012-statistically-based-t-e-using-design-of-experiments/ This document outlines the charter for the Committee to Institutionalize Scientific Test Design and Rigor in Test and Evaluation. The charter defines the problem, identifies potential steps in a roadmap for accomplishing the goals of the committee and lists committeemembership. Once the committee is assembled, the members will revise this document as needed. The charter will be endorsed by DOT&E and DDT&E, once finalize. Suggested Citation Freeman, Laura. Statistically Based T&E Using Design of Experiments. This document outlines the charter for the Committee to Institutionalize Scientific Test Design and Rigor in Test and Evaluation. The charter defines the problem, identifies potential steps in a roadmap for accomplishing the goals of the committee and lists committeemembership. Once the committee is assembled, the members will revise this document as needed. The charter will be endorsed by DOT&E and DDT&E, once finalize.

Suggested Citation

Freeman, Laura. Statistically Based T&E Using Design of Experiments. IDA Document D-4548. Alexandria, VA: Institute for Defense Analyses, 2012.

Slides:

]]>