The ITEA Journal of Test and Evaluation on Test Science Research Document Library https://research.testscience.org/venues/the-itea-journal-of-test-and-evaluation/ Recent content in The ITEA Journal of Test and Evaluation on Test Science Research Document Library Hugo -- 0.129.0 en-us Institute for Defense Analyses Tue, 01 Jan 2019 00:00:00 +0000 Designing Experiments for Model Validation- The Foundations for Uncertainty Quantification https://research.testscience.org/post/2019-designing-experiments-for-model-validation-the-foundations-for-uncertainty-quantification/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-designing-experiments-for-model-validation-the-foundations-for-uncertainty-quantification/ Advances in computational power have allowed both greater fidelity and more extensive use of such models. Numerous complex military systems have a corresponding model that simulates its performance in the field. In response, the DoD needs defensible practices for validating these models. Design of Experiments and statistical analysis techniques are the foundational building blocks for validating the use of computer models and quantifying uncertainty in that validation. Recent developments in uncertainty quantification have the potential to benefit the DoD in using modeling and simulation to inform operational evaluations. Advances in computational power have allowed both greater fidelity and more extensive use of such models. Numerous complex military systems have a corresponding model that simulates its performance in the field. In response, the DoD needs defensible practices for validating these models. Design of Experiments and statistical analysis techniques are the foundational building blocks for validating the use of computer models and quantifying uncertainty in that validation. Recent developments in uncertainty quantification have the potential to benefit the DoD in using modeling and simulation to inform operational evaluations.

Suggested Citation

Wojton, Heather, Kelly Avery, Laura Freeman, and Thomas Johnson. “Designing Experiments for Model Validation – The Foundations for Uncertainty Quantification.” The ITEA Journal of Test and Evaluation 40, no. 1 (2019).

Paper:

]]> Managing T&E Data to Encourage Reuse https://research.testscience.org/post/2019-managing-t-e-data-to-encourage-reuse/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-managing-t-e-data-to-encourage-reuse/ Reusing Test and Evaluation (T&E) datasets multiple times at different points throughout a program’s lifecycle is one way to realize their full value. Data management plays an important role in enabling - and even encouraging – this practice. Although Department-level policy on data management is supportive of reuse and consistent with best practices from industry and academia, the documents that shape the day-to-day activities of T&E practitioners are much less so. Reusing Test and Evaluation (T&E) datasets multiple times at different points throughout a program’s lifecycle is one way to realize their full value. Data management plays an important role in enabling - and even encouraging – this practice. Although Department-level policy on data management is supportive of reuse and consistent with best practices from industry and academia, the documents that shape the day-to-day activities of T&E practitioners are much less so. As a result, reuse of T&E datasets does not occur on a consistent basis or in a formalized way. To fill this apparent gap, this article expands upon four best practices – addressed in different ways in Service-specific T&E policies – that can increase the reuse of T&E datasets.

Suggested Citation

Medlin, Rebecca, and Andrew Flack. “Managing T&E Data to Encourage Reuse.” The ITEA Journal of Test and Evaluation of Test and Evaluation, 2019.

Paper:

]]> A Groundswell for Test and Evaluation https://research.testscience.org/post/2018-a-groundswell-for-test-and-evaluation/ Mon, 01 Jan 2018 00:00:00 +0000 https://research.testscience.org/post/2018-a-groundswell-for-test-and-evaluation/ The fundamental purpose of test and evaluation (T&E) in the Department of Defense (DOD) is to provide knowledge to answer critical questions that help decision makers manage the risk involved in developing, producing, operating, and sustaining systems and capabilities. At its core, T&E takes data and translates it into information for decision makers. Subject matter expertise of the platform and operational mission have always been critical components of developing defensible test and evaluation strategies. The fundamental purpose of test and evaluation (T&E) in the Department of Defense (DOD) is to provide knowledge to answer critical questions that help decision makers manage the risk involved in developing, producing, operating, and sustaining systems and capabilities. At its core, T&E takes data and translates it into information for decision makers. Subject matter expertise of the platform and operational mission have always been critical components of developing defensible test and evaluation strategies. Recent innovations in data science have improved our ability to collect, store, manage, transfer, process and visualize data. Additionally, advances in statistics and uncertainty quantification are revolutionizing how we think about predictions from all types of data. The ability to integrate system and scientific knowledge, coupled with advances in data science and statistics, will enable us to better target testing, make efficient use of resources, quantify risk, and lead to well informed decisions.

Suggested Citation

Freeman, Laura J. “A Groundswell for Test and Evaluation.” The ITEA Journal 39, no. 4 (December 2018).

Paper:

]]> Rigorous Test and Evaluation for Defense, Aerospace, and National Security https://research.testscience.org/post/2016-rigorous-test-and-evaluation-for-defense-aerospace-and-national-security/ Fri, 01 Jan 2016 00:00:00 +0000 https://research.testscience.org/post/2016-rigorous-test-and-evaluation-for-defense-aerospace-and-national-security/ In April 2016, NASA, DOT&E, and IDA collaborated on a workshopdesigned to strengthen the community around statistical approaches to test andevaluation in defense and aerospace. The workshop brought practitioners, analysts,technical leadership, and statistical academics together for a three day exchange ofinformation with opportunities to attend world renowned short courses, share commodchallenges, and learn new skill sets from a variety of tutorials. A highlight of theworkshop was the Tuesday afternoon technical leadership panel chaired by Dr. In April 2016, NASA, DOT&E, and IDA collaborated on a workshopdesigned to strengthen the community around statistical approaches to test andevaluation in defense and aerospace. The workshop brought practitioners, analysts,technical leadership, and statistical academics together for a three day exchange ofinformation with opportunities to attend world renowned short courses, share commodchallenges, and learn new skill sets from a variety of tutorials. A highlight of theworkshop was the Tuesday afternoon technical leadership panel chaired by Dr.Catherine Warner, Science Advisor, DOT&E. This article summarizes core themesdiscuss during the panel session.

Suggested Citation

Freeman, Laura. “Rigorous Test and Evaluation for Defense Aerospace, and National Security: A Panel Session Summary.” The ITEA Journal of Test and Evaluation 37, no. 4 (2016).

Paper:

]]> Improving Reliability Estimates with Bayesian Statistics https://research.testscience.org/post/2015-improving-reliability-estimates-with-bayesian-statistics/ Thu, 01 Jan 2015 00:00:00 +0000 https://research.testscience.org/post/2015-improving-reliability-estimates-with-bayesian-statistics/ This paper shows how Bayesian methods are ideal for the assessment of complex system reliability assessments. Several examples illustrate the methodology. Suggested Citation Freeman, Laura J, and Kassandra Fronczyk. “Improving Reliability Estimates with Bayesian Statistics.” ITEA Journal of Test and Evaluation 37, no. 4 (June 2015). Paper: This paper shows how Bayesian methods are ideal for the assessment of complex system reliability assessments. Several examples illustrate the methodology.

Suggested Citation

Freeman, Laura J, and Kassandra Fronczyk. “Improving Reliability Estimates with Bayesian Statistics.” ITEA Journal of Test and Evaluation 37, no. 4 (June 2015).

Paper:

]]> Validating the PRA Testbed Using a Statistically Rigorous Approach https://research.testscience.org/post/2015-validating-the-pra-testbed-using-a-statistically-rigorous-approach/ Thu, 01 Jan 2015 00:00:00 +0000 https://research.testscience.org/post/2015-validating-the-pra-testbed-using-a-statistically-rigorous-approach/ For many systems, testing is expensive and only a few live test events are conducted. When this occurs, testers frequently use a model to extend the test results. However, testers must validate the model to show that it is an accurate representation of the real world from the perspective of the intended uses of the model. This raises a problem when only a small number of live test events are conducted, only limited data are available to validate the model, and some testers struggle with model validation. For many systems, testing is expensive and only a few live test events are conducted. When this occurs, testers frequently use a model to extend the test results. However, testers must validate the model to show that it is an accurate representation of the real world from the perspective of the intended uses of the model. This raises a problem when only a small number of live test events are conducted, only limited data are available to validate the model, and some testers struggle with model validation. This article describes a statistically rigorous approach for validating a model with only a small number of live test results. We discuss a specific application for validating a model of a naval surface combatant defending itself against a cruise missile attack. The approach takes into account potential correlation in the data and other factors that may drive system performance.

Suggested Citation

Thomas, Dean, and Rebecca Dickinson. “Validating the Probability of Raid Annihilation Testbed Using a Statistical Approach.” The ITEA Journal of Test and Evaluation 36, no. 2 (June 2015).

Paper:

]]> Taking the Next Step- Improving the Science of Test in DoD T&E https://research.testscience.org/post/2014-taking-the-next-step-improving-the-science-of-test-in-dod-t-e/ Wed, 01 Jan 2014 00:00:00 +0000 https://research.testscience.org/post/2014-taking-the-next-step-improving-the-science-of-test-in-dod-t-e/ The current fiscal climate demands now, more than ever, that test and evaluation(T&E) provide relevant and credible characterization of system capabilities andshortfalls across all relevant operational conditions as efficiently as possible. Indetermining the answer to the question, “How much testing is enough?” it isimperative that we use a scientifically defensible methodology. Design ofExperiments (DOE) has a proven track record in Operational Test andEvaluation (OT&E) of not only quantifying how much testing is enough, but alsowhere in the operational space the test points should be placed. The current fiscal climate demands now, more than ever, that test and evaluation(T&E) provide relevant and credible characterization of system capabilities andshortfalls across all relevant operational conditions as efficiently as possible. Indetermining the answer to the question, “How much testing is enough?” it isimperative that we use a scientifically defensible methodology. Design ofExperiments (DOE) has a proven track record in Operational Test andEvaluation (OT&E) of not only quantifying how much testing is enough, but alsowhere in the operational space the test points should be placed. Over the last fewyears, the T&E community has made great strides in the application of DOE toOT&E, but there is still work to be done in ensuring that the scientificcommunity’s full toolset is utilized. In particular, many test programs have yet tocapitalize on the power of the test design when conducting the data analysis.Employing empirical statistical models (e.g., regression techniques, analysis ofvariance (ANOVA)) allows us to maximize the information from every data point,resulting in defensible analyses that provide crucial information about systemperformance that decision-makers and warfighters need to know. DOT&E willcontinue to work to ensure the highest technical caliber in every DOT&Eevaluation, and that Test and Evaluation Master Plans (TEMPs) are adequate tosupport these robust evaluations. As we improve in our use of these test designsand analysis methods, we need to ensure these practices are institutionalizedacross the entire T&E community and applied across all phases of DoD testing

Suggested Citation

Freeman, Laura, and V. Bram Lillard. “Taking the Next Step: Improving the Science of Test in DoD T and E.” The ITEA Journal of Test and Evaluation 35, no. 1 (March 2014). https://apps.dtic.mil/sti/citations/trecms/AD1123777.

Slides:

]]> Use of Statistically Designed Experiments to Inform Decisions in a Resource Constrained Environment https://research.testscience.org/post/2011-use-of-statistically-designed-experiments-to-inform-decisions-in-a-resource-constrained-environment/ Sat, 01 Jan 2011 00:00:00 +0000 https://research.testscience.org/post/2011-use-of-statistically-designed-experiments-to-inform-decisions-in-a-resource-constrained-environment/ There has been recent emphasis on the increased use of statistics, including the use of statistically designed experiments, to plan and execute tests that support Department of Defense (DoD) acquisition programs. The use of statistical methods, including experimental design, has shown great benefits in industry, especially when used in an integrated fashion; for example see the literature on Six Sigma. The structured approach of experimental design allows the user to determine what data need to be collected and how it should be analyzed to achieve specific decision making objectives. There has been recent emphasis on the increased use of statistics, including the use of statistically designed experiments, to plan and execute tests that support Department of Defense (DoD) acquisition programs. The use of statistical methods, including experimental design, has shown great benefits in industry, especially when used in an integrated fashion; for example see the literature on Six Sigma. The structured approach of experimental design allows the user to determine what data need to be collected and how it should be analyzed to achieve specific decision making objectives. This focuses decision making processes, improves test efficiency and provides objective data for evidence-based decision-making. Today the DoD Test and Evaluation (T&E) community is investigating the use of statistical methods to provide efficient and effective testing. This paper discusses the use of statistics in T&E to assist T&E practitioners and acquisition management in understanding how to improve the quantity and quality of information made available to decision makers to make risk assessments, even in a resource constrained environment.

Suggested Citation

Freeman, Laura, Karl Glaeser, and Alethea Rucker. “Use of Statistically Design Experiments to Inform Decisions in a Resource Constrained Environment.” ITEA Journal of Test and Evaluation. 32, no. 3 (2011): 267–76.

Paper:

]]>