Operational Test and Evaluation on Test Science Research Document Library https://research.testscience.org/keywords/operational-test-and-evaluation/ Recent content in Operational Test and Evaluation on Test Science Research Document Library Hugo -- 0.129.0 en-us Institute for Defense Analyses Mon, 01 Jan 2024 00:00:00 +0000 A Practitioner’s Framework for Federated Model Validation Resource Allocation https://research.testscience.org/post/2024-a-practitioner-s-framework-for-federated-model-validation-resource-allocation/ Mon, 01 Jan 2024 00:00:00 +0000 https://research.testscience.org/post/2024-a-practitioner-s-framework-for-federated-model-validation-resource-allocation/ Recent advances in computation and statistics led to an increasing use of federated models for end-to-end system test and evaluation. A federated model is a collection of interconnected models where the outputs of a model act as inputs to subsequent models. However, the process of verifying and validating federated models is poorly understood, especially when testers have limited resources, knowledge-based uncertainties, and concerns over operational realism. Testers often struggle with determining how to best allocate limited test resources for model validation.

Recent advances in computation and statistics led to an increasing use of federated models for end-to-end system test and evaluation. A federated model is a collection of interconnected models where the outputs of a model act as inputs to subsequent models. However, the process of verifying and validating federated models is poorly understood, especially when testers have limited resources, knowledge-based uncertainties, and concerns over operational realism. Testers often struggle with determining how to best allocate limited test resources for model validation. We propose a network-based representation of federated models, where the network encodes the connections between the federation of models. Nodes of the graph are given by sub-models. A directed edge from node a to node b is drawn if a inputs into b. We quantify their uncertainties through edge weights using meta-modeling and variance-based sensitivity analysis. The network-based framework allows us to propagate the uncertainties through the federated model and optimize resource allocation for validation based on the uncertainties.

Suggested Citation

Capp, Jo Anna, John T Haman, and Dhruv Patel. A Practitioner’s Framework for Federated Model Validation Resource Allocation. IDA Product ID 3001838. Alexandria, VA: Institute for Defense Analyses, 2024.

Slides:

]]> Simulation Insights on Power Analysis with Binary Responses--from SNR Methods to 'skprJMP' https://research.testscience.org/post/2024-simulation-insights-on-power-analysis-with-binary-responses-from-snr-methods-to-skprjmp/ Mon, 01 Jan 2024 00:00:00 +0000 https://research.testscience.org/post/2024-simulation-insights-on-power-analysis-with-binary-responses-from-snr-methods-to-skprjmp/ Logistic regression is a commonly-used method for analyzing tests with probabilistic responses in the test community, yet calculating power for these tests has historically been challenging. This difficulty prompted the development of methods based on signal-to-noise ratio (SNR) approximations over the last decade, tailored to address the intricacies of logistic regression’s binary outcomes. However, advancements and improvements in statistical software and computational power have reduced the need for such approximate methods.

Logistic regression is a commonly-used method for analyzing tests with probabilistic responses in the test community, yet calculating power for these tests has historically been challenging. This difficulty prompted the development of methods based on signal-to-noise ratio (SNR) approximations over the last decade, tailored to address the intricacies of logistic regression’s binary outcomes. However, advancements and improvements in statistical software and computational power have reduced the need for such approximate methods. Our research presents a detailed simulation study that compares SNR-based power estimates with those derived from exact Monte Carlo simulations, highlighting the inadequacies of SNR approximations. To address these shortcomings, we will discuss improvements in the open-source R package “skpr” as well as present “skprJMP,” a new plug-in that offers more accurate and reliable power calculations for logistic regression analyses.

Suggested Citation

Atkins, Robert, Tyler Morgan-Wall, and Curtis Miller. “With Binary Responses–From SNR Methods to ‘skprJMP.’” Institute for Defense Analyses IDA Product ID 3002093 (April 2024).

Slides:

Paper:

Poster:

]]> Comparing Normal and Binary D-Optimal Designs by Statistical Power https://research.testscience.org/post/2023-comparing-normal-and-binary-d-optimal-designs-by-statistical-power/ Sun, 01 Jan 2023 00:00:00 +0000 https://research.testscience.org/post/2023-comparing-normal-and-binary-d-optimal-designs-by-statistical-power/ In many Department of Defense test and evaluation applications, binary response variables are unavoidable. Many have considered D-optimal design of experiments for generalized linear models. However, little consideration has been given to assessing how these new designs perform in terms of statistical power for a given hypothesis test. Monte Carlo simulations and exact power calculations suggest that D optimal designs generally yield higher power than binary D-optimal designs, despite using logistic regression in the analysis after data have been collected.

In many Department of Defense test and evaluation applications, binary response variables are unavoidable. Many have considered D-optimal design of experiments for generalized linear models. However, little consideration has been given to assessing how these new designs perform in terms of statistical power for a given hypothesis test. Monte Carlo simulations and exact power calculations suggest that D optimal designs generally yield higher power than binary D-optimal designs, despite using logistic regression in the analysis after data have been collected. Results from using statistical power to compare designs contradict standard design of experiments comparisons, which employ D-efficiency ratios and fractional design space plots. Power calculations suggest that practitioners that are primarily interested in the resulting statistical power of a design should use normal D optimal designs over binary D-optimal designs when logistic regression is to be used in the data analysis after data collection

Suggested Citation

Medlin, Rebecca M, and Addison D Adams. Comparing Normal and Binary D-Optimal Design of Experiments by Statistical Power. IDA Document 3000032. Alexandria, VA: Institute for Defense Analyses, 2023.

Slides:

]]> Framework for Operational Test Design- An Example Application of Design Thinking https://research.testscience.org/post/2023-framework-for-operational-test-design-an-example-application-of-design-thinking/ Sun, 01 Jan 2023 00:00:00 +0000 https://research.testscience.org/post/2023-framework-for-operational-test-design-an-example-application-of-design-thinking/ This poster provides an example of how a design thinking framework can facilitate operational test design. Design thinking is a problem-solving approach of interest to many groups including those in the test and evaluation community. Design thinking promotes the principles of human-centeredness, iteration, and diversity and it can be accomplished via a five-phased approach. Following this approach, designers create innovated product solutions by (l) conducting research to empathize with their users, (2) defining specific user problems, (3) ideating on solutions that address the defined problems, (4) prototyping the product, and (5) testing the prototype. This poster provides an example of how a design thinking framework can facilitate operational test design. Design thinking is a problem-solving approach of interest to many groups including those in the test and evaluation community. Design thinking promotes the principles of human-centeredness, iteration, and diversity and it can be accomplished via a five-phased approach. Following this approach, designers create innovated product solutions by (l) conducting research to empathize with their users, (2) defining specific user problems, (3) ideating on solutions that address the defined problems, (4) prototyping the product, and (5) testing the prototype.

Suggested Citation

Avery, Kelly M, and Miriam E Armstrong. An Example Application of Design Thinking. IDA Document NS D-33368. Alexandria, VA: Institute for Defense Analyses, 2023.

Slides:

]]> Introduction to Design of Experiments for Testers https://research.testscience.org/post/2023-introduction-to-design-of-experiments-for-testers/ Sun, 01 Jan 2023 00:00:00 +0000 https://research.testscience.org/post/2023-introduction-to-design-of-experiments-for-testers/ This training provides details regarding the use of design of experiments, from choosing proper response variables, to identifying factors that could affect such responses, to determining the amount of data necessary to collect. The training also explains the benefits of using a Design of Experiments approach to testing and provides an overview of commonly used designs (e.g., factorial, optimal, and space-filling). The briefing illustrates the concepts discussed using several case studies. This training provides details regarding the use of design of experiments, from choosing proper response variables, to identifying factors that could affect such responses, to determining the amount of data necessary to collect. The training also explains the benefits of using a Design of Experiments approach to testing and provides an overview of commonly used designs (e.g., factorial, optimal, and space-filling). The briefing illustrates the concepts discussed using several case studies.

Suggested Citation

Haman, John T, Breeana Anderson, Rebecca Medlin, Kelly M Avery, and Keyla Pagan-Rivera. I/ITSEC DOE Tutorial. IDA Document NS-D-33561. Alexandria, VA: Institute for Defense Analyses, 2023.

Slides:

]]> Statistical Methods Development Work for M&S Validation https://research.testscience.org/post/2023-statistical-methods-development-work-for-m-s-validation/ Sun, 01 Jan 2023 00:00:00 +0000 https://research.testscience.org/post/2023-statistical-methods-development-work-for-m-s-validation/ We discuss four areas in which statistically rigorous methods contribute to modeling and simulation validation studies. These areas are statistical risk analysis, space-filling experimental designs, metamodel construction, and statistical validation. Taken together, these areas implement DOT&E guidance on model validation. In each area, IDA has contributed either research methods, user-friendly tools, or both. We point to our tools on testscience.org, and survey the research methods that we’ve contributed to the M&S validation literature We discuss four areas in which statistically rigorous methods contribute to modeling and simulation validation studies. These areas are statistical risk analysis, space-filling experimental designs, metamodel construction, and statistical validation. Taken together, these areas implement DOT&E guidance on model validation. In each area, IDA has contributed either research methods, user-friendly tools, or both. We point to our tools on testscience.org, and survey the research methods that we’ve contributed to the M&S validation literature

Suggested Citation

Miller, Curtis G. “Statistical Methods Development Work for M&S Validation.” International Test and Evaluation Association 44, no. 3 (September 11, 2023). https://doi.org/10.61278/itea.44.3.1010.

Slides:

Paper:

]]> A Review of Sequential Analysis https://research.testscience.org/post/2020-a-review-of-sequential-analysis/ Wed, 01 Jan 2020 00:00:00 +0000 https://research.testscience.org/post/2020-a-review-of-sequential-analysis/ Sequential analysis concerns statistical evaluation in situations in which the number, pattern, or composition of the data is not determined at the start of the investigation, but instead depends upon the information acquired throughout the course of the investigation. Expanding the use of sequential analysis has the potential to save resources and reduce test time (National Research Council, 1998). This paper summarizes the literature on sequential analysis and offers fundamental information for providing recommendations for its use in DoD test and evaluation. Sequential analysis concerns statistical evaluation in situations in which the number, pattern, or composition of the data is not determined at the start of the investigation, but instead depends upon the information acquired throughout the course of the investigation. Expanding the use of sequential analysis has the potential to save resources and reduce test time (National Research Council, 1998). This paper summarizes the literature on sequential analysis and offers fundamental information for providing recommendations for its use in DoD test and evaluation.

Suggested Citation

Wojton, Heather, Rebecca Medlin, John Dennis, Keyla Pagan-Rivera, and Leonard Wilkins. A Review of Sequential Analysis. IDA Document NS D-20487. Alexandria, VA: Institute for Defense Analyses, 2020.

Paper:

]]> Sample Size Determination Methods Using Acceptance Sampling by Variables https://research.testscience.org/post/2019-sample-size-determination-methods-using-acceptance-sampling-by-variables/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-sample-size-determination-methods-using-acceptance-sampling-by-variables/ Acceptance Sampling by Variables (ASbV) is a statistical testing technique used in Personal Protective Equipment programs to determine the quality of the equipment in First Article and Lot Acceptance Tests. This article intends to remedy the lack of existing references that discuss the similarities between ASbV and certain techniques used in different sub-disciplines within statistics. Understanding ASbV from a statistical perspective allows testers to create customized test plans, beyond what is available in MIL-STD-414. Acceptance Sampling by Variables (ASbV) is a statistical testing technique used in Personal Protective Equipment programs to determine the quality of the equipment in First Article and Lot Acceptance Tests. This article intends to remedy the lack of existing references that discuss the similarities between ASbV and certain techniques used in different sub-disciplines within statistics. Understanding ASbV from a statistical perspective allows testers to create customized test plans, beyond what is available in MIL-STD-414.

Suggested Citation

Walzl, Kerry, Lindsey A Davis, Thomas H Johnson, and Heather M Wojton. Sample Size Determination Methods Using Acceptance Sampling by Variables. IDA Document NS D-10666. Alexandria, VA: Institute for Defense Analyses, 2019.

Paper:

]]> Comparing M&S Output to Live Test Data- A Missile System Case Study https://research.testscience.org/post/2018-comparing-m-s-output-to-live-test-data-a-missile-system-case-study/ Mon, 01 Jan 2018 00:00:00 +0000 https://research.testscience.org/post/2018-comparing-m-s-output-to-live-test-data-a-missile-system-case-study/ In the operational testing of DoD weapons systems, modeling and simulation (M&S) is often used to supplement live test data in order to support a more complete and rigorous evaluation. Before the output of the M&S is included in reports to decision makers, it must first be thoroughly verified and validated to show that it adequately represents the real world for the purposes of the intended use. Part of the validation process should include a statistical comparison of live data to M&S output. In the operational testing of DoD weapons systems, modeling and simulation (M&S) is often used to supplement live test data in order to support a more complete and rigorous evaluation. Before the output of the M&S is included in reports to decision makers, it must first be thoroughly verified and validated to show that it adequately represents the real world for the purposes of the intended use. Part of the validation process should include a statistical comparison of live data to M&S output. This presentation includes an example of one such validation analysis for a tactical missile system. In this case, the goal is to validate a lethality model that predicts the likelihood of destroying a particular enemy target. Using design of experiments, along with basic analysis techniques such as the Kolmogorov-Smirnov test and Poisson regression, we can explore differences between the M&S and live data across multiple operational conditions and quantify the associated uncertainties.

Suggested Citation

Thomas, Dean, and Kelly M Avery. Comparing M&S Output to Live Test Data: A Missile System Case Study. IDA Non-Standard Document NS D-9002. Alexandria, VA: Institute for Defense Analyses, 2018.

Slides:

]]> JEDIS Briefing and Tutorial https://research.testscience.org/post/2018-jedis-briefing-and-tutorial/ Mon, 01 Jan 2018 00:00:00 +0000 https://research.testscience.org/post/2018-jedis-briefing-and-tutorial/ Are you sick of having to manually iterate your way through sizing your design of experiments? Come learn about JEDIS, the new IDA-developed JMP Add-In for automating design of experiments power calculations. JEDIS builds multiple test designs in JMP over user-specified ranges of sample sizes, Signal-to-Noise Ratios (SNR), and alpha (1 -confidence) levels. It then automatically calculates the statistical power to detect an effect due to each factor and any specified interactions for each design. Are you sick of having to manually iterate your way through sizing your design of experiments? Come learn about JEDIS, the new IDA-developed JMP Add-In for automating design of experiments power calculations. JEDIS builds multiple test designs in JMP over user-specified ranges of sample sizes, Signal-to-Noise Ratios (SNR), and alpha (1 -confidence) levels. It then automatically calculates the statistical power to detect an effect due to each factor and any specified interactions for each design. When finished, JEDIS presents the statistical power vs. design metrics in interactive plots and stores the data in an easy to use format. JEDIS creates factorial and optimal designs, but does not currently support split plot designs. If you already have a pre-made design table, the JEDIS Light feature can compute power for the design over ranges of SNR and alpha levels.

Suggested Citation

Pechkis, Daniel, and Jason P Sheldon. JEDIS Briefing and Tutorial. IDA Document NS D-8964. Alexandria, VA: Institute for Defense Analyses, 2018.

Paper:

]]> Testing Defense Systems https://research.testscience.org/post/2018-testing-defense-systems/ Mon, 01 Jan 2018 00:00:00 +0000 https://research.testscience.org/post/2018-testing-defense-systems/ The complex, multifunctional nature of defense systems, along with the wide variety of system types, demands a structured but flexible analytical process for testing systems. This chapter summarizes commonly used techniques in defense system testing and specific challenges imposed by the nature of defense system testing. It highlights the core statistical methodologies that have proven useful in testing defense systems. Case studies illustrate the value of using statistical techniques in the design of tests and analysis of the resulting data. The complex, multifunctional nature of defense systems, along with the wide variety of system types, demands a structured but flexible analytical process for testing systems. This chapter summarizes commonly used techniques in defense system testing and specific challenges imposed by the nature of defense system testing. It highlights the core statistical methodologies that have proven useful in testing defense systems. Case studies illustrate the value of using statistical techniques in the design of tests and analysis of the resulting data. The chapter focuses on the unique statistical challenges of designing operational tests, many of which can be attributed to the process, but some of which are inherent to the complexity of the systems and the missions system operators must complete. It provides an overview of the process of designing experiments for military systems with operational users in an operational environment.

Suggested Citation

Freeman, Laura J., Thomas Johnson, Matthew Avery, V. Bram Lillard, and Justace Clutter. “Testing Defense Systems.” In Analytic Methods in Systems and Software Testing, 439–87. John Wiley & Sons, Ltd, 2018. https://doi.org/10.1002/9781119357056.ch18.

Paper:

]]> Statistical Methods for Defense Testing https://research.testscience.org/post/2017-statistical-methods-for-defense-testing/ Sun, 01 Jan 2017 00:00:00 +0000 https://research.testscience.org/post/2017-statistical-methods-for-defense-testing/ In the increasingly complex and data‐limited world of military defense testing, statisticians play a valuable role in many applications. Before the DoD acquires any major new capability, that system must undergo realistic testing in its intended environment with military users. Although the typical test environment is highly variable and factors are often uncontrolled, design of experiments techniques can add objectivity, efficiency, and rigor to the process of test planning. Statistical analyses help system evaluators get the most information out of limited data sets. In the increasingly complex and data‐limited world of military defense testing, statisticians play a valuable role in many applications. Before the DoD acquires any major new capability, that system must undergo realistic testing in its intended environment with military users. Although the typical test environment is highly variable and factors are often uncontrolled, design of experiments techniques can add objectivity, efficiency, and rigor to the process of test planning. Statistical analyses help system evaluators get the most information out of limited data sets. Oftentimes new or complex analysis techniques are needed to support the goal of characterizing or predicting system performance across the operational space. Finally, the growing need for computer models or simulations to supplement live testing also means that these models must be appropriately validated before their output can be deemed sufficient for use. Statistical design and analysis techniques are essential for rigorous evaluation of these models.

Suggested Citation

Avery, Matthew R., Kelly M. Avery, and Laura J. Freeman. “Statistical Methods for Defense Testing.” In Wiley StatsRef: Statistics Reference Online, edited by Ron S. Kenett, Nicholas T. Longford, Walter W. Piegorsch, and Fabrizio Ruggeri, 1st ed., 1–5. Wiley, 2018. https://doi.org/10.1002/9781118445112.stat07946.

Paper:

]]> Tutorial on Sensitivity Testing in Live Fire Test and Evaluation https://research.testscience.org/post/2016-tutorial-on-sensitivity-testing-in-live-fire-test-and-evaluation/ Fri, 01 Jan 2016 00:00:00 +0000 https://research.testscience.org/post/2016-tutorial-on-sensitivity-testing-in-live-fire-test-and-evaluation/ A sensitivity experiment is a special type of experimental design that is used when the response variable is binary and the covariate is continuous. Armor protection and projectile lethality tests often use sensitivity experiments to characterize a projectile’s probability of penetrating the armor. In this mini-tutorial we illustrate the challenge of modeling a binary response with a limited sample size, and show how sensitivity experiments can mitigate this problem. We review eight different single covariate sensitivity experiments and present a comparison of these designs using simulation. A sensitivity experiment is a special type of experimental design that is used when the response variable is binary and the covariate is continuous. Armor protection and projectile lethality tests often use sensitivity experiments to characterize a projectile’s probability of penetrating the armor. In this mini-tutorial we illustrate the challenge of modeling a binary response with a limited sample size, and show how sensitivity experiments can mitigate this problem. We review eight different single covariate sensitivity experiments and present a comparison of these designs using simulation. Additionally, we cover sensitivity experiments for cases that include more than one covariate, and highlight recent research in this area.

Suggested Citation

Johnson, Thomas, Laura Freeman, and Raymond Chen. Tutorial on Sensitivity Testing in Live Fire Test and Evaluation. IDA Document NS D-5829. Alexandria, VA: Institute for Defense Analyses, 2016.

Slides:

]]> Surveys in Operational Test and Evaluation https://research.testscience.org/post/2015-surveys-in-operational-test-and-evaluation/ Thu, 01 Jan 2015 00:00:00 +0000 https://research.testscience.org/post/2015-surveys-in-operational-test-and-evaluation/ Recently DOT&E signed out a memo providing Guidance on the Use and Design of Surveys in Operational Test and Evaluation. This guidance memo helps the Human Systems Integration (HSI) community to ensure that useful and accurate HSI data are collected. Information about how HSI experts can leverage the guidance is presented. Specifically, the presentation will cover which HSI metrics can and cannot be answered by surveys. Suggested Citation Grier, Rebecca A, and Laura Freeman. Recently DOT&E signed out a memo providing Guidance on the Use and Design of Surveys in Operational Test and Evaluation. This guidance memo helps the Human Systems Integration (HSI) community to ensure that useful and accurate HSI data are collected. Information about how HSI experts can leverage the guidance is presented. Specifically, the presentation will cover which HSI metrics can and cannot be answered by surveys.

Suggested Citation

Grier, Rebecca A, and Laura Freeman. Surveys in Operational Test & Evaluation. IDA Document D-5410. Alexandria, VA: Institute for Defense Analyses, 2015.

Slides:

]]>