Rebecca Medlin on Test Science Research Document Library https://research.testscience.org/researchers/rebecca-medlin/ Recent content in Rebecca Medlin on Test Science Research Document Library Hugo -- 0.129.0 en-us Institute for Defense Analyses Mon, 01 Jan 2024 00:00:00 +0000 A Reliability Assurance Test Planning and Analysis Tool https://research.testscience.org/post/2024-a-reliability-assurance-test-planning-and-analysis-tool/ Mon, 01 Jan 2024 00:00:00 +0000 https://research.testscience.org/post/2024-a-reliability-assurance-test-planning-and-analysis-tool/ This presentation documents the work of IDA 2024 Summer Associate Emma Mitchell. The work presented details an R Shiny application developed to provide a user-friendly software tool for researchers to use in planning for and analyzing system reliability. Specifically, the presentation details how one can plan for a reliability test using Bayesian Reliability Assurance test methods. Such tests utilize supplementary data and information, including reliability models, prior test results, expert judgment, and knowledge of environmental conditions, to plan for reliability testing, which in turn can often help in reducing the required amount of testing. This presentation documents the work of IDA 2024 Summer Associate Emma Mitchell. The work presented details an R Shiny application developed to provide a user-friendly software tool for researchers to use in planning for and analyzing system reliability. Specifically, the presentation details how one can plan for a reliability test using Bayesian Reliability Assurance test methods. Such tests utilize supplementary data and information, including reliability models, prior test results, expert judgment, and knowledge of environmental conditions, to plan for reliability testing, which in turn can often help in reducing the required amount of testing. In the planning phase, the application enables researchers to use Bayesian methods to incorporate supplementary data when determining appropriate test lengths. In the analysis phase, the tool allows researchers to combine information through Bayesian methods, resulting in better uncertainty quantification than traditional methods.

Suggested Citation

Haman, John T, Rebecca M Medlin, Emma P Mitchell, Keyla Pagán-Rivera, and Dhruv K Patel. A Reliability Assurance Test Planning and Analysis Tool. IDA Product ID 3003359. Institute for Defense Analyses, 2024.

Slides:

]]> Improving Test Efficiency- A Bayesian Assurance Case Study https://research.testscience.org/post/2023-improving-test-efficiency-a-bayesian-assurance-case-study/ Sun, 01 Jan 2023 00:00:00 +0000 https://research.testscience.org/post/2023-improving-test-efficiency-a-bayesian-assurance-case-study/ To improve test planning for evaluating system reliability, we propose the use of Bayesian methods to incorporate supplementary data and reduce testing duration. Furthermore, we recommend Bayesian methods be employed in the analysis phase to better quantify uncertainty. We find that when using Bayesian Methods for test planning we can scope smaller tests and using Bayesian methods in analysis results in a more precise estimate of reliability – improving uncertainty quantification. To improve test planning for evaluating system reliability, we propose the use of Bayesian methods to incorporate supplementary data and reduce testing duration. Furthermore, we recommend Bayesian methods be employed in the analysis phase to better quantify uncertainty. We find that when using Bayesian Methods for test planning we can scope smaller tests and using Bayesian methods in analysis results in a more precise estimate of reliability – improving uncertainty quantification.

Suggested Citation

Medlin, Rebecca M. A Bayesian Assurance Case Study. IDA Document NS 3000024. Alexandria, VA: Institute for Defense Analyses, 2023.

Slides:

]]> Introduction to Design of Experiments for Testers https://research.testscience.org/post/2023-introduction-to-design-of-experiments-for-testers/ Sun, 01 Jan 2023 00:00:00 +0000 https://research.testscience.org/post/2023-introduction-to-design-of-experiments-for-testers/ This training provides details regarding the use of design of experiments, from choosing proper response variables, to identifying factors that could affect such responses, to determining the amount of data necessary to collect. The training also explains the benefits of using a Design of Experiments approach to testing and provides an overview of commonly used designs (e.g., factorial, optimal, and space-filling). The briefing illustrates the concepts discussed using several case studies. This training provides details regarding the use of design of experiments, from choosing proper response variables, to identifying factors that could affect such responses, to determining the amount of data necessary to collect. The training also explains the benefits of using a Design of Experiments approach to testing and provides an overview of commonly used designs (e.g., factorial, optimal, and space-filling). The briefing illustrates the concepts discussed using several case studies.

Suggested Citation

Haman, John T, Breeana Anderson, Rebecca Medlin, Kelly M Avery, and Keyla Pagan-Rivera. I/ITSEC DOE Tutorial. IDA Document NS-D-33561. Alexandria, VA: Institute for Defense Analyses, 2023.

Slides:

]]> Case Study on Applying Sequential Analyses in Operational Testing https://research.testscience.org/post/2022-case-study-on-applying-sequential-analyses-in-operational-testing/ Sat, 01 Jan 2022 00:00:00 +0000 https://research.testscience.org/post/2022-case-study-on-applying-sequential-analyses-in-operational-testing/ Sequential analysis concerns statistical evaluation in which the number, pattern, or composition of the data is not determined at the start of the investigation, but instead depends on the information acquired during the investigation. Although sequential analysis originated in ballistics testing for the Department of Defense (DoD)and it is widely used in other disciplines, it is underutilized in the DoD. Expanding the use of sequential analysis may save money and reduce test time.

Sequential analysis concerns statistical evaluation in which the number, pattern, or composition of the data is not determined at the start of the investigation, but instead depends on the information acquired during the investigation. Although sequential analysis originated in ballistics testing for the Department of Defense (DoD)and it is widely used in other disciplines, it is underutilized in the DoD. Expanding the use of sequential analysis may save money and reduce test time. In this paper, we introduce sequential analysis, describe its current and potential uses in operational test and evaluation (OT&E), and present a method for applying it to the test and evaluation of defense systems. We evaluate the proposed method by performing simulation studies and applying the method to a case study. Additionally, we discuss challenges to address for sequential analysis in OT&E. Lastly, while operational testing is the focus in this paper, the methodology presented is applicable to campaigns of experimentation and general testing across numerous disciplines.

Suggested Citation

Ahrens, Monica, Rebecca Medlin, Keyla Pagán-Rivera, and John W. Dennis. “Case Study on Applying Sequential Analyses in Operational Testing.” Quality Engineering 35, no. 3 (July 3, 2023): 534–45. https://doi.org/10.1080/08982112.2022.2146510.

Slides:

Paper:

]]> Thoughts on Applying Design of Experiments (DOE) to Cyber Testing https://research.testscience.org/post/2022-thoughts-on-applying-design-of-experiments-doe-to-cyber-testing/ Sat, 01 Jan 2022 00:00:00 +0000 https://research.testscience.org/post/2022-thoughts-on-applying-design-of-experiments-doe-to-cyber-testing/ This briefing presented at Dataworks 2022 provides examples of potential ways in which Design of Experiments (DOE) could be applied to initially scope cyber assessments and, based on the results of those assessments, subsequently design in greater detail cyber tests. Suggested Citation Gilmore, James M, Kelly M Avery, Matthew R Girardi, and Rebecca M Medlin. Thoughts on Applying Design of Experiments (DOE) to Cyber Testing. IDA Document NS D-33023. Alexandria, VA: Institute for Defense Analyses, 2022. This briefing presented at Dataworks 2022 provides examples of potential ways in which Design of Experiments (DOE) could be applied to initially scope cyber assessments and, based on the results of those assessments, subsequently design in greater detail cyber tests.

Suggested Citation

Gilmore, James M, Kelly M Avery, Matthew R Girardi, and Rebecca M Medlin. Thoughts on Applying Design of Experiments (DOE) to Cyber Testing. IDA Document NS D-33023. Alexandria, VA: Institute for Defense Analyses, 2022.

Slides:

]]> Determining How Much Testing is Enough- An Exploration of Progress in the Department of Defense Test and Evaluation Community https://research.testscience.org/post/2021-determining-how-much-testing-is-enough-an-exploration-of-progress-in-the-department-of-defense-test-and-evaluation-community/ Fri, 01 Jan 2021 00:00:00 +0000 https://research.testscience.org/post/2021-determining-how-much-testing-is-enough-an-exploration-of-progress-in-the-department-of-defense-test-and-evaluation-community/ This paper describes holistic progress in answering the question of “How much testing is enough?” It covers areas in which the T&E community has made progress, areas in which progress remains elusive, and issues that have emerged since 1994 that provide additional challenges. The selected case studies used to highlight progress are especially interesting examples, rather than a comprehensive look at all programs since 1994. Suggested Citation Medlin, Rebecca, Matthew R Avery, James R Simpson, and Heather M Wojton. This paper describes holistic progress in answering the question of “How much testing is enough?” It covers areas in which the T&E community has made progress, areas in which progress remains elusive, and issues that have emerged since 1994 that provide additional challenges. The selected case studies used to highlight progress are especially interesting examples, rather than a comprehensive look at all programs since 1994.

Suggested Citation

Medlin, Rebecca, Matthew R Avery, James R Simpson, and Heather M Wojton. Determining How Much Testing Is Enough: An Exploration of Progress in the Department of Defense Test and Evaluation Community. IDA Document NS D-21561. Alexandria, VA: Institute for Defense Analyses, 2021.

Paper:

]]> Introduction to Bayesian Analysis https://research.testscience.org/post/2021-introduction-to-bayesian-analysis/ Fri, 01 Jan 2021 00:00:00 +0000 https://research.testscience.org/post/2021-introduction-to-bayesian-analysis/ As operational testing becomes increasingly integrated and research questions become more difficult to answer, IDA’s Test Science team has found Bayesian models to be powerful data analysis methods. Analysts and decision-makers should understand the differences between this approach and the conventional way of analyzing data. It is also important to recognize when an analysis could benefit from the inclusion of prior information—what we already know about a system’s performance—and to understand the proper way to incorporate that information. As operational testing becomes increasingly integrated and research questions become more difficult to answer, IDA’s Test Science team has found Bayesian models to be powerful data analysis methods. Analysts and decision-makers should understand the differences between this approach and the conventional way of analyzing data. It is also important to recognize when an analysis could benefit from the inclusion of prior information—what we already know about a system’s performance—and to understand the proper way to incorporate that information. To apply Bayesian methods, analysts need to comprehend some technical aspects of this approach and know how to properly use appropriate statistical software. In this course, students learn the intuition behind Bayesian statistics, the mathematical details of posterior distributions, how to fit simple Bayesian models using computer software, and how to assess model fit.

Suggested Citation

Wojton, Heather M, Keyla Pagan-Rivera, John T Haman, and Rebecca M Medlin. Introduction to Bayesian Analysis. IDA Document NS D-20484. Alexandria, VA: Institute for Defense Analyses, 2021.

Slides:

]]> Why are Statistical Engineers Needed for Test & Evaluation? https://research.testscience.org/post/2021-why-are-statistical-engineers-needed-for-test-evaluation/ Fri, 01 Jan 2021 00:00:00 +0000 https://research.testscience.org/post/2021-why-are-statistical-engineers-needed-for-test-evaluation/ The Department of Defense (DoD) develops and acquires some of the world’s most advanced and sophisticated systems. As new technologies emerge and are incorporated into systems, OSD/DOT&E faces the challenge of ensuring that these systems undergo adequate and efficient test and evaluation (T&E) prior to operational use. Statistical engineering is a collaborative, analytical approach to problem solving that integrates statistical thinking, methods, and tools with other relevant disciplines. The statistical engineering process provides better solutions to large, unstructured, real-world problems and supports rigorous decision-making. The Department of Defense (DoD) develops and acquires some of the world’s most advanced and sophisticated systems. As new technologies emerge and are incorporated into systems, OSD/DOT&E faces the challenge of ensuring that these systems undergo adequate and efficient test and evaluation (T&E) prior to operational use. Statistical engineering is a collaborative, analytical approach to problem solving that integrates statistical thinking, methods, and tools with other relevant disciplines. The statistical engineering process provides better solutions to large, unstructured, real-world problems and supports rigorous decision-making. In this talk, we provide two case study examples related to looking at ways to improve approaches to integrate testing and data collection across the full system lifecycle. These case studies highlight why we believe statistical engineers are necessary for successful T&E.

Suggested Citation

Medlin, Rebecca, Kayla Pagan-Rivera, and Monica Ahrens. Why Are Statistical Engineers Needed for Test & Evaluation? IDA Document NS-D-22722. Alexandria, VA: Institute for Defense Analyses, 2021.

Slides:

]]> A Review of Sequential Analysis https://research.testscience.org/post/2020-a-review-of-sequential-analysis/ Wed, 01 Jan 2020 00:00:00 +0000 https://research.testscience.org/post/2020-a-review-of-sequential-analysis/ Sequential analysis concerns statistical evaluation in situations in which the number, pattern, or composition of the data is not determined at the start of the investigation, but instead depends upon the information acquired throughout the course of the investigation. Expanding the use of sequential analysis has the potential to save resources and reduce test time (National Research Council, 1998). This paper summarizes the literature on sequential analysis and offers fundamental information for providing recommendations for its use in DoD test and evaluation. Sequential analysis concerns statistical evaluation in situations in which the number, pattern, or composition of the data is not determined at the start of the investigation, but instead depends upon the information acquired throughout the course of the investigation. Expanding the use of sequential analysis has the potential to save resources and reduce test time (National Research Council, 1998). This paper summarizes the literature on sequential analysis and offers fundamental information for providing recommendations for its use in DoD test and evaluation.

Suggested Citation

Wojton, Heather, Rebecca Medlin, John Dennis, Keyla Pagan-Rivera, and Leonard Wilkins. A Review of Sequential Analysis. IDA Document NS D-20487. Alexandria, VA: Institute for Defense Analyses, 2020.

Paper:

]]> Trustworthy Autonomy- A Roadmap to Assurance -- Part 1- System Effectiveness https://research.testscience.org/post/2020-trustworthy-autonomy-a-roadmap-to-assurance-part-1-system-effectiveness/ Wed, 01 Jan 2020 00:00:00 +0000 https://research.testscience.org/post/2020-trustworthy-autonomy-a-roadmap-to-assurance-part-1-system-effectiveness/ The Department of Defense (DoD) has invested significant effort over the past decade considering the role of artificial intelligence and autonomy in national security (e.g., Defense Science Board, 2012, 2016, Deputy Secretary of Defense, 2012, Endsley, 2015, Executive Order No. 13859, 2019, US Department of Defense, 2011, 2019, Zacharias, 2019a). However, these efforts were broadly scoped and only partially touched on how the DoD will certify the safety and performance of these systems. The Department of Defense (DoD) has invested significant effort over the past decade considering the role of artificial intelligence and autonomy in national security (e.g., Defense Science Board, 2012, 2016, Deputy Secretary of Defense, 2012, Endsley, 2015, Executive Order No. 13859, 2019, US Department of Defense, 2011, 2019, Zacharias, 2019a). However, these efforts were broadly scoped and only partially touched on how the DoD will certify the safety and performance of these systems. More recent work has done this big-picture thinking for the test and evaluation (T&E) community (e.g., Ahner & Parson, 2016, Haugh, Sparrow, & Tate, 2018, Porter et al., 2018, Sparrow, Tate, Biddle, Kaminski, & Madhavan, 2018, Zacharias, 2019b). In parallel, individual programs have been generating their own working-level solutions for their own particular use-cases and challenges.

The framework proposed in the current work bridges the gap between the big picture policy recommendations already made and individual program needs. It is meant to serve as a roadmap framework that the T&E community can follow in order to provide evidence that artificial intelligence (AI)-enabled and autonomous systems function as intended. At times we echo broad policy recommendations made by others as they will also enable T&E activities. In other places we make more specific recommendations relating to test planning and analysis. In this document, we present part one of our two-part roadmap. We discuss the challenges and possible solutions to assessing system effectiveness. A future part two will deal with test efficiency, simulation, and infrastructure. Due to the scope of this project, even the main body of this document only provides a survey of the challenges and our proposed solutions. However, this roadmap serves as an outline to a future series of technical papers covering these topics in detail for working-level testers and analysts

Suggested Citation

Porter, Daniel, Michael McAnally, Chad Bieber, Heather Wojton, and Rebecca Medlin. Trustworthy Autonomy: A Roadmap to Assurance Part I: System Effectiveness. IDA Document P-10768-NS. Alexandria, VA: Institute for Defense Analyses, 2020.

Slides:

Paper:

]]> Bayesian Component Reliability- An F-35 Case Study https://research.testscience.org/post/2019-bayesian-component-reliability-an-f-35-case-study/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-bayesian-component-reliability-an-f-35-case-study/ A challenging aspect ofa system reliability assessment is integratingmultiple sources of information, such as component, subsystem, and full-system data,along with previous test data or subject matter expert (SME) opinion. A powerfulfeature of Bayesian analyses is the ability to combine these multiple sources of dataand variability in an informed way to perform statistical inference. This feature isparticularly valuable in assessing system reliability where testing is limited and only asmall number of failures (or none at all) are observed. A challenging aspect ofa system reliability assessment is integratingmultiple sources of information, such as component, subsystem, and full-system data,along with previous test data or subject matter expert (SME) opinion. A powerfulfeature of Bayesian analyses is the ability to combine these multiple sources of dataand variability in an informed way to perform statistical inference. This feature isparticularly valuable in assessing system reliability where testing is limited and only asmall number of failures (or none at all) are observed.The F-35 is DoD’s largest program; approximately one-third of the operations andsustainment cost is attributed to the cost of spare parts and the removal, replacement,and repair of components. The failure rate of those components is the drivingparameter for a significant portion of the sustainment cost, and yet for many of thesecomponents, available estimates of the failure rate are poor. For many programs, thecontractor produces estimates of component failure rates based on engineering analysisand legacy systems with similar parts. While these estimates are useful, the actualremoval rates provide a more accurate estimate of the removal and replacement ratesthe program will experience in future years.In this document, we show how we applied a Bayesian analysis to combine theengineering reliability estimates with the actual failure data to estimate componentreliability. Our analysis technique also allows for us to overcome the problems of caseswhere few or no failures have been observed. We are able to show that combining theengineering knowledge of reliability with the observed operational reliability results inboth a more informed estimate of each individual component’s reliaiblity and a moreinformed estimate of overall F-35 maintenance costs.The technique presented is broadly applicable to any progam where multiple sourcesof reliability information need to be combined for the best estimation of componentfailure rates, and ultimately of sustainment costs.

Suggested Citation

Medlin, Rebecca M, and V. Bram Lillard. Bayesian Component Reliability Estimation: An F-35 Case Study. IDA Document NS D-10561. Alexandria, VA: Institute for Defense Analyses, 2019.

Slides:

]]> Challenges and New Methods for Designing Reliability Experiments https://research.testscience.org/post/2019-challenges-and-new-methods-for-designing-reliability-experiments/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-challenges-and-new-methods-for-designing-reliability-experiments/ Engineers use reliability experiments to determine the factors that drive product reliability, build robust products, and predict reliability under use conditions. This article uses recent testing of a Howitzer to illustrate the challenges in designing reliability experiments for complex, repairable systems. We leverage lessons learned from current research and propose methods for designing an experiment for a complex, repairable system. Suggested Citation Freeman, Laura J., Rebecca M. Medlin, and Thomas H. Engineers use reliability experiments to determine the factors that drive product reliability, build robust products, and predict reliability under use conditions. This article uses recent testing of a Howitzer to illustrate the challenges in designing reliability experiments for complex, repairable systems. We leverage lessons learned from current research and propose methods for designing an experiment for a complex, repairable system.

Suggested Citation

Freeman, Laura J., Rebecca M. Medlin, and Thomas H. Johnson. “Challenges and New Methods for Designing Reliability Experiments.” Quality Engineering 31, no. 1 (January 2, 2019): 108–21. https://doi.org/10.1080/08982112.2018.1546394.

Paper:

]]> Managing T&E Data to Encourage Reuse https://research.testscience.org/post/2019-managing-t-e-data-to-encourage-reuse/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-managing-t-e-data-to-encourage-reuse/ Reusing Test and Evaluation (T&E) datasets multiple times at different points throughout a program’s lifecycle is one way to realize their full value. Data management plays an important role in enabling - and even encouraging – this practice. Although Department-level policy on data management is supportive of reuse and consistent with best practices from industry and academia, the documents that shape the day-to-day activities of T&E practitioners are much less so. Reusing Test and Evaluation (T&E) datasets multiple times at different points throughout a program’s lifecycle is one way to realize their full value. Data management plays an important role in enabling - and even encouraging – this practice. Although Department-level policy on data management is supportive of reuse and consistent with best practices from industry and academia, the documents that shape the day-to-day activities of T&E practitioners are much less so. As a result, reuse of T&E datasets does not occur on a consistent basis or in a formalized way. To fill this apparent gap, this article expands upon four best practices – addressed in different ways in Service-specific T&E policies – that can increase the reuse of T&E datasets.

Suggested Citation

Medlin, Rebecca, and Andrew Flack. “Managing T&E Data to Encourage Reuse.” The ITEA Journal of Test and Evaluation of Test and Evaluation, 2019.

Paper:

]]> Analysis of Split-Plot Reliability Experiments with Subsampling https://research.testscience.org/post/2018-analysis-of-split-plot-reliability-experiments-with-subsampling/ Mon, 01 Jan 2018 00:00:00 +0000 https://research.testscience.org/post/2018-analysis-of-split-plot-reliability-experiments-with-subsampling/ Reliability experiments are important for determining which factors drive product reliability. The data collected in these experiments can be challenging to analyze. Often, the reliability or lifetime data collected follow distinctly nonnormal distributions and include censored observations. Additional challenges in the analysis arise when the experiment is executed with restrictions on randomization. The focus of this paper is on the proper analysis of reliability data collected from a nonrandomized reliability experiments. Reliability experiments are important for determining which factors drive product reliability. The data collected in these experiments can be challenging to analyze. Often, the reliability or lifetime data collected follow distinctly nonnormal distributions and include censored observations. Additional challenges in the analysis arise when the experiment is executed with restrictions on randomization. The focus of this paper is on the proper analysis of reliability data collected from a nonrandomized reliability experiments. Specifically, we focus on the analysis of lifetime data from a split-plot experimental design. We outline a nonlinear mixed-model analysis for a split-plot reliability experiment with subsampling and right-censored Weibull distributed lifetime data. A simulation study compares the proposed method with a two-stage method of analysis.

Suggested Citation

Medlin, Rebecca M., Laura J. Freeman, Jennifer L.K. Kensler, and G. Geoffrey Vining. “Analysis of Split-Plot Reliability Experiments with Subsampling.” Quality and Reliability Engineering International 35, no. 3 (2019): 738–49. https://doi.org/10.1002/qre.2394.

Paper:

]]> Power Approximations for Reliability Test Designs https://research.testscience.org/post/2018-power-approximations-for-reliability-test-designs/ Mon, 01 Jan 2018 00:00:00 +0000 https://research.testscience.org/post/2018-power-approximations-for-reliability-test-designs/ Reliability tests determine which factors drive system reliability. Often, the reliability or failure time data collected in these tests tend to follow distinctly non- normal distributions and include censored observations. The experimental design should accommodate the skewed nature of the response and allow for censored observations, which occur when systems under test do not fail within the allotted test time. To account for these design and analysis considerations, Monte Carlo simulations are frequently used to evaluate experimental design properties. Reliability tests determine which factors drive system reliability. Often, the reliability or failure time data collected in these tests tend to follow distinctly non- normal distributions and include censored observations. The experimental design should accommodate the skewed nature of the response and allow for censored observations, which occur when systems under test do not fail within the allotted test time. To account for these design and analysis considerations, Monte Carlo simulations are frequently used to evaluate experimental design properties. Simulation provides accurate power calculations as a function of sample size, allowing researchers to determine adequate sample sizes at each level of the treatment. However, simulation may be inefficient for comparing multiple experiments of various sizes. In this document, we present a closed form approach for calculating power, based on the non- central chi-squared approximation to the distribution of the likelihood ratio statistic. The solution can be used to compare multiple designs and accommodate trade-space analyses between power, effect size, model formulation, sample size, censoring rates, and design type. To demonstrate the efficiency of our approach, we provide a comparison to estimates that are generated using Monte Carlo simulation.

Suggested Citation

Johnson, Thomas H., Rebecca M. Medlin, and Laura Freeman. “Power Approximations for Failure-Time Regression Models.” Quality and Reliability Engineering International 35, no. 6 (2019): 1666–75. https://doi.org/10.1002/qre.2467.

Slides:

Paper:

]]> Comparing Live Missile Fire and Simulation https://research.testscience.org/post/2017-comparing-live-missile-fire-and-simulation/ Sun, 01 Jan 2017 00:00:00 +0000 https://research.testscience.org/post/2017-comparing-live-missile-fire-and-simulation/ Modeling and Simulation is frequently used in Test and Evaluation (T&E) of air-to-air weapon systems to evaluate the effectiveness of a weapons. The AirIntercept Missile-9X (AIM-9X) program uses modeling and simulationextensively to evaluate missile miss distances. Since flight testing isexpensive, the test program uses relatively few flight tests and supplementsthose data with large numbers of miss distances from simulated tests acrossthe weapons operational space. However, before modeling and simulation canbe used to predict performance it must first be validated. Modeling and Simulation is frequently used in Test and Evaluation (T&E) of air-to-air weapon systems to evaluate the effectiveness of a weapons. The AirIntercept Missile-9X (AIM-9X) program uses modeling and simulationextensively to evaluate missile miss distances. Since flight testing isexpensive, the test program uses relatively few flight tests and supplementsthose data with large numbers of miss distances from simulated tests acrossthe weapons operational space. However, before modeling and simulation canbe used to predict performance it must first be validated. Validation isespecially challenging when working with a limited number of live test data. Inthis presentation, we show that even with a limited number of live test points(e.g., 16 missile fires), we can still perform a statistical analysis for thevalidation. We introduce a validation technique known as Fisher’s CombinedProbability Test and show how to apply Fisher’s test to validate the AIM-9Xmodel and simulation.

Suggested Citation

Medlin, Rebecca, Pamela Rambow, and Douglas Peek. Comparing Live Missile Fire and Simulation. IDA Document NS D-8443. Alexandria, VA: Institute for Defense Analyses, 2017.

Slides:

]]> On Scoping a Test that Addresses the Wrong Objective https://research.testscience.org/post/2017-on-scoping-a-test-that-addresses-the-wrong-objective/ Sun, 01 Jan 2017 00:00:00 +0000 https://research.testscience.org/post/2017-on-scoping-a-test-that-addresses-the-wrong-objective/ Statistical literature refers to a type of error that is committed by giving the right answer to the wrong question. If a test design is adequately scoped to address an irrelevant objective, one could say that a Type III error occurs. In this paper, we focus on a specific Type III error that on some occasions test planners commit to reduce test size and resources. Suggested Citation Johnson, Thomas H., Rebecca M. Statistical literature refers to a type of error that is committed by giving the right answer to the wrong question. If a test design is adequately scoped to address an irrelevant objective, one could say that a Type III error occurs. In this paper, we focus on a specific Type III error that on some occasions test planners commit to reduce test size and resources.

Suggested Citation

Johnson, Thomas H., Rebecca M. Medlin, Laura J. Freeman, and James R. Simpson. “On Scoping a Test That Addresses the Wrong Objective.” Quality Engineering 31, no. 2 (April 3, 2019): 230–39. https://doi.org/10.1080/08982112.2018.1479035.

Paper:

]]> DOT&E Reliability Course https://research.testscience.org/post/2016-dot-e-reliability-course/ Fri, 01 Jan 2016 00:00:00 +0000 https://research.testscience.org/post/2016-dot-e-reliability-course/ This reliability course provides information to assist DOT&E action officers in their review and assessment of system reliability. Course briefings cover reliability planning and analysis activities that span the acquisition life cycle. Each briefing discusses review criteria relevant to DOT&E action officers based on DoD policies and lessons learned from previous oversight efforts. Suggested Citation Avery, Matthew, Jonathan Bell, Rebecca Medlin, and Freeman Laura. DOT&E Reliability Course. IDA Document NS D-5836. This reliability course provides information to assist DOT&E action officers in their review and assessment of system reliability. Course briefings cover reliability planning and analysis activities that span the acquisition life cycle. Each briefing discusses review criteria relevant to DOT&E action officers based on DoD policies and lessons learned from previous oversight efforts.

Suggested Citation

Avery, Matthew, Jonathan Bell, Rebecca Medlin, and Freeman Laura. DOT&E Reliability Course. IDA Document NS D-5836. Alexandria, VA: Institute for Defense Analyses, 2016.

Slides:

]]> Best Practices for Statistically Validating Modeling and Simulation (M&S) Tools Used in Operational Testing https://research.testscience.org/post/2015-best-practices-for-statistically-validating-modeling-and-simulation-m-s-tools-used-in-operational-testing/ Thu, 01 Jan 2015 00:00:00 +0000 https://research.testscience.org/post/2015-best-practices-for-statistically-validating-modeling-and-simulation-m-s-tools-used-in-operational-testing/ In many situations, collecting sufficient data to evaluate system performance against operationally realistic threats is not possible due to cost and resource restrictions, safety concerns, or lack of adequate or representative threats. Modeling and simulation tools that have been verified, validated, and accredited can be used to supplement live testing in order to facilitate a more complete evaluation of performance. Two key questions that frequently arise when planning an operational test are (1) which (and how many) points within the operational space should be chosen in the simulation space and the live space for optimal ability to verify and validate the M&S, and (2) once that data is collected, what is the best way to compare the live trials to the simulated trials for the purpose of validating the M&S? In many situations, collecting sufficient data to evaluate system performance against operationally realistic threats is not possible due to cost and resource restrictions, safety concerns, or lack of adequate or representative threats. Modeling and simulation tools that have been verified, validated, and accredited can be used to supplement live testing in order to facilitate a more complete evaluation of performance. Two key questions that frequently arise when planning an operational test are (1) which (and how many) points within the operational space should be chosen in the simulation space and the live space for optimal ability to verify and validate the M&S, and (2) once that data is collected, what is the best way to compare the live trials to the simulated trials for the purpose of validating the M&S? This conference presentation addresses various strategies for addressing these two questions. The best methodologies for designing and analyzing will vary depending on the goal of operational test, the type of model used in the simulation, and the amount of live and simulated data available.

Suggested Citation

Avery, Kelly, Laura Freeman, and Rebecca Medlin. Best Practices for Statistically Validating Modeling and Simulation (M&S) Tools Used in Operational Testing. IDA Document NS D-5582. Alexandria, VA: Institute for Defense Analyses, 2015.

Slides:

]]> Estimating System Reliability from Heterogeneous Data https://research.testscience.org/post/2015-estimating-system-reliability-from-heterogeneous-data/ Thu, 01 Jan 2015 00:00:00 +0000 https://research.testscience.org/post/2015-estimating-system-reliability-from-heterogeneous-data/ This briefing provides an example of some of the nuanced issues in reliability estimation in operational testing. The statistical models are motivated by an example of the Paladin Integrated Management (PIM). We demonstrate how to use a Bayesian approach to reliability estimation that uses data from all phases of testing. Suggested Citation Browning, Caleb, Laura Freeman, Alyson Wilson, Kassandra Fronczyk, and Rebecca Dickinson. “Estimating System Reliability from Heterogeneous Data.” Presented at the Conference on Applied Statistics in Defense, George Mason University, October 2015. This briefing provides an example of some of the nuanced issues in reliability estimation in operational testing. The statistical models are motivated by an example of the Paladin Integrated Management (PIM). We demonstrate how to use a Bayesian approach to reliability estimation that uses data from all phases of testing.

Suggested Citation

Browning, Caleb, Laura Freeman, Alyson Wilson, Kassandra Fronczyk, and Rebecca Dickinson. “Estimating System Reliability from Heterogeneous Data.” Presented at the Conference on Applied Statistics in Defense, George Mason University, October 2015.

Slides:

]]> Statistical Models for Combining Information Stryker Reliability Case Study https://research.testscience.org/post/2015-statistical-models-for-combining-information-stryker-reliability-case-study/ Thu, 01 Jan 2015 00:00:00 +0000 https://research.testscience.org/post/2015-statistical-models-for-combining-information-stryker-reliability-case-study/ Reliability is an essential element in assessing the operational suitability of Department of Defense weapon systems. Reliability takes a prominent role in both the design and analysis of operational tests. In the current era of reduced budgets and increased reliability requirements, it is challenging to verify reliability requirements in a single test. Furthermore, all available data should be considered in order to ensure evaluations provide the most appropriate analysis of the system’s reliability. Reliability is an essential element in assessing the operational suitability of Department of Defense weapon systems. Reliability takes a prominent role in both the design and analysis of operational tests. In the current era of reduced budgets and increased reliability requirements, it is challenging to verify reliability requirements in a single test. Furthermore, all available data should be considered in order to ensure evaluations provide the most appropriate analysis of the system’s reliability. This paper describes the benefits of using parametric statistical models to combine information across multiple testing events. Both frequentist and Bayesian inference techniques are employed and they are compared and contrasted to illustrate different statistical methods for combining information. We apply these methods to data collected during the developmental and operational test phases for the Stryker family of vehicles. We show that, when we combine the available information across two test phases for the Stryker family of vehicles, reliability estimates are more accurate and precise than those reported previously using traditional methods that use only operational test data in their reliability assessments.

Suggested Citation

Steiner, Stefan, Rebecca M. Dickinson, Laura J. Freeman, Bruce A. Simpson, and Alyson G. Wilson. “Statistical Methods for Combining Information: Stryker Family of Vehicles Reliability Case Study.” Journal of Quality Technology 47, no. 4 (October 2015): 400–415. https://doi.org/10.1080/00224065.2015.11918142.

Slides:

Paper:

]]> Validating the PRA Testbed Using a Statistically Rigorous Approach https://research.testscience.org/post/2015-validating-the-pra-testbed-using-a-statistically-rigorous-approach/ Thu, 01 Jan 2015 00:00:00 +0000 https://research.testscience.org/post/2015-validating-the-pra-testbed-using-a-statistically-rigorous-approach/ For many systems, testing is expensive and only a few live test events are conducted. When this occurs, testers frequently use a model to extend the test results. However, testers must validate the model to show that it is an accurate representation of the real world from the perspective of the intended uses of the model. This raises a problem when only a small number of live test events are conducted, only limited data are available to validate the model, and some testers struggle with model validation. For many systems, testing is expensive and only a few live test events are conducted. When this occurs, testers frequently use a model to extend the test results. However, testers must validate the model to show that it is an accurate representation of the real world from the perspective of the intended uses of the model. This raises a problem when only a small number of live test events are conducted, only limited data are available to validate the model, and some testers struggle with model validation. This article describes a statistically rigorous approach for validating a model with only a small number of live test results. We discuss a specific application for validating a model of a naval surface combatant defending itself against a cruise missile attack. The approach takes into account potential correlation in the data and other factors that may drive system performance.

Suggested Citation

Thomas, Dean, and Rebecca Dickinson. “Validating the Probability of Raid Annihilation Testbed Using a Statistical Approach.” The ITEA Journal of Test and Evaluation 36, no. 2 (June 2015).

Paper:

]]>