2019 on Test Science Research Document Library https://research.testscience.org/year/2019/ Recent content in 2019 on Test Science Research Document Library Hugo -- 0.129.0 en-us Institute for Defense Analyses Tue, 01 Jan 2019 00:00:00 +0000 Bayesian Component Reliability- An F-35 Case Study https://research.testscience.org/post/2019-bayesian-component-reliability-an-f-35-case-study/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-bayesian-component-reliability-an-f-35-case-study/ A challenging aspect ofa system reliability assessment is integratingmultiple sources of information, such as component, subsystem, and full-system data,along with previous test data or subject matter expert (SME) opinion. A powerfulfeature of Bayesian analyses is the ability to combine these multiple sources of dataand variability in an informed way to perform statistical inference. This feature isparticularly valuable in assessing system reliability where testing is limited and only asmall number of failures (or none at all) are observed. A challenging aspect ofa system reliability assessment is integratingmultiple sources of information, such as component, subsystem, and full-system data,along with previous test data or subject matter expert (SME) opinion. A powerfulfeature of Bayesian analyses is the ability to combine these multiple sources of dataand variability in an informed way to perform statistical inference. This feature isparticularly valuable in assessing system reliability where testing is limited and only asmall number of failures (or none at all) are observed.The F-35 is DoD’s largest program; approximately one-third of the operations andsustainment cost is attributed to the cost of spare parts and the removal, replacement,and repair of components. The failure rate of those components is the drivingparameter for a significant portion of the sustainment cost, and yet for many of thesecomponents, available estimates of the failure rate are poor. For many programs, thecontractor produces estimates of component failure rates based on engineering analysisand legacy systems with similar parts. While these estimates are useful, the actualremoval rates provide a more accurate estimate of the removal and replacement ratesthe program will experience in future years.In this document, we show how we applied a Bayesian analysis to combine theengineering reliability estimates with the actual failure data to estimate componentreliability. Our analysis technique also allows for us to overcome the problems of caseswhere few or no failures have been observed. We are able to show that combining theengineering knowledge of reliability with the observed operational reliability results inboth a more informed estimate of each individual component’s reliaiblity and a moreinformed estimate of overall F-35 maintenance costs.The technique presented is broadly applicable to any progam where multiple sourcesof reliability information need to be combined for the best estimation of componentfailure rates, and ultimately of sustainment costs.

Suggested Citation

Medlin, Rebecca M, and V. Bram Lillard. Bayesian Component Reliability Estimation: An F-35 Case Study. IDA Document NS D-10561. Alexandria, VA: Institute for Defense Analyses, 2019.

Slides:

]]> Challenges and New Methods for Designing Reliability Experiments https://research.testscience.org/post/2019-challenges-and-new-methods-for-designing-reliability-experiments/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-challenges-and-new-methods-for-designing-reliability-experiments/ Engineers use reliability experiments to determine the factors that drive product reliability, build robust products, and predict reliability under use conditions. This article uses recent testing of a Howitzer to illustrate the challenges in designing reliability experiments for complex, repairable systems. We leverage lessons learned from current research and propose methods for designing an experiment for a complex, repairable system. Suggested Citation Freeman, Laura J., Rebecca M. Medlin, and Thomas H. Engineers use reliability experiments to determine the factors that drive product reliability, build robust products, and predict reliability under use conditions. This article uses recent testing of a Howitzer to illustrate the challenges in designing reliability experiments for complex, repairable systems. We leverage lessons learned from current research and propose methods for designing an experiment for a complex, repairable system.

Suggested Citation

Freeman, Laura J., Rebecca M. Medlin, and Thomas H. Johnson. “Challenges and New Methods for Designing Reliability Experiments.” Quality Engineering 31, no. 1 (January 2, 2019): 108–21. https://doi.org/10.1080/08982112.2018.1546394.

Paper:

]]> D-Optimal as an Alternative to Full Factorial Designs- a Case Study https://research.testscience.org/post/2019-d-optimal-as-an-alternative-to-full-factorial-designs-a-case-study/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-d-optimal-as-an-alternative-to-full-factorial-designs-a-case-study/ The use of Bayesian statistics and experimental design as tools to scope testing and analyze data related to defense has increased in recent years. Planning a test using experimental design will allow testers to cover the operational space while maximizing the information obtained from each run. Understanding which factors can affect a detector’s performance can influence military tactics, techniques and procedures, and improve a commander’s situational awareness when making decisions in an operational environment. The use of Bayesian statistics and experimental design as tools to scope testing and analyze data related to defense has increased in recent years. Planning a test using experimental design will allow testers to cover the operational space while maximizing the information obtained from each run. Understanding which factors can affect a detector’s performance can influence military tactics, techniques and procedures, and improve a commander’s situational awareness when making decisions in an operational environment. This presentation will explain how a D-optimal experimental design could be an option for planning a test when the number of runs is limited but an adequate test is desired. Additionally, it will describe how the results of a Bayesian multiple logistic model could be used to show in what way the operational environment can affect the detector’s performance.

Suggested Citation

Anderson, Breeana G, Heather M Wojton, and Keyla Pagan-Rivera. D-Optimal as an Alternative to Full Factorial Designs: A Case Study. IDA Document NS D-10580. Alexandria, VA: Institute for Defense Analyses, 2019.

Poster:

]]> Demystifying the Black Box- A Test Strategy for Autonomy https://research.testscience.org/post/2019-demystifying-the-black-box-a-test-strategy-for-autonomy/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-demystifying-the-black-box-a-test-strategy-for-autonomy/ The purpose of this briefing is to provide a high-level overview of how to frame the question of testing autonomous systems in a way that will enable development of successful test strategies. The brief outlines the challenges and broad-stroke reforms needed to get ready for the test challenges of the next century. Suggested Citation Wojton, Heather M, and Daniel J Porter. Demystifying the Black Box: A Test Strategy for Autonomy. IDA Document NS D-10465-NS. The purpose of this briefing is to provide a high-level overview of how to frame the question of testing autonomous systems in a way that will enable development of successful test strategies. The brief outlines the challenges and broad-stroke reforms needed to get ready for the test challenges of the next century.

Suggested Citation

Wojton, Heather M, and Daniel J Porter. Demystifying the Black Box: A Test Strategy for Autonomy. IDA Document NS D-10465-NS. Alexandria, VA: Institute for Defense Analyses, 2019.

Paper:

]]> Designing Experiments for Model Validation- The Foundations for Uncertainty Quantification https://research.testscience.org/post/2019-designing-experiments-for-model-validation-the-foundations-for-uncertainty-quantification/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-designing-experiments-for-model-validation-the-foundations-for-uncertainty-quantification/ Advances in computational power have allowed both greater fidelity and more extensive use of such models. Numerous complex military systems have a corresponding model that simulates its performance in the field. In response, the DoD needs defensible practices for validating these models. Design of Experiments and statistical analysis techniques are the foundational building blocks for validating the use of computer models and quantifying uncertainty in that validation. Recent developments in uncertainty quantification have the potential to benefit the DoD in using modeling and simulation to inform operational evaluations. Advances in computational power have allowed both greater fidelity and more extensive use of such models. Numerous complex military systems have a corresponding model that simulates its performance in the field. In response, the DoD needs defensible practices for validating these models. Design of Experiments and statistical analysis techniques are the foundational building blocks for validating the use of computer models and quantifying uncertainty in that validation. Recent developments in uncertainty quantification have the potential to benefit the DoD in using modeling and simulation to inform operational evaluations.

Suggested Citation

Wojton, Heather, Kelly Avery, Laura Freeman, and Thomas Johnson. “Designing Experiments for Model Validation – The Foundations for Uncertainty Quantification.” The ITEA Journal of Test and Evaluation 40, no. 1 (2019).

Paper:

]]> Handbook on Statistical Design & Analysis Techniques for Modeling & Simulation Validation https://research.testscience.org/post/2019-handbook-on-statistical-design-analysis-techniques-for-modeling-simulation-validation/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-handbook-on-statistical-design-analysis-techniques-for-modeling-simulation-validation/ This handbook focuses on methods for data-driven validation to supplement the vast existing literature for Verification, Validation, and Accreditation (VV&A) and the emerging references on uncertainty quantification (UQ). The goal of this handbook is to aid the test and evaluation (T&E) community in developing test strategies that support model validation (both external validation and parametric analysis) and statistical UQ. Suggested Citation Wojton, Heather, Kelly M Avery, Laura J Freeman, Samuel H Parry, Gregory S Whittier, Thomas H Johnson, and Andrew C Flack. This handbook focuses on methods for data-driven validation to supplement the vast existing literature for Verification, Validation, and Accreditation (VV&A) and the emerging references on uncertainty quantification (UQ). The goal of this handbook is to aid the test and evaluation (T&E) community in developing test strategies that support model validation (both external validation and parametric analysis) and statistical UQ.

Suggested Citation

Wojton, Heather, Kelly M Avery, Laura J Freeman, Samuel H Parry, Gregory S Whittier, Thomas H Johnson, and Andrew C Flack. Handbook on Statistical Design & Analysis Techniques for Modeling & Simulation Validation. IDA Document NS D-10455. Alexandria, VA: Institute for Defense Analyses, 2019.

Slides:

Paper:

]]> Impact of Conditions which Affect Exploratory Factor Analysis https://research.testscience.org/post/2019-impact-of-conditions-which-affect-exploratory-factor-analysis/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-impact-of-conditions-which-affect-exploratory-factor-analysis/ Some responses cannot be observed directly and must be inferred from multiple indirect measurements, for example human experiences accessed through a variety of survey questions. Exploratory Factor Analysis (EFA) is a data-driven method to optimally combine these indirect measurements to infer some number of unobserved factors. Ideally, EFA should identify how many unobserved factors the indirect measures help estimate (factor extraction), as well as accurately capture how well each indirect measure estimates each factor (parameter recovery). Some responses cannot be observed directly and must be inferred from multiple indirect measurements, for example human experiences accessed through a variety of survey questions. Exploratory Factor Analysis (EFA) is a data-driven method to optimally combine these indirect measurements to infer some number of unobserved factors. Ideally, EFA should identify how many unobserved factors the indirect measures help estimate (factor extraction), as well as accurately capture how well each indirect measure estimates each factor (parameter recovery).

Suggested Citation

Krost, Kevin, Daniel J Porter, Stephanie T Lane, and Heather M Wojton. Impact of Conditions Which Affect Exploratory Factor Analysis. IDA Document NS D-10622. Alexandria, VA: Institute for Defense Analyses, 2019.

Poster:

]]> Initial Validation of the Trust of Automated Systems Test (TOAST) https://research.testscience.org/post/2019-initial-validation-of-the-trust-of-automated-systems-test-toast/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-initial-validation-of-the-trust-of-automated-systems-test-toast/ Trust is a key determinant of whether people rely on automated systems in the military and the public. However, there is currently no standard for measuring trust in automated systems. In the present studies we propose a scale to measure trust in automated systems that is grounded in current research and theory on trust formation, which we refer to as the Trust in Automated Systems Test (TOAST). We evaluated both the reliability of the scale structure and criterion validity using independent, military-affiliated and civilian samples.

Trust is a key determinant of whether people rely on automated systems in the military and the public. However, there is currently no standard for measuring trust in automated systems. In the present studies we propose a scale to measure trust in automated systems that is grounded in current research and theory on trust formation, which we refer to as the Trust in Automated Systems Test (TOAST). We evaluated both the reliability of the scale structure and criterion validity using independent, military-affiliated and civilian samples. In both studies we found that the TOAST exhibited a two-factor structure, measuring system understanding and performance (respectively), and that factor scores significantly predicted scores on theoretically related constructs demonstrating clear criterion validity. We discuss the implications of our findings for advancing the empirical literature and in improving interface design.

Suggested Citation

Wojton, Heather M., Daniel Porter, Stephanie T Lane, Chad Bieber, and Poornima Madhavan. “Initial Validation of the Trust of Automated Systems Test (TOAST).” The Journal of Social Psychology 160, no. 6 (November 1, 2020): 735–50. https://doi.org/10.1080/00224545.2020.1749020.

Paper:

]]> M&S Validation for the Joint Air-to-Ground Missile https://research.testscience.org/post/2019-m-s-validation-for-the-joint-air-to-ground-missile/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-m-s-validation-for-the-joint-air-to-ground-missile/ An operational test is resource-limited and must therefore rely on both live test data and modeling and simulation (M&S) data to inform a full evaluation. For the Joint Air-to-Ground Missile (JAGM) system, we needed to create a test design that accomplished dual goals, characterizing missile performance across the operational space and supporting rigorous validation of the M&S. Our key question is which statistical techniques should be used to compare the M&S to the live data? An operational test is resource-limited and must therefore rely on both live test data and modeling and simulation (M&S) data to inform a full evaluation. For the Joint Air-to-Ground Missile (JAGM) system, we needed to create a test design that accomplished dual goals, characterizing missile performance across the operational space and supporting rigorous validation of the M&S. Our key question is which statistical techniques should be used to compare the M&S to the live data?

Suggested Citation

Crabtree, Brent, Andrew Cseko, Joel Williamson, and Kelly Avery. M&S Validation for the Joint Air-to-Ground Missile. Alexandria, VA: Institute for Defense Analyses, 2019.

Poster:

]]> Managing T&E Data to Encourage Reuse https://research.testscience.org/post/2019-managing-t-e-data-to-encourage-reuse/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-managing-t-e-data-to-encourage-reuse/ Reusing Test and Evaluation (T&E) datasets multiple times at different points throughout a program’s lifecycle is one way to realize their full value. Data management plays an important role in enabling - and even encouraging – this practice. Although Department-level policy on data management is supportive of reuse and consistent with best practices from industry and academia, the documents that shape the day-to-day activities of T&E practitioners are much less so. Reusing Test and Evaluation (T&E) datasets multiple times at different points throughout a program’s lifecycle is one way to realize their full value. Data management plays an important role in enabling - and even encouraging – this practice. Although Department-level policy on data management is supportive of reuse and consistent with best practices from industry and academia, the documents that shape the day-to-day activities of T&E practitioners are much less so. As a result, reuse of T&E datasets does not occur on a consistent basis or in a formalized way. To fill this apparent gap, this article expands upon four best practices – addressed in different ways in Service-specific T&E policies – that can increase the reuse of T&E datasets.

Suggested Citation

Medlin, Rebecca, and Andrew Flack. “Managing T&E Data to Encourage Reuse.” The ITEA Journal of Test and Evaluation of Test and Evaluation, 2019.

Paper:

]]> Operational Testing of Systems with Autonomy https://research.testscience.org/post/2019-operational-testing-of-systems-with-autonomy/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-operational-testing-of-systems-with-autonomy/ Systems with autonomy pose unique challenges for operational test. This document provides an executive level overview of these issues and the proposed solutions and reforms. In order to be ready for the testing challenges of the next century, we will need to change the entire acquisition life cycle, starting even from initial system conceptualization. This briefing was presented to the Director, Operational Test & Evaluation along with his deputies and Chief Scientist. Systems with autonomy pose unique challenges for operational test. This document provides an executive level overview of these issues and the proposed solutions and reforms. In order to be ready for the testing challenges of the next century, we will need to change the entire acquisition life cycle, starting even from initial system conceptualization. This briefing was presented to the Director, Operational Test & Evaluation along with his deputies and Chief Scientist.

Suggested Citation

Wojton, Heather M, Daniel Porter, Yevgeniya Pinelis, Chad Bieber, Heather Wojton, Michael McAnally, and Laura Freeman. Operational Testing of Systems with Autonomy. IDA Document NS D-9266. Alexandria, VA: Institute for Defense Analyses, 2019.

Slides:

]]> Pilot Training Next- Modeling Skill Transfer in a Military Learning Environment https://research.testscience.org/post/2019-pilot-training-next-modeling-skill-transfer-in-a-military-learning-environment/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-pilot-training-next-modeling-skill-transfer-in-a-military-learning-environment/ Pilot Training Next is an exploratory investigation of new technologies and procedures to increase the efficiency of Undergraduate Pilot Training in the United States Air Force. IDA analysts present a method of quantifying skill transfer from simulators to aircraft under realistic, uncontrolled conditions. Suggested Citation Porter, Daniel, Emily Fedele, and Heather Wojton. Pilot Training Next: Modeling Skill Transfer in a Military Learning Environment. IDA Document NS D-10927. Alexandria, VA: Institute for Defense Analyses, 2019. Pilot Training Next is an exploratory investigation of new technologies and procedures to increase the efficiency of Undergraduate Pilot Training in the United States Air Force. IDA analysts present a method of quantifying skill transfer from simulators to aircraft under realistic, uncontrolled conditions.

Suggested Citation

Porter, Daniel, Emily Fedele, and Heather Wojton. Pilot Training Next: Modeling Skill Transfer in a Military Learning Environment. IDA Document NS D-10927. Alexandria, VA: Institute for Defense Analyses, 2019.

Slides:

]]> Reproducible Research Mini-Tutorial https://research.testscience.org/post/2019-reproducible-research-mini-tutorial/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-reproducible-research-mini-tutorial/ Analyses are reproducible if the same methods applied to the same data produce identical results when run again by another researcher (or you in the future). Reproducible analyses are transparent and easy for reviewers to verify, as results and figures can be traced directly to the data and methods that produced them. There are also direct benefits to the researcher. Real-world analysis workflows inevitably require changes to incorporate new or additional data, or to address feedback from collaborators, reviewers, or sponsors.

Analyses are reproducible if the same methods applied to the same data produce identical results when run again by another researcher (or you in the future). Reproducible analyses are transparent and easy for reviewers to verify, as results and figures can be traced directly to the data and methods that produced them. There are also direct benefits to the researcher. Real-world analysis workflows inevitably require changes to incorporate new or additional data, or to address feedback from collaborators, reviewers, or sponsors.

Suggested Citation

Wojton, Heather, Andrew Flack, John Haman, and Kevin Kirshenbaum. Reproducible Research Mini-Tutorial. IDA Document NS D-10581. Alexandria, VA: Institute for Defense Analyses, 2019.

Slides:

]]> Sample Size Determination Methods Using Acceptance Sampling by Variables https://research.testscience.org/post/2019-sample-size-determination-methods-using-acceptance-sampling-by-variables/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-sample-size-determination-methods-using-acceptance-sampling-by-variables/ Acceptance Sampling by Variables (ASbV) is a statistical testing technique used in Personal Protective Equipment programs to determine the quality of the equipment in First Article and Lot Acceptance Tests. This article intends to remedy the lack of existing references that discuss the similarities between ASbV and certain techniques used in different sub-disciplines within statistics. Understanding ASbV from a statistical perspective allows testers to create customized test plans, beyond what is available in MIL-STD-414. Acceptance Sampling by Variables (ASbV) is a statistical testing technique used in Personal Protective Equipment programs to determine the quality of the equipment in First Article and Lot Acceptance Tests. This article intends to remedy the lack of existing references that discuss the similarities between ASbV and certain techniques used in different sub-disciplines within statistics. Understanding ASbV from a statistical perspective allows testers to create customized test plans, beyond what is available in MIL-STD-414.

Suggested Citation

Walzl, Kerry, Lindsey A Davis, Thomas H Johnson, and Heather M Wojton. Sample Size Determination Methods Using Acceptance Sampling by Variables. IDA Document NS D-10666. Alexandria, VA: Institute for Defense Analyses, 2019.

Paper:

]]> Statistics Boot Camp https://research.testscience.org/post/2019-statistics-boot-camp/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-statistics-boot-camp/ In the test community, we frequently use statistics to extract meaning from data. These inferences may be drawn with respect to topics ranging from system performance to human factors. In this mini-tutorial, we will begin by discussing the use of descriptive and inferential statistics. We will continue by discussing commonly used parametric and nonparametric statistics within the defense community, ranging from comparisons of distributions to comparisons of means. We will conclude with a brief discussion of how to present your statistical findings graphically for maximum impact.

In the test community, we frequently use statistics to extract meaning from data. These inferences may be drawn with respect to topics ranging from system performance to human factors. In this mini-tutorial, we will begin by discussing the use of descriptive and inferential statistics. We will continue by discussing commonly used parametric and nonparametric statistics within the defense community, ranging from comparisons of distributions to comparisons of means. We will conclude with a brief discussion of how to present your statistical findings graphically for maximum impact.

Suggested Citation

Wojton, Heather M, Rebecca M Medlin, Kelly M Avery, and Stephanie T Lane. Statistics Bootcamp. IDA Document NS D-10565. Alexandria, VA: Institute for Defense Analyses, 2019.

Slides:

]]> Survey Testing Automation Tool (STAT) https://research.testscience.org/post/2019-survey-testing-automation-tool-stat/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-survey-testing-automation-tool-stat/ In operational testing, survey administration is typically a manual, paper-driven process. We developed a web-based tool called Survey Testing Automation Tool (STAT), which integrates and automates survey construction, administration, and analysis procedures. STAT introduces a standardized approach to the construction of surveys and includes capabilities for survey management, survey planning, and form generation. Suggested Citation Finnegan, Gary M, Kelly Tran, Tara A McGovern, and William R Whitledge. Survey Testing Automation Tool (STAT). In operational testing, survey administration is typically a manual, paper-driven process. We developed a web-based tool called Survey Testing Automation Tool (STAT), which integrates and automates survey construction, administration, and analysis procedures. STAT introduces a standardized approach to the construction of surveys and includes capabilities for survey management, survey planning, and form generation.

Suggested Citation

Finnegan, Gary M, Kelly Tran, Tara A McGovern, and William R Whitledge. Survey Testing Automation Tool (STAT). IDA Document NS D-10566. Alexandria, VA: Institute for Defense Analyses, 2019.

Poster:

]]> The Effect of Extremes in Small Sample Size on Simple Mixed Models- A Comparison of Level-1 and Level-2 Size https://research.testscience.org/post/2019-the-effect-of-extremes-in-small-sample-size-on-simple-mixed-models-a-comparison-of-level-1-and-level-2-size/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-the-effect-of-extremes-in-small-sample-size-on-simple-mixed-models-a-comparison-of-level-1-and-level-2-size/ We present a simulation study that examines the impact of small sample sizes in both observation and nesting levels of the model on the fixed effect bias, type I error, and the power of a simple mixed model analysis. Despite the need for adjustments to control for type I error inflation, our findings indicate that smaller samples than previously recognized can be used for mixed models under certain conditions prevalent in applied research. We present a simulation study that examines the impact of small sample sizes in both observation and nesting levels of the model on the fixed effect bias, type I error, and the power of a simple mixed model analysis. Despite the need for adjustments to control for type I error inflation, our findings indicate that smaller samples than previously recognized can be used for mixed models under certain conditions prevalent in applied research.

Suggested Citation

Carter, Kristina A, Heather M Wojton, and Stephanie T Lane. “The Effect of Extremes in Small Sample Size on Simple Mixed Models: A Comparison of Level-1 and Level-2 Size.” The ITEA Journal of Test and Evaluation 40, no. 1 (2019): 16–29.

Paper:

]]> The Purpose of Mixed-Effects Models in Test and Evaluation https://research.testscience.org/post/2019-the-purpose-of-mixed-effects-models-in-test-and-evaluation/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-the-purpose-of-mixed-effects-models-in-test-and-evaluation/ Mixed-effects models are the standard technique for analyzing data with grouping structure. In defense testing, these models are useful because they allow us to account for correlations between observations, a feature common in many operational tests. In this article, we describe the advantages of modeling data from a mixed-effects perspective and discuss an R package—ciTools—that equips the user with easy methods for presenting results from this type of model. Suggested Citation Haman, John, Matthew Avery, and Heather Wojton. Mixed-effects models are the standard technique for analyzing data with grouping structure. In defense testing, these models are useful because they allow us to account for correlations between observations, a feature common in many operational tests. In this article, we describe the advantages of modeling data from a mixed-effects perspective and discuss an R package—ciTools—that equips the user with easy methods for presenting results from this type of model.

Suggested Citation

Haman, John, Matthew Avery, and Heather Wojton. “The Purpose of Mixed-Effects Models in Test and Evaluation.” The ITEA Journal of Test and Evaluation 40, no. 4 (2019): 249–55.

Slides:

Paper:

]]> Use of Design of Experiments in Survivability Testing https://research.testscience.org/post/2019-use-of-design-of-experiments-in-survivability-testing/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-use-of-design-of-experiments-in-survivability-testing/ The purpose of survivability testing is to provide decision makers with relevant, credible evidence about the survivability of an aircraft that is conveyed with some degree of certainty or inferential weight. In developing an experiment to accomplish this goal, a test planner faces numerous questions What critical issue or issues are being address? What data are needed to answer the critical issues? What test conditions should be varied? What is the most economical way of varying those conditions? The purpose of survivability testing is to provide decision makers with relevant, credible evidence about the survivability of an aircraft that is conveyed with some degree of certainty or inferential weight. In developing an experiment to accomplish this goal, a test planner faces numerous questions What critical issue or issues are being address? What data are needed to answer the critical issues? What test conditions should be varied? What is the most economical way of varying those conditions? How many test articles are needed? Design of Experiments provides an analytical basis for test planning tradeoffs when answering these questions.

Suggested Citation

Couch, Mark, John Haman, Thomas Johnson, and Heather Wojton. “Designs of Experiments (DOE) in Survivability Testing.” Joint Aircraft Survivability Program - JASP Online (blog), March 2019. https://www.jasp-online.org/asjournal/summer-2019/designs-of-experiments-doe-in-survivability-testing/.

Paper:

]]>