2021 on Test Science Research Document Library https://research.testscience.org/year/2021/ Recent content in 2021 on Test Science Research Document Library Hugo -- 0.129.0 en-us Institute for Defense Analyses Fri, 01 Jan 2021 00:00:00 +0000 Artificial Intelligence & Autonomy Test & Evaluation Roadmap Goals https://research.testscience.org/post/2021-artificial-intelligence-autonomy-test-evaluation-roadmap-goals/ Fri, 01 Jan 2021 00:00:00 +0000 https://research.testscience.org/post/2021-artificial-intelligence-autonomy-test-evaluation-roadmap-goals/ As the Department of Defense acquires new systems with artificial intelligence (AI) and autonomous (AI&A) capabilities, the test and evaluation (T&E) community will need to adapt to the challenges that these novel technologies present. The goals listed in this AI Roadmap address the broad range of tasks that the T&E community will need to achieve in order to properly test, evaluate, verify, and validate AI-enabled and autonomous systems. It includes issues that are unique to AI and autonomous systems, as well as legacy T&E shortcomings that will be compounded by newer technologies. As the Department of Defense acquires new systems with artificial intelligence (AI) and autonomous (AI&A) capabilities, the test and evaluation (T&E) community will need to adapt to the challenges that these novel technologies present. The goals listed in this AI Roadmap address the broad range of tasks that the T&E community will need to achieve in order to properly test, evaluate, verify, and validate AI-enabled and autonomous systems. It includes issues that are unique to AI and autonomous systems, as well as legacy T&E shortcomings that will be compounded by newer technologies.

Suggested Citation

Wojton, Heather, Brian Vickers, Daniel Porter, and Rachel Haga. Artificial Intelligence & Autonomy Test & Evaluation Roadmap Goals. IDA Document NS D-22750. Alexandria, VA: Institute for Defense Analyses, 2021.

Slides:

]]> Determining How Much Testing is Enough- An Exploration of Progress in the Department of Defense Test and Evaluation Community https://research.testscience.org/post/2021-determining-how-much-testing-is-enough-an-exploration-of-progress-in-the-department-of-defense-test-and-evaluation-community/ Fri, 01 Jan 2021 00:00:00 +0000 https://research.testscience.org/post/2021-determining-how-much-testing-is-enough-an-exploration-of-progress-in-the-department-of-defense-test-and-evaluation-community/ This paper describes holistic progress in answering the question of “How much testing is enough?” It covers areas in which the T&E community has made progress, areas in which progress remains elusive, and issues that have emerged since 1994 that provide additional challenges. The selected case studies used to highlight progress are especially interesting examples, rather than a comprehensive look at all programs since 1994. Suggested Citation Medlin, Rebecca, Matthew R Avery, James R Simpson, and Heather M Wojton. This paper describes holistic progress in answering the question of “How much testing is enough?” It covers areas in which the T&E community has made progress, areas in which progress remains elusive, and issues that have emerged since 1994 that provide additional challenges. The selected case studies used to highlight progress are especially interesting examples, rather than a comprehensive look at all programs since 1994.

Suggested Citation

Medlin, Rebecca, Matthew R Avery, James R Simpson, and Heather M Wojton. Determining How Much Testing Is Enough: An Exploration of Progress in the Department of Defense Test and Evaluation Community. IDA Document NS D-21561. Alexandria, VA: Institute for Defense Analyses, 2021.

Paper:

]]> Introduction to Bayesian Analysis https://research.testscience.org/post/2021-introduction-to-bayesian-analysis/ Fri, 01 Jan 2021 00:00:00 +0000 https://research.testscience.org/post/2021-introduction-to-bayesian-analysis/ As operational testing becomes increasingly integrated and research questions become more difficult to answer, IDA’s Test Science team has found Bayesian models to be powerful data analysis methods. Analysts and decision-makers should understand the differences between this approach and the conventional way of analyzing data. It is also important to recognize when an analysis could benefit from the inclusion of prior information—what we already know about a system’s performance—and to understand the proper way to incorporate that information. As operational testing becomes increasingly integrated and research questions become more difficult to answer, IDA’s Test Science team has found Bayesian models to be powerful data analysis methods. Analysts and decision-makers should understand the differences between this approach and the conventional way of analyzing data. It is also important to recognize when an analysis could benefit from the inclusion of prior information—what we already know about a system’s performance—and to understand the proper way to incorporate that information. To apply Bayesian methods, analysts need to comprehend some technical aspects of this approach and know how to properly use appropriate statistical software. In this course, students learn the intuition behind Bayesian statistics, the mathematical details of posterior distributions, how to fit simple Bayesian models using computer software, and how to assess model fit.

Suggested Citation

Wojton, Heather M, Keyla Pagan-Rivera, John T Haman, and Rebecca M Medlin. Introduction to Bayesian Analysis. IDA Document NS D-20484. Alexandria, VA: Institute for Defense Analyses, 2021.

Slides:

]]> Introduction to Qualitative Methods https://research.testscience.org/post/2021-introduction-to-qualitative-methods/ Fri, 01 Jan 2021 00:00:00 +0000 https://research.testscience.org/post/2021-introduction-to-qualitative-methods/ Qualitative data, captured through free-form comment boxes, interviews, focus groups, and activity observation is heavily employed in testing and evaluation (T&E). The qualitative research approach can offer many benefits, but knowledge of how to implement methods, collect data, and analyze data according to rigorous qualitative research standards is not broadly understood within the T&E community. This tutorial offers insight into the foundational concepts of method and practice that embody defensible approaches to qualitative research.

Qualitative data, captured through free-form comment boxes, interviews, focus groups, and activity observation is heavily employed in testing and evaluation (T&E). The qualitative research approach can offer many benefits, but knowledge of how to implement methods, collect data, and analyze data according to rigorous qualitative research standards is not broadly understood within the T&E community.

This tutorial offers insight into the foundational concepts of method and practice that embody defensible approaches to qualitative research. We discuss where qualitative data comes from, how it can be captured, what kind of value it offers, and how to capitalize on that value through methods and best practices.

Suggested Citation

Medlin, Rebecca, Kristina Carter, Emily Fedele, and Daniel Hellmann. Introduction to Qualitative Methods. IDA Document NS D-21591. Alexandria, VA: Institute for Defense Analyses, 2021.

Slides:

]]> Space-Filling Designs for Modeling & Simulation https://research.testscience.org/post/2021-space-filling-designs-for-modeling-simulation/ Fri, 01 Jan 2021 00:00:00 +0000 https://research.testscience.org/post/2021-space-filling-designs-for-modeling-simulation/ This document presents arguments and methods for using space-filling designs (SFDs) to plan modeling and simulation (M&S) data collection. Suggested Citation Avery, Kelly, John T Haman, Thomas Johnson, Curtis Miller, Dhruv Patel, and Han Yi. Test Design Challenges in Defense Testing. IDA Product ID 3002855. Alexandria, VA: Institute for Defense Analyses, 2024. Slides: Paper: This document presents arguments and methods for using space-filling designs (SFDs) to plan modeling and simulation (M&S) data collection.

Suggested Citation

Avery, Kelly, John T Haman, Thomas Johnson, Curtis Miller, Dhruv Patel, and Han Yi. Test Design Challenges in Defense Testing. IDA Product ID 3002855. Alexandria, VA: Institute for Defense Analyses, 2024.

Slides:

Paper:

]]> Warhead Arena Analysis Advancements https://research.testscience.org/post/2021-warhead-arena-analysis-advancements/ Fri, 01 Jan 2021 00:00:00 +0000 https://research.testscience.org/post/2021-warhead-arena-analysis-advancements/ Fragmentation analysis is a critical piece of the live fire test and evaluation (LFT&E) of the lethality and vulnerability aspects of warheads. But the traditional methods for data collection are expensive and laborious. New optical tracking technology is promising to increase the fidelity of fragmentation data, and decrease the time and costs associated with data collection. However, the new data will be complex, three-dimensional “fragmentation clouds,” possibly with a time component as well, and there will be a larger number of individual data points. Fragmentation analysis is a critical piece of the live fire test and evaluation (LFT&E) of the lethality and vulnerability aspects of warheads. But the traditional methods for data collection are expensive and laborious. New optical tracking technology is promising to increase the fidelity of fragmentation data, and decrease the time and costs associated with data collection. However, the new data will be complex, three-dimensional “fragmentation clouds,” possibly with a time component as well, and there will be a larger number of individual data points. This raises questions about how testers can effectively summarize spatial data and use it to draw conclusions about warhead performance for sponsors. In this briefing, we will discuss Bayesian spatial models that are effective for characterizing the mass and velocity fragmentation distributions, along with several exploratory data analysis techniques that help us make sense of the data. Our goals are to

  1. Produce simple statistics and visuals that help the live fire analyst compare and contrast warhead fragmentations.

  2. Characterize important performance attributes or confirm design/spec compliance.

  3. Provide data methods that ensure higher fidelity data collection translates to higher fidelity modeling and simulation down the line.

Suggested Citation

Couch, Mark, Thomas Johnson, John Haman, Kerry Walzl, Heather Wojton, Thomas Hatch-Aguilar, and David Higdon. Warhead Arena Analysis Advancements. IDA Document NS-D-11038. Alexandria, VA: Institute for Defense Analyses, 2021.

Slides:

]]> Why are Statistical Engineers Needed for Test & Evaluation? https://research.testscience.org/post/2021-why-are-statistical-engineers-needed-for-test-evaluation/ Fri, 01 Jan 2021 00:00:00 +0000 https://research.testscience.org/post/2021-why-are-statistical-engineers-needed-for-test-evaluation/ The Department of Defense (DoD) develops and acquires some of the world’s most advanced and sophisticated systems. As new technologies emerge and are incorporated into systems, OSD/DOT&E faces the challenge of ensuring that these systems undergo adequate and efficient test and evaluation (T&E) prior to operational use. Statistical engineering is a collaborative, analytical approach to problem solving that integrates statistical thinking, methods, and tools with other relevant disciplines. The statistical engineering process provides better solutions to large, unstructured, real-world problems and supports rigorous decision-making. The Department of Defense (DoD) develops and acquires some of the world’s most advanced and sophisticated systems. As new technologies emerge and are incorporated into systems, OSD/DOT&E faces the challenge of ensuring that these systems undergo adequate and efficient test and evaluation (T&E) prior to operational use. Statistical engineering is a collaborative, analytical approach to problem solving that integrates statistical thinking, methods, and tools with other relevant disciplines. The statistical engineering process provides better solutions to large, unstructured, real-world problems and supports rigorous decision-making. In this talk, we provide two case study examples related to looking at ways to improve approaches to integrate testing and data collection across the full system lifecycle. These case studies highlight why we believe statistical engineers are necessary for successful T&E.

Suggested Citation

Medlin, Rebecca, Kayla Pagan-Rivera, and Monica Ahrens. Why Are Statistical Engineers Needed for Test & Evaluation? IDA Document NS-D-22722. Alexandria, VA: Institute for Defense Analyses, 2021.

Slides:

]]>