Artificial Intelligence and Autonomy on Test Science Research Document Library https://research.testscience.org/areas/artificial-intelligence-and-autonomy/ Recent content in Artificial Intelligence and Autonomy on Test Science Research Document Library Hugo -- 0.129.0 en-us Institute for Defense Analyses Mon, 01 Jan 2024 00:00:00 +0000 Developing AI Trust- From Theory to Testing and the Myths in Between https://research.testscience.org/post/2024-developing-ai-trust-from-theory-to-testing-and-the-myths-in-between/ Mon, 01 Jan 2024 00:00:00 +0000 https://research.testscience.org/post/2024-developing-ai-trust-from-theory-to-testing-and-the-myths-in-between/ This introductory work aims to provide members of the Test and Evaluation community with a clear understanding of trust and trustworthiness to support responsible and effective evaluation of AI systems. The paper provides a set of working definitions and works toward dispelling confusion and myths surrounding trust. Suggested Citation Razin, Yosef S., and Kristen Alexander. “Developing AI Trust: From Theory to Testing and the Myths in Between.” The ITEA Journal of Test and Evaluation 45, no.

This introductory work aims to provide members of the Test and Evaluation community with a clear understanding of trust and trustworthiness to support responsible and effective evaluation of AI systems. The paper provides a set of working definitions and works toward dispelling confusion and myths surrounding trust.

Suggested Citation

Razin, Yosef S., and Kristen Alexander. “Developing AI Trust: From Theory to Testing and the Myths in Between.” The ITEA Journal of Test and Evaluation 45, no. 1 (March 31, 2024). https://itea.org/journals/volume-45-1/developing-ai-trust-from-theory-to-testing-and-the-myths-in-between/.

Slides:

Paper:

]]> Operational T&E of AI-Supported Data Integration, Fusion, and Analysis Systems https://research.testscience.org/post/2024-operational-t-e-of-ai-supported-data-integration-fusion-and-analysis-systems/ Mon, 01 Jan 2024 00:00:00 +0000 https://research.testscience.org/post/2024-operational-t-e-of-ai-supported-data-integration-fusion-and-analysis-systems/ AI will play an important role in future military systems. However, large questions remain about how to test AI systems, especially in operational settings. Here, we discuss an approach for the operational test and evaluation (OT&E) of AI-supported data integration, fusion, and analysis systems. We highlight new challenges posed by AI-supported systems and we discuss new and existing OT&E methods for overcoming them. We demonstrate how to apply these OT&E methods via a notional test concept that focuses on evaluating an AI-supported data integration system in terms of its technical performance (how accurate is the AI output?

AI will play an important role in future military systems. However, large questions remain about how to test AI systems, especially in operational settings. Here, we discuss an approach for the operational test and evaluation (OT&E) of AI-supported data integration, fusion, and analysis systems. We highlight new challenges posed by AI-supported systems and we discuss new and existing OT&E methods for overcoming them. We demonstrate how to apply these OT&E methods via a notional test concept that focuses on evaluating an AI-supported data integration system in terms of its technical performance (how accurate is the AI output?) and human systems interaction (how does the AI affect users?).

Suggested Citation

Anderson, Breeana G, Adam M Miller, Logan K Ausman, John T Haman, Keyla Pagan-Rivera, Sarah A Shaffer, and Brian D Vickers. Data Integration, Fusion, and Analysis Systems. IDA Product ID 3001848. Alexandria, VA: Institute for Defense Analyses, 2024.

Slides:

]]> A Team-Centric Metric Framework for Testing and Evaluation of Human-Machine Teams https://research.testscience.org/post/2023-a-team-centric-metric-framework-for-testing-and-evaluation-of-human-machine-teams/ Sun, 01 Jan 2023 00:00:00 +0000 https://research.testscience.org/post/2023-a-team-centric-metric-framework-for-testing-and-evaluation-of-human-machine-teams/ We propose and present a parallelized metric framework for evaluating human-machine teams that draws upon current knowledge of human-systems interfacing and integration but is rooted in team-centric concepts. Humans and machines working together as a team involves interactions that will only increase in complexity as machines become more intelligent, capable teammates. Assessing such teams will require explicit focus on not just the human-machine interfacing but the full spectrum of interactions between and among agents. We propose and present a parallelized metric framework for evaluating human-machine teams that draws upon current knowledge of human-systems interfacing and integration but is rooted in team-centric concepts. Humans and machines working together as a team involves interactions that will only increase in complexity as machines become more intelligent, capable teammates. Assessing such teams will require explicit focus on not just the human-machine interfacing but the full spectrum of interactions between and among agents. As opposed to focusing on isolated qualities, capabilities, and performance contributions of individual team members, the proposed framework emphasizes the collective team as the fundamental unit of analysis and the interactions of the team as the key evaluation targets, with individual human and machine metrics still vital but secondary. With teammate interaction as the organizing diagnostic concept, the resulting framework arrives at a parallel assessment of the humans and machines, analyzing their individual capabilities less with respect to purely human or machine qualities and more through the prism of contributions to the team as a whole. This treatment reflects the increased machine capabilities and will allow for continued relevance as machines develop to exercise more authority and responsibility. This framework allows for identification of features specific to human-machine teaming that influence team performance and efficiency, and it provides a basis for operationalizing in specific scenarios. Potential applications of this research include test and evaluation of complex systems that rely on human-system interaction, including—though not limited to—autonomous vehicles, command and control systems, and pilot control systems.

Suggested Citation

Wilkins, Jay, David A. Sparrow, Caitlan A. Fealing, Brian D. Vickers, Kristina A. Ferguson, and Heather Wojton. “A Team-Centric Metric Framework for Testing and Evaluation of Human-Machine Teams.” Systems Engineering 27, no. 3 (May 1, 2024): 466–84. https://doi.org/10.1002/sys.21730.

Slides:

Paper:

]]> AI + Autonomy T&E in DoD https://research.testscience.org/post/2023-ai-autonomy-t-e-in-dod/ Sun, 01 Jan 2023 00:00:00 +0000 https://research.testscience.org/post/2023-ai-autonomy-t-e-in-dod/ Test and evaluation (T&E) of AI-enabled systems (AIES) often emphasizes algorithm accuracy over robust, holistic system performance. While this narrow focus may be adequate for some applications of AI, for many complex uses, T&E paradigms removed from operational realism are insufficient. However, leveraging traditional operational testing (OT) methods for to evaluate AIESs can fail to capture novel sources of risk. This brief establishes a common AI vocabulary and highlights OT challenges posed by AIESs by answering the following questions Test and evaluation (T&E) of AI-enabled systems (AIES) often emphasizes algorithm accuracy over robust, holistic system performance. While this narrow focus may be adequate for some applications of AI, for many complex uses, T&E paradigms removed from operational realism are insufficient. However, leveraging traditional operational testing (OT) methods for to evaluate AIESs can fail to capture novel sources of risk. This brief establishes a common AI vocabulary and highlights OT challenges posed by AIESs by answering the following questions

  1. What is “Artificial Intelligence (AI)”?

a. A brief “AI Primer” defines some common terms, highlights words that are used inconsistently, and discusses where definitions are insufficient for identifying systems that require additional T&E considerations.

  1. How does AI impact T&E?

a. AI isn’t new, but systems with AI pose new challenges and may require structural changes to how we T&E.

  1. What makes DoD applications of AI unique?

a. Many Silicon Valley applications of AI often lack the task complexity and severe consequences of risk faced by DoD.

  1. What is the warfighter’s role?

a. T&E must assure warfighters have calibrated trust & an adequate understanding of system behavior.

  1. What is the state of DoD AI T&E in IDA and OED?

Suggested Citation

Vickers, Brian D, Matthew R Avery, Rachel A Haga, Mark R Herrera, Daniel J Porter, Stuart M Rodgers, and Rebecca M Medlin. AI + Autonomy T&E in DoD. IDA Document NS 3000083. Alexandria, VA: Institute for Defense Analyses, 2023.

Paper:

]]> Artificial Intelligence & Autonomy Test & Evaluation Roadmap Goals https://research.testscience.org/post/2021-artificial-intelligence-autonomy-test-evaluation-roadmap-goals/ Fri, 01 Jan 2021 00:00:00 +0000 https://research.testscience.org/post/2021-artificial-intelligence-autonomy-test-evaluation-roadmap-goals/ As the Department of Defense acquires new systems with artificial intelligence (AI) and autonomous (AI&A) capabilities, the test and evaluation (T&E) community will need to adapt to the challenges that these novel technologies present. The goals listed in this AI Roadmap address the broad range of tasks that the T&E community will need to achieve in order to properly test, evaluate, verify, and validate AI-enabled and autonomous systems. It includes issues that are unique to AI and autonomous systems, as well as legacy T&E shortcomings that will be compounded by newer technologies. As the Department of Defense acquires new systems with artificial intelligence (AI) and autonomous (AI&A) capabilities, the test and evaluation (T&E) community will need to adapt to the challenges that these novel technologies present. The goals listed in this AI Roadmap address the broad range of tasks that the T&E community will need to achieve in order to properly test, evaluate, verify, and validate AI-enabled and autonomous systems. It includes issues that are unique to AI and autonomous systems, as well as legacy T&E shortcomings that will be compounded by newer technologies.

Suggested Citation

Wojton, Heather, Brian Vickers, Daniel Porter, and Rachel Haga. Artificial Intelligence & Autonomy Test & Evaluation Roadmap Goals. IDA Document NS D-22750. Alexandria, VA: Institute for Defense Analyses, 2021.

Slides:

]]> T&E Contributions to Avoiding Unintended Behaviors in Autonomous Systems https://research.testscience.org/post/2020-t-e-contributions-to-avoiding-unintended-behaviors-in-autonomous-systems/ Wed, 01 Jan 2020 00:00:00 +0000 https://research.testscience.org/post/2020-t-e-contributions-to-avoiding-unintended-behaviors-in-autonomous-systems/ To provide assurance that AI-enabled systems will behave appropriately across the range of their operating conditions without performing exhaustive testing, the DoD will need to make inferences about system decision making. However, making these inferences validly requires understanding what causally drives system decision-making, which is not possible when systems are black boxes. In this briefing, we discuss the state of the art and gaps in techniques for obtaining, verifying, validating, and accrediting (OVVA) models of system decision-making. To provide assurance that AI-enabled systems will behave appropriately across the range of their operating conditions without performing exhaustive testing, the DoD will need to make inferences about system decision making. However, making these inferences validly requires understanding what causally drives system decision-making, which is not possible when systems are black boxes. In this briefing, we discuss the state of the art and gaps in techniques for obtaining, verifying, validating, and accrediting (OVVA) models of system decision-making.

Suggested Citation

Porter, Daniel J, and Heather Wojton. T&E Contributions to Avoiding Unintended Behaviors in Autonomous Systems. Vol. IDA Document NS D-12078. Alexandria, VA: Institute for Defense Analyses, 2020.

Slides:

]]> Test & Evaluation of AI-Enabled and Autonomous Systems- A Literature Review https://research.testscience.org/post/2020-test-evaluation-of-ai-enabled-and-autonomous-systems-a-literature-review/ Wed, 01 Jan 2020 00:00:00 +0000 https://research.testscience.org/post/2020-test-evaluation-of-ai-enabled-and-autonomous-systems-a-literature-review/ We summarize a subset of the literature regarding the challenges to and recommendations for the test, evaluation, verification, and validation (TEV&V) of autonomous military systems. This literature review is meant for informational purposes only and does not make any recommendations of its own. A synthesis of the literature identified the following categories of TEV&V challenges Problems arising from the complexity of autonomous systems, Challenges imposed by the structure of the current acquisition system, We summarize a subset of the literature regarding the challenges to and recommendations for the test, evaluation, verification, and validation (TEV&V) of autonomous military systems. This literature review is meant for informational purposes only and does not make any recommendations of its own. A synthesis of the literature identified the following categories of TEV&V challenges

  1. Problems arising from the complexity of autonomous systems,

  2. Challenges imposed by the structure of the current acquisition system,

  3. Lack of methods, tools, and infrastructure for testing,

  4. Novel safety and security issues,

  5. A lack of consensus on policy, standards, and metrics,

  6. Issues around how to integrate humans into the operation and testing of these systems.

Recommendations for how to test autonomous military systems can be sorted into five broad groups

  1. Use certain processes for writing requirements, or for designing and developing systems,

  2. Make targeted investments to develop methods or tools, improve our test infrastructure, or enhance our workforce’s AI skillsets,

  3. Use specific proposed test frameworks,

  4. Employ novel methods for system safety or cybersecurity, and

  5. Adopt specific proposed policies, standards, or metrics.

Suggested Citation

Wojton, Heather M, Daniel J Porter, and John W Dennis. Test & Evaluation of AI-Enabled and Autonomous Systems: A Literature Review. IDA Document NS-D-14331. Alexandria, VA: Institute for Defense Analyses, 2020.

Paper:

]]> Trustworthy Autonomy- A Roadmap to Assurance -- Part 1- System Effectiveness https://research.testscience.org/post/2020-trustworthy-autonomy-a-roadmap-to-assurance-part-1-system-effectiveness/ Wed, 01 Jan 2020 00:00:00 +0000 https://research.testscience.org/post/2020-trustworthy-autonomy-a-roadmap-to-assurance-part-1-system-effectiveness/ The Department of Defense (DoD) has invested significant effort over the past decade considering the role of artificial intelligence and autonomy in national security (e.g., Defense Science Board, 2012, 2016, Deputy Secretary of Defense, 2012, Endsley, 2015, Executive Order No. 13859, 2019, US Department of Defense, 2011, 2019, Zacharias, 2019a). However, these efforts were broadly scoped and only partially touched on how the DoD will certify the safety and performance of these systems. The Department of Defense (DoD) has invested significant effort over the past decade considering the role of artificial intelligence and autonomy in national security (e.g., Defense Science Board, 2012, 2016, Deputy Secretary of Defense, 2012, Endsley, 2015, Executive Order No. 13859, 2019, US Department of Defense, 2011, 2019, Zacharias, 2019a). However, these efforts were broadly scoped and only partially touched on how the DoD will certify the safety and performance of these systems. More recent work has done this big-picture thinking for the test and evaluation (T&E) community (e.g., Ahner & Parson, 2016, Haugh, Sparrow, & Tate, 2018, Porter et al., 2018, Sparrow, Tate, Biddle, Kaminski, & Madhavan, 2018, Zacharias, 2019b). In parallel, individual programs have been generating their own working-level solutions for their own particular use-cases and challenges.

The framework proposed in the current work bridges the gap between the big picture policy recommendations already made and individual program needs. It is meant to serve as a roadmap framework that the T&E community can follow in order to provide evidence that artificial intelligence (AI)-enabled and autonomous systems function as intended. At times we echo broad policy recommendations made by others as they will also enable T&E activities. In other places we make more specific recommendations relating to test planning and analysis. In this document, we present part one of our two-part roadmap. We discuss the challenges and possible solutions to assessing system effectiveness. A future part two will deal with test efficiency, simulation, and infrastructure. Due to the scope of this project, even the main body of this document only provides a survey of the challenges and our proposed solutions. However, this roadmap serves as an outline to a future series of technical papers covering these topics in detail for working-level testers and analysts

Suggested Citation

Porter, Daniel, Michael McAnally, Chad Bieber, Heather Wojton, and Rebecca Medlin. Trustworthy Autonomy: A Roadmap to Assurance Part I: System Effectiveness. IDA Document P-10768-NS. Alexandria, VA: Institute for Defense Analyses, 2020.

Slides:

Paper:

]]> Demystifying the Black Box- A Test Strategy for Autonomy https://research.testscience.org/post/2019-demystifying-the-black-box-a-test-strategy-for-autonomy/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-demystifying-the-black-box-a-test-strategy-for-autonomy/ The purpose of this briefing is to provide a high-level overview of how to frame the question of testing autonomous systems in a way that will enable development of successful test strategies. The brief outlines the challenges and broad-stroke reforms needed to get ready for the test challenges of the next century. Suggested Citation Wojton, Heather M, and Daniel J Porter. Demystifying the Black Box: A Test Strategy for Autonomy. IDA Document NS D-10465-NS. The purpose of this briefing is to provide a high-level overview of how to frame the question of testing autonomous systems in a way that will enable development of successful test strategies. The brief outlines the challenges and broad-stroke reforms needed to get ready for the test challenges of the next century.

Suggested Citation

Wojton, Heather M, and Daniel J Porter. Demystifying the Black Box: A Test Strategy for Autonomy. IDA Document NS D-10465-NS. Alexandria, VA: Institute for Defense Analyses, 2019.

Paper:

]]> Operational Testing of Systems with Autonomy https://research.testscience.org/post/2019-operational-testing-of-systems-with-autonomy/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-operational-testing-of-systems-with-autonomy/ Systems with autonomy pose unique challenges for operational test. This document provides an executive level overview of these issues and the proposed solutions and reforms. In order to be ready for the testing challenges of the next century, we will need to change the entire acquisition life cycle, starting even from initial system conceptualization. This briefing was presented to the Director, Operational Test & Evaluation along with his deputies and Chief Scientist. Systems with autonomy pose unique challenges for operational test. This document provides an executive level overview of these issues and the proposed solutions and reforms. In order to be ready for the testing challenges of the next century, we will need to change the entire acquisition life cycle, starting even from initial system conceptualization. This briefing was presented to the Director, Operational Test & Evaluation along with his deputies and Chief Scientist.

Suggested Citation

Wojton, Heather M, Daniel Porter, Yevgeniya Pinelis, Chad Bieber, Heather Wojton, Michael McAnally, and Laura Freeman. Operational Testing of Systems with Autonomy. IDA Document NS D-9266. Alexandria, VA: Institute for Defense Analyses, 2019.

Slides:

]]>