Justace Clutter on Test Science Research Document Library https://research.testscience.org/researchers/justace-clutter/ Recent content in Justace Clutter on Test Science Research Document Library Hugo -- 0.129.0 en-us Institute for Defense Analyses Mon, 01 Jan 2018 00:00:00 +0000 Testing Defense Systems https://research.testscience.org/post/2018-testing-defense-systems/ Mon, 01 Jan 2018 00:00:00 +0000 https://research.testscience.org/post/2018-testing-defense-systems/ The complex, multifunctional nature of defense systems, along with the wide variety of system types, demands a structured but flexible analytical process for testing systems. This chapter summarizes commonly used techniques in defense system testing and specific challenges imposed by the nature of defense system testing. It highlights the core statistical methodologies that have proven useful in testing defense systems. Case studies illustrate the value of using statistical techniques in the design of tests and analysis of the resulting data. The complex, multifunctional nature of defense systems, along with the wide variety of system types, demands a structured but flexible analytical process for testing systems. This chapter summarizes commonly used techniques in defense system testing and specific challenges imposed by the nature of defense system testing. It highlights the core statistical methodologies that have proven useful in testing defense systems. Case studies illustrate the value of using statistical techniques in the design of tests and analysis of the resulting data. The chapter focuses on the unique statistical challenges of designing operational tests, many of which can be attributed to the process, but some of which are inherent to the complexity of the systems and the missions system operators must complete. It provides an overview of the process of designing experiments for military systems with operational users in an operational environment.

Suggested Citation

Freeman, Laura J., Thomas Johnson, Matthew Avery, V. Bram Lillard, and Justace Clutter. “Testing Defense Systems.” In Analytic Methods in Systems and Software Testing, 439–87. John Wiley & Sons, Ltd, 2018. https://doi.org/10.1002/9781119357056.ch18.

Paper:

]]> Design of Experiments for in-Lab Operational Testing of the an/BQQ-10 Submarine Sonar System https://research.testscience.org/post/2014-design-of-experiments-for-in-lab-operational-testing-of-the-an-bqq-10-submarine-sonar-system/ Wed, 01 Jan 2014 00:00:00 +0000 https://research.testscience.org/post/2014-design-of-experiments-for-in-lab-operational-testing-of-the-an-bqq-10-submarine-sonar-system/ Operational testing of the AN/BQQ-10 submarine sonar system has never been able to show significant improvements in software versions because of the high variability of at sea measurements. To mitigate this problem, in the most recent AN/BQQ-10 operational test, the Navy’s operational test agency (in consultation with IDA under the direction of Director, Operational Test and Evaluation) supplemented the at sea testing with an operationally focused in-lab comparison. This test used recorded real data played back on two different versions of the sonar system. Operational testing of the AN/BQQ-10 submarine sonar system has never been able to show significant improvements in software versions because of the high variability of at sea measurements. To mitigate this problem, in the most recent AN/BQQ-10 operational test, the Navy’s operational test agency (in consultation with IDA under the direction of Director, Operational Test and Evaluation) supplemented the at sea testing with an operationally focused in-lab comparison. This test used recorded real data played back on two different versions of the sonar system. For each version, the test recorded the time it took multiple operations, with varying operational experience, to detect a submarine target once it appeared on the display. This new test methodology had several benefits: (1) the laboratory setting allowed for the use of design of experiments to control factors that are traditionally infeasible to control during an at sea test; (2) the direct comparison between the two systems resulted in demonstrating a statistically significant reduction in the detection time for the new system. Although laboratory testing cannot replace at sea testing, the results provide strong indication that we can expect performance improvements in the operational environment. This case study shows that laboratory testing and design of experiments have a place in operational testing and should be expanded to improve testing for other systems.

Suggested Citation

Clutter, Justace R, George Khoury, and Laura Freeman. Design of Experiments for In-Lab Operational Testing of the AN/BQQ-10 Submarine Sonar System. IDA Document NS D-5486. Alexandria, VA: Institute for Defense Analyses, 2014.

Slides:

]]>