Operational testing of the AN/BQQ-10 submarine sonar system has never been able to show significant improvements in software versions because of the high variability of at sea measurements. To mitigate this problem, in the most recent AN/BQQ-10 operational test, the Navy’s operational test agency (in consultation with IDA under the direction of Director, Operational Test and Evaluation) supplemented the at sea testing with an operationally focused in-lab comparison. This test used recorded real data played back on two different versions of the sonar system. For each version, the test recorded the time it took multiple operations, with varying operational experience, to detect a submarine target once it appeared on the display. This new test methodology had several benefits: (1) the laboratory setting allowed for the use of design of experiments to control factors that are traditionally infeasible to control during an at sea test; (2) the direct comparison between the two systems resulted in demonstrating a statistically significant reduction in the detection time for the new system. Although laboratory testing cannot replace at sea testing, the results provide strong indication that we can expect performance improvements in the operational environment. This case study shows that laboratory testing and design of experiments have a place in operational testing and should be expanded to improve testing for other systems.
Suggested Citation
Clutter, Justace R, George Khoury, and Laura Freeman. Design of Experiments for In-Lab Operational Testing of the AN/BQQ-10 Submarine Sonar System. IDA Document NS D-5486. Alexandria, VA: Institute for Defense Analyses, 2014.