Statistical Power on Test Science Research Document Library https://research.testscience.org/keywords/statistical-power/ Recent content in Statistical Power on Test Science Research Document Library Hugo -- 0.129.0 en-us Institute for Defense Analyses Mon, 01 Jan 2024 00:00:00 +0000 Simulation Insights on Power Analysis with Binary Responses--from SNR Methods to 'skprJMP' https://research.testscience.org/post/2024-simulation-insights-on-power-analysis-with-binary-responses-from-snr-methods-to-skprjmp/ Mon, 01 Jan 2024 00:00:00 +0000 https://research.testscience.org/post/2024-simulation-insights-on-power-analysis-with-binary-responses-from-snr-methods-to-skprjmp/ Logistic regression is a commonly-used method for analyzing tests with probabilistic responses in the test community, yet calculating power for these tests has historically been challenging. This difficulty prompted the development of methods based on signal-to-noise ratio (SNR) approximations over the last decade, tailored to address the intricacies of logistic regression’s binary outcomes. However, advancements and improvements in statistical software and computational power have reduced the need for such approximate methods.

Logistic regression is a commonly-used method for analyzing tests with probabilistic responses in the test community, yet calculating power for these tests has historically been challenging. This difficulty prompted the development of methods based on signal-to-noise ratio (SNR) approximations over the last decade, tailored to address the intricacies of logistic regression’s binary outcomes. However, advancements and improvements in statistical software and computational power have reduced the need for such approximate methods. Our research presents a detailed simulation study that compares SNR-based power estimates with those derived from exact Monte Carlo simulations, highlighting the inadequacies of SNR approximations. To address these shortcomings, we will discuss improvements in the open-source R package “skpr” as well as present “skprJMP,” a new plug-in that offers more accurate and reliable power calculations for logistic regression analyses.

Suggested Citation

Atkins, Robert, Tyler Morgan-Wall, and Curtis Miller. “With Binary Responses–From SNR Methods to ‘skprJMP.’” Institute for Defense Analyses IDA Product ID 3002093 (April 2024).

Slides:

Paper:

Poster:

]]> Case Study on Applying Sequential Analyses in Operational Testing https://research.testscience.org/post/2022-case-study-on-applying-sequential-analyses-in-operational-testing/ Sat, 01 Jan 2022 00:00:00 +0000 https://research.testscience.org/post/2022-case-study-on-applying-sequential-analyses-in-operational-testing/ Sequential analysis concerns statistical evaluation in which the number, pattern, or composition of the data is not determined at the start of the investigation, but instead depends on the information acquired during the investigation. Although sequential analysis originated in ballistics testing for the Department of Defense (DoD)and it is widely used in other disciplines, it is underutilized in the DoD. Expanding the use of sequential analysis may save money and reduce test time.

Sequential analysis concerns statistical evaluation in which the number, pattern, or composition of the data is not determined at the start of the investigation, but instead depends on the information acquired during the investigation. Although sequential analysis originated in ballistics testing for the Department of Defense (DoD)and it is widely used in other disciplines, it is underutilized in the DoD. Expanding the use of sequential analysis may save money and reduce test time. In this paper, we introduce sequential analysis, describe its current and potential uses in operational test and evaluation (OT&E), and present a method for applying it to the test and evaluation of defense systems. We evaluate the proposed method by performing simulation studies and applying the method to a case study. Additionally, we discuss challenges to address for sequential analysis in OT&E. Lastly, while operational testing is the focus in this paper, the methodology presented is applicable to campaigns of experimentation and general testing across numerous disciplines.

Suggested Citation

Ahrens, Monica, Rebecca Medlin, Keyla Pagán-Rivera, and John W. Dennis. “Case Study on Applying Sequential Analyses in Operational Testing.” Quality Engineering 35, no. 3 (July 3, 2023): 534–45. https://doi.org/10.1080/08982112.2022.2146510.

Slides:

Paper:

]]> On Scoping a Test that Addresses the Wrong Objective https://research.testscience.org/post/2017-on-scoping-a-test-that-addresses-the-wrong-objective/ Sun, 01 Jan 2017 00:00:00 +0000 https://research.testscience.org/post/2017-on-scoping-a-test-that-addresses-the-wrong-objective/ Statistical literature refers to a type of error that is committed by giving the right answer to the wrong question. If a test design is adequately scoped to address an irrelevant objective, one could say that a Type III error occurs. In this paper, we focus on a specific Type III error that on some occasions test planners commit to reduce test size and resources. Suggested Citation Johnson, Thomas H., Rebecca M. Statistical literature refers to a type of error that is committed by giving the right answer to the wrong question. If a test design is adequately scoped to address an irrelevant objective, one could say that a Type III error occurs. In this paper, we focus on a specific Type III error that on some occasions test planners commit to reduce test size and resources.

Suggested Citation

Johnson, Thomas H., Rebecca M. Medlin, Laura J. Freeman, and James R. Simpson. “On Scoping a Test That Addresses the Wrong Objective.” Quality Engineering 31, no. 2 (April 3, 2019): 230–39. https://doi.org/10.1080/08982112.2018.1479035.

Paper:

]]>