Simulation on Test Science Research Document Library https://research.testscience.org/keywords/simulation/ Recent content in Simulation on Test Science Research Document Library Hugo -- 0.129.0 en-us Institute for Defense Analyses Sat, 01 Jan 2022 00:00:00 +0000 Metamodeling Techniques for Verification and Validation of Modeling and Simulation Data https://research.testscience.org/post/2022-metamodeling-techniques-for-verification-and-validation-of-modeling-and-simulation-data/ Sat, 01 Jan 2022 00:00:00 +0000 https://research.testscience.org/post/2022-metamodeling-techniques-for-verification-and-validation-of-modeling-and-simulation-data/ Modeling and simulation (M&S) outputs help the Director, Operational Test and Evaluation (DOT&E) assess the effectiveness, survivability, lethality, and suitability of systems. To use M&S outputs, DOT&E needs models and simulators to be sufficiently verified and validated. The purpose of this paper is to improve the state of verification and validation by recommending and demonstrating a set of statistical techniques—metamodels, also called statistical emulators—to the M&S community. The paper expands on DOT&E’s existing guidance about metamodel usage by creating methodological recommendations the M&S community could apply to its activities.

Modeling and simulation (M&S) outputs help the Director, Operational Test and Evaluation (DOT&E) assess the effectiveness, survivability, lethality, and suitability of systems. To use M&S outputs, DOT&E needs models and simulators to be sufficiently verified and validated. The purpose of this paper is to improve the state of verification and validation by recommending and demonstrating a set of statistical techniques—metamodels, also called statistical emulators—to the M&S community.

The paper expands on DOT&E’s existing guidance about metamodel usage by creating methodological recommendations the M&S community could apply to its activities. For a deterministic, discrete response variable, we recommend using a nearest neighbor or decision tree model. For a deterministic, continuous response variable, we recommend Gaussian process interpolation. For a stochastic response variable, we recommend a generalized additive model. We also present a set of techniques that testers can use to assess the adequacy of their metamodels. We conclude with a notional example that demonstrates the recommended techniques.

Suggested Citation

Haman, John T, and Curtis G Miller. Metamodeling Techniques for Verification and Validation of Modeling and Simulation Data. IDA Paper P-33230. Alexandria, VA: Institute for Defense Analyses, 2022.

Slides:

Paper:

]]> What Statisticians Should Do to Improve M&S Validation Studies https://research.testscience.org/post/2022-what-statisticians-should-do-to-improve-m-s-validation-studies/ Sat, 01 Jan 2022 00:00:00 +0000 https://research.testscience.org/post/2022-what-statisticians-should-do-to-improve-m-s-validation-studies/ It is often said that many research findings – from social sciences, medicine, economics, and other disciplines – are false. This fact is trumpeted in the media and by many statisticians. There are several reasons that false research is published, but to what extent should we be worried about them in defense testing and modeling and simulation? In this talk I will present several recommendations for actions that statisticians and data scientists can take to improve the quality of our validations and evaluations.

It is often said that many research findings – from social sciences, medicine, economics, and other disciplines – are false. This fact is trumpeted in the media and by many statisticians. There are several reasons that false research is published, but to what extent should we be worried about them in defense testing and modeling and simulation? In this talk I will present several recommendations for actions that statisticians and data scientists can take to improve the quality of our validations and evaluations.

Suggested Citation

Haman, John T. What Statisticians Should Do to Improve M&S Validation Studies. Alexandria, VA: Institute for Defense Analyses, 2022.

Slides:

]]>