Data Management on Test Science Research Document Library https://research.testscience.org/areas/data-management/ Recent content in Data Management on Test Science Research Document Library Hugo -- 0.129.0 en-us Institute for Defense Analyses Sun, 01 Jan 2023 00:00:00 +0000 Data Principles for Operational and Live-Fire Testing https://research.testscience.org/post/2023-data-principles-for-operational-and-live-fire-testing/ Sun, 01 Jan 2023 00:00:00 +0000 https://research.testscience.org/post/2023-data-principles-for-operational-and-live-fire-testing/ Many DOD systems undergo operational testing, which is a field test involving realistic combat conditions. Data, analysis, and reporting are the fundamental outcomes of operational test, which support leadership decisions. The importance of data standardization and interoperability is widely recognized by leadership in DoD, however, there are no generally recognized standards for the management and handling of data (format, pedigree, architecture, transferability, etc.) in the DOD. In this presentation, I will review a set of data principles that we believe DOD should adopt to improve how it manages test data. Many DOD systems undergo operational testing, which is a field test involving realistic combat conditions. Data, analysis, and reporting are the fundamental outcomes of operational test, which support leadership decisions. The importance of data standardization and interoperability is widely recognized by leadership in DoD, however, there are no generally recognized standards for the management and handling of data (format, pedigree, architecture, transferability, etc.) in the DOD. In this presentation, I will review a set of data principles that we believe DOD should adopt to improve how it manages test data. I will explain the current state of data management, each of the data principles, why the DOD should adopt them, and some of the difficulties of improving data handling.

Suggested Citation

Medlin, Rebecca, John Haman, and Matthew Avery. Data Principles for Operational and Live-Fire Testing. IDA Document NS - 1038201. Alexandria, VA: Institute for Defense Analyses, 2023.

Slides:

]]> Introduction to Git https://research.testscience.org/post/2022-introduction-to-git/ Sat, 01 Jan 2022 00:00:00 +0000 https://research.testscience.org/post/2022-introduction-to-git/ Version control software manages, archives, and (optionally) distributes different versions of files. The most popular program for version control is Git, which serves as the backbone of websites such as Github, Bitbucket, and others. In this mini- tutorial, we will introduce basics of version control in general, and Git in particular. We explain what role Git plays in a reproducible research context. The goal of the course is to get participants started using Git.

Version control software manages, archives, and (optionally) distributes different versions of files. The most popular program for version control is Git, which serves as the backbone of websites such as Github, Bitbucket, and others. In this mini- tutorial, we will introduce basics of version control in general, and Git in particular. We explain what role Git plays in a reproducible research context. The goal of the course is to get participants started using Git. We will create and clone repositories, add and track files in a repository, and manage Git branches. We also discuss a few Git best practices.

Suggested Citation

Miller, Curtis G, and John T Haman. Introduction to Git. IDA Document NS D-33021. Alexandria, VA: Institute for Defense Analyses, 2022.

Slides:

]]> Visualizing Data- I Don't Remember that Memo, but I Do Remember that Graph https://research.testscience.org/post/2020-visualizing-data-i-don-t-remember-that-memo-but-i-do-remember-that-graph/ Wed, 01 Jan 2020 00:00:00 +0000 https://research.testscience.org/post/2020-visualizing-data-i-don-t-remember-that-memo-but-i-do-remember-that-graph/ IDA analysts strive to communicate clearly and effectively. Good data visualizations can enhance reports by making the conclusions easier to understand and more memorable. The goal of this seminar is to help you avoid settling for factory defaults and instead present your conclusions through visually appealing and understandable charts. Topics covered include choosing the right level of detail, guidelines for different types of graphical elements (titles, legends, annotations, etc.), selecting the right variable encodings (color, plot symbol, etc. IDA analysts strive to communicate clearly and effectively. Good data visualizations can enhance reports by making the conclusions easier to understand and more memorable. The goal of this seminar is to help you avoid settling for factory defaults and instead present your conclusions through visually appealing and understandable charts. Topics covered include choosing the right level of detail, guidelines for different types of graphical elements (titles, legends, annotations, etc.), selecting the right variable encodings (color, plot symbol, etc.), advice on practical implementations, and determining whether to include a chart at all. Most of the time, there’s no single “right” answer, so this presentation will include audience discussion to examine the trade-offs associated with different options.

Suggested Citation

Avery, Matthew, Heather Wojton, Andrew Flack, and Brian Vickers. Visualizing Data: I Don’t Remember That Memo, but I Do Remember That Graph. Alexandria, VA: Institute for Defense Analyses, 2020.

Slides:

Poster:

]]> Managing T&E Data to Encourage Reuse https://research.testscience.org/post/2019-managing-t-e-data-to-encourage-reuse/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-managing-t-e-data-to-encourage-reuse/ Reusing Test and Evaluation (T&E) datasets multiple times at different points throughout a program’s lifecycle is one way to realize their full value. Data management plays an important role in enabling - and even encouraging – this practice. Although Department-level policy on data management is supportive of reuse and consistent with best practices from industry and academia, the documents that shape the day-to-day activities of T&E practitioners are much less so. Reusing Test and Evaluation (T&E) datasets multiple times at different points throughout a program’s lifecycle is one way to realize their full value. Data management plays an important role in enabling - and even encouraging – this practice. Although Department-level policy on data management is supportive of reuse and consistent with best practices from industry and academia, the documents that shape the day-to-day activities of T&E practitioners are much less so. As a result, reuse of T&E datasets does not occur on a consistent basis or in a formalized way. To fill this apparent gap, this article expands upon four best practices – addressed in different ways in Service-specific T&E policies – that can increase the reuse of T&E datasets.

Suggested Citation

Medlin, Rebecca, and Andrew Flack. “Managing T&E Data to Encourage Reuse.” The ITEA Journal of Test and Evaluation of Test and Evaluation, 2019.

Paper:

]]> Reproducible Research Mini-Tutorial https://research.testscience.org/post/2019-reproducible-research-mini-tutorial/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-reproducible-research-mini-tutorial/ Analyses are reproducible if the same methods applied to the same data produce identical results when run again by another researcher (or you in the future). Reproducible analyses are transparent and easy for reviewers to verify, as results and figures can be traced directly to the data and methods that produced them. There are also direct benefits to the researcher. Real-world analysis workflows inevitably require changes to incorporate new or additional data, or to address feedback from collaborators, reviewers, or sponsors.

Analyses are reproducible if the same methods applied to the same data produce identical results when run again by another researcher (or you in the future). Reproducible analyses are transparent and easy for reviewers to verify, as results and figures can be traced directly to the data and methods that produced them. There are also direct benefits to the researcher. Real-world analysis workflows inevitably require changes to incorporate new or additional data, or to address feedback from collaborators, reviewers, or sponsors.

Suggested Citation

Wojton, Heather, Andrew Flack, John Haman, and Kevin Kirshenbaum. Reproducible Research Mini-Tutorial. IDA Document NS D-10581. Alexandria, VA: Institute for Defense Analyses, 2019.

Slides:

]]> Survey Testing Automation Tool (STAT) https://research.testscience.org/post/2019-survey-testing-automation-tool-stat/ Tue, 01 Jan 2019 00:00:00 +0000 https://research.testscience.org/post/2019-survey-testing-automation-tool-stat/ In operational testing, survey administration is typically a manual, paper-driven process. We developed a web-based tool called Survey Testing Automation Tool (STAT), which integrates and automates survey construction, administration, and analysis procedures. STAT introduces a standardized approach to the construction of surveys and includes capabilities for survey management, survey planning, and form generation. Suggested Citation Finnegan, Gary M, Kelly Tran, Tara A McGovern, and William R Whitledge. Survey Testing Automation Tool (STAT). In operational testing, survey administration is typically a manual, paper-driven process. We developed a web-based tool called Survey Testing Automation Tool (STAT), which integrates and automates survey construction, administration, and analysis procedures. STAT introduces a standardized approach to the construction of surveys and includes capabilities for survey management, survey planning, and form generation.

Suggested Citation

Finnegan, Gary M, Kelly Tran, Tara A McGovern, and William R Whitledge. Survey Testing Automation Tool (STAT). IDA Document NS D-10566. Alexandria, VA: Institute for Defense Analyses, 2019.

Poster:

]]> A Groundswell for Test and Evaluation https://research.testscience.org/post/2018-a-groundswell-for-test-and-evaluation/ Mon, 01 Jan 2018 00:00:00 +0000 https://research.testscience.org/post/2018-a-groundswell-for-test-and-evaluation/ The fundamental purpose of test and evaluation (T&E) in the Department of Defense (DOD) is to provide knowledge to answer critical questions that help decision makers manage the risk involved in developing, producing, operating, and sustaining systems and capabilities. At its core, T&E takes data and translates it into information for decision makers. Subject matter expertise of the platform and operational mission have always been critical components of developing defensible test and evaluation strategies. The fundamental purpose of test and evaluation (T&E) in the Department of Defense (DOD) is to provide knowledge to answer critical questions that help decision makers manage the risk involved in developing, producing, operating, and sustaining systems and capabilities. At its core, T&E takes data and translates it into information for decision makers. Subject matter expertise of the platform and operational mission have always been critical components of developing defensible test and evaluation strategies. Recent innovations in data science have improved our ability to collect, store, manage, transfer, process and visualize data. Additionally, advances in statistics and uncertainty quantification are revolutionizing how we think about predictions from all types of data. The ability to integrate system and scientific knowledge, coupled with advances in data science and statistics, will enable us to better target testing, make efficient use of resources, quantify risk, and lead to well informed decisions.

Suggested Citation

Freeman, Laura J. “A Groundswell for Test and Evaluation.” The ITEA Journal 39, no. 4 (December 2018).

Paper:

]]>