Supporting Statement B

SUPPORTING STATEMENT – PART B

DoD-wide Data Collection and Analysis for Department of Defense Data Collection in Support of the Independent Review Commission on Sexual Assault Recommendations (OMB Control Number 0704-0644)

Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The respondent universe for the pilot evaluation study are Soldiers (E1 to E4) at Army Garrison Ft. Leonard Wood; Army Drill Sergeants (E5 to E7) at Ft. Leonard Wood; and Army Performance Experts who will serve as coaches for the Coaching for Resilience (CFR) pilot. This is a new data collection and effort.

The evaluation will use a quasi-experimental design (QED). QEDs involve statistically matching intervention participants at the time of analysis (to similar individuals who do not receive the program). The QED will produce methodologically rigorous evidence of the effects of CFR and TILT, implemented as a pilot, on key outcomes at the Soldier level, yielding the necessary information for the United States Army leadership to make informed decisions and policies regarding the implementation of CFR and TILT.

Assignment to each condition will be at the company level. Ft. Leonard Wood leadership will select two incoming One Station Unit Training (OSUT) companies during “Phase A” of the evaluation – one will be assigned to the CFR program (Soldiers in this company will receive individual coaching during Advanced Individual Training [AIT]) and one to the comparison condition (Soldiers in this company will not receive coaching). During “Phase B,” Drill Sergeants at Ft. Leonard Wood will take the TILT program and be trained in trauma-informed approaches to leadership. In “Phase C,” Ft. Leonard Wood leadership will select two additional incoming OSUT companies of Soldiers; as in Phase A, one company will be assigned to receive coaching and the other will not receive coaching. However, in Phase C, both companies will have Drill Sergeants who have received TILT. This model will result in a four-cell evaluation design, allowing for the testing of the impact of CFR alone, of TILT alone, as well as of the combination of CFR and TILT on Soldier outcomes, compared to the same outcomes among Soldiers exposed to neither CFR nor Drill Sergeants who have completed TILT.

The evaluation will enroll 720 Soldiers (E1 to E4) as participants. Response rates and data quality will be monitored throughout the data collection period to ensure the evaluation meets the target sample sizes. We anticipate a 25% retention rate at the 3-month post-coaching follow-up, for a total of 180 Soldiers with completed baseline and follow-up surveys. Participants who receive coaching will be asked to complete a feedback form during the final coaching session. It is anticipated that 75% of the 360 Soldiers in the intervention condition (about 270 Soldiers) will complete a participant feedback form. Thirty-six Soldiers will participate in interviews three-months post-coaching. Interviews with 12 Performance Experts who served as coaches will also be conducted.

The evaluation will also aim to enroll 180 Drill Sergeant to participate in the TILT evaluation retrospective pre-post survey. Twelve Drill Sergeants will also participate in interviews.

	Baseline Surveys	Follow-Up Surveys	Training Feedback Forms	Qualitative Interviews
Coached Soldiers	N~360	N~360	N~360	N~36
Non-Coached Soldiers	N~360	N~360	N/A	N/A
Drill Sergeants	N~180	N/A	N/A	N~12
PEs (Coaches)	N/A	N/A	N/A	N~12

TABLE 1. SAMPLE SIZE ESTIMATES FOR DATA COLLECTION

2. Procedures for the Collection of Information

1. Statistical methodologies for stratification and sample selection.

The CFR evaluation design involves selection into treatment at the company level. To account for the multi-level design, data collected from participating Soldiers in Phase A companies will reflect CFR coaching vs. no coaching with both companies lead by Drill Sergeants who have been trained as usual. In comparison, data collection from Soldiers in Phase C will reflect two additional companies (CFR coaching vs. no coaching) under leadership by Drill Sergeants who have all completed TILT (Phase B). Propensity score weighting will be used to balance the four Soldier samples (no intervention, CFR only, TILT only, CFR plus TILT) for the purpose of outcome analyses. Qualitative interviews will be conducted with a random selection of 36 Soldiers (stratified by sex) who had completed the follow-up CFR survey and expressed interest in participating in interviews. Up to 12 Army Performance Experts and a random sample of 12 Drill Sergeants (who agree to be contacted again after they complete the TILT evaluation survey) will be invited to participate in interviews as well.

2. Estimation procedures.

No sample estimation procedures are included in this program evaluation design. That is, all eligible cases during the study period will be entered into the study if the participants agree to participate in the evaluation.

3. Degree of accuracy needed for the Purpose discussed in the justification.

To ensure the credibility of the evaluation findings, NORC has conducted statistical power calculations to determine the credibility of detecting a significant program effect at specific sample sizes. NORC calculated power estimates for Cohen’s f using G*Power 3.1 software, assuming 80% power and an alpha level of 0.05. Cohen’s f is the standardized effect measure for ANOVA and regression models, where 0.10 is a small effect, 0.25 is medium, and 0.40 is large. NORC proposes recruiting 720 soldiers to the evaluation at baseline, with an even spread over the four evaluation groups (CFR and TILT, CFR with no TILT, no CFR and TILT, and no CFR no TILT). NORC expects to retain 40% of the baseline sample at follow-up, and we will be able to detect small effects (f=0.15) with the minimum expected analytic sample. To the extent that a greater proportion of the baseline sample is retained, NORC will be able to detect smaller effects. Power for subgroup analyses (such as by sex) will vary depending on the final sample size and subgroup of interest.

4. Unusual problems requiring specialized sampling procedures.

NORC does not anticipate a need for specialized sampling procedures given the evaluation design.

5. Use of periodic or cyclical data collections to reduce respondent burden.

As DoD SAPRO and the U.S. Army are predominantly interested in the impact of CFR and TILT on Soldier outcomes, two data points from each Soldier are necessary to identify change over time. Surveys will be administered at baseline and 3 months post-coaching (about 4 months post baseline). Without a follow-up data collection, NORC will be unable to assess the Soldier outcomes associated with the CFR training program and the experience of AIT under the leadership of TILT-trained Drill Sergeants. In other words, given the need to identify change over time, two surveys (baseline/follow-up) are the least number of surveys possible to evaluate the impact of CFR and TILT on Soldiers. The evaluation of TILT’s impact on Drill Sergeants is limited to one retrospective pre-post (RPP) survey, capturing their knowledge and attitudes toward the TILT content immediately after completing the training.

3. Maximization of Response Rates, Non-response, and Reliability

There are several factors of the research design that will contribute to a strong response rate for this data collection and thus the overall rigor of the CFR/TILT pilot evaluation effort. The evaluation team has been engaged with Army personnel at Ft. Leonard Wood, the Center for Initial Military Training, and the Comprehensive Soldier and Family Fitness over the past two years to understand the context of the programs and implementation considerations. The recruitment protocols and survey language have been carefully reviewed with Army and DoD SAPRO staff, and internal NORC experts, and with a small group of Drill Sergeants to assure that the TILT survey language is understandable, relatable, and acceptable to Drill Sergeants.

The baseline CFR survey will be collected consistent with standard assessments of all new recruits during BCT at Ft. Leonard Wood. Those Soldiers who agree to participate in the pilot evaluation at the time of the baseline survey will provide additional information pertinent to the evaluation design. Efforts will be made to collect Soldiers’ personal and military email addresses as well as their cell phone numbers to facilitate follow-up data collection. At the 3-month follow-up period, the Soldiers will no longer be located at Ft. Leonard Wood and will be contacted to participate in the online follow-up survey. NORC has tested the online survey link on different web browser platforms using different NORC laptops, personal computing devices, and personal mobile devices to ensure participants at follow-up will also encounter a user-friendly design for Soldiers to complete when they have access to the internet.

NORC has developed the data collection protocols to be consistent with best practices of survey design and implementation. NORC has worked with Ft. Leonard Wood to ensure that Soldiers are briefed on the importance of the evaluation, including participating in follow-up surveys.

Respondents will also be offered modest financial incentives for participation following established practices in the field of survey research for reliable and valid data collection. There are incentives associated with each component of the evaluation to bolster participation. While attrition at the follow-up stage is a common phenomenon in survey data collection, Soldiers are not regularly offered tokens of appreciation for survey participation, and thus, the current data collection is expected to achieve a response rate that will support the planned analyses.

To address unit-level missing data on the surveys, NORC will compare responders with non-responders at baseline and follow-up with basic aggregated demographic information and other information (e.g., demographics, years of service, etc.) available on all Soldiers and Drill Sergeants who agreed to participate in the evaluation. Any statistically significant differences by demographic or background variables will be addressed as covariates in later outcome models.

To address item-level missing data (i.e., if respondents skip some questions), NORC will first assess the amount of missing data and whether missingness is at random. If there is little missing data (e.g., under 5% - 10%), NORC will assess if it is statistically appropriate to use listwise deletion of these cases. If necessary, NORC will compare the impact of employing various methods to handle missing data (e.g., Full Information Maximum Likelihood and Multiple Imputation procedures)¹ to fill in missing values for the surveys that are only partially completed. NORC is experienced in various imputation methods (e.g., nearest neighbor “hot deck”), including multiple imputation (e.g., Rubin’s multiple imputation strategy² to replace each missing value with a set of plausible values that represent the uncertainty about the correct value).

4. Tests of Procedures

NORC has collected feedback from current Army Drill Sergeants on the TILT survey questions to ensure they were clearly worded and relevant to the Army Drill Sergeant position. In addition, NORC has engaged a team of stakeholders in an Evaluation Working Group (EWG) to review all study instruments and protocols. EWG members included representation from DoD SAPRO and U.S. Army.

5. Statistical Consultation and Information Analysis

Provide names and telephone number of individual(s) consulted on statistical aspects of the design.

• Elizabeth Mumford: 301-634-9435

• Bruce Taylor: 301-634-9512

• Cynthia Simko: 312-759-4066

Provide name and organization of person(s) who will collect and analyze the collected information.

• Elizabeth Mumford (NORC)

• Bruce Taylor (NORC)

• Cynthia Simko (NORC)

1 Melissa J. Azur et al., “Multiple Imputation by Chained Equations: What Is It and How Does It Work?,” International Journal of Methods in Psychiatric Research 20, no. 1 (2011): 40–49, https://doi.org/10.1002/mpr.329.

2 D.B. Rubin, Multiple Imputation for Nonresponse in Surveys (New York: John Wiley & Sons, 1987); D. B. Rubin, “Inference and Missing Data,” Biometrika 63, no. 3 (1976): 581–90.