Supporting Statement B

SUPPORTING STATEMENT – PART B

Title of Collection: Climate Transformation Task Force (CTTF) Evaluation at the Military Service Academies

Expected Fielding Dates: 02 September 2025 – 31 October 2026

B.  COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1.  Description of the Activity

The Department of Defense contracted NORC at the University of Chicago (NORC) to conduct a cross-site evaluation of the Climate Transformation Task Force (CTTF) activities at the United States Military Academy (USMA), United States Naval Academy (USNA), and United States Air Force Academy (USAFA). For this evaluation, NORC will conduct (1) two cross-sectional surveys with Company/Squadron commanders (“Survey”) and (2) up to 36 focus groups with faculty, staff, alumni, and parents across the MSAs (“Focus Groups”).

The respondent universe for the cross-sectional Surveys includes company and squadron commanders from the United States Military Academy (USMA), United States Navy Academy (USNA), and United States Air Force Academy (USAFA). This is a new data collection and evaluation effort. Eligible participants for the Surveys include all (TACs/TACs NCOs, COs/SELs, and AOCs/AMTs) during the time of survey fielding. NORC will enroll up to 504 company and squadron commanders through the two cross-sectional implementations of the survey.

The respondent universe for the Focus Groups includes faculty, staff, parents of current MSA students, and MSA alumni. This is a new data collection effort for these groups.

NORC will enroll up to 432 participants in the focus groups.

Table 1. Data Collection by Evaluation Group with recruitment ranges

Groups	Data Collection component	Recruitment range for data collection components.
Company and Squadron Commanders	1st cross-sectional survey 2nd cross-sectional survey	1st cross-sectional: n= 252 2nd cross-sectional: n=252
Faculty, staff, alumni, and parents	36 Focus groups	432 respondents

2.  Procedures for the Collection of Information

To answer the guiding evaluation questions, this CTTF evaluation will employ multiple methods and phases which are described below.

Company/Squadron Commander Surveys: NORC will conduct a cross-sectional survey at two time points with Company/Squadron Commanders. NORC developed the survey based on standardized learning objectives and outcomes expected to be associated with the Company/Squadron Commanders’ trainings. Once the survey instrument was drafted, NORC conducted three interviews with members of the eligible participant population to assess the appropriateness and relevancy of the draft language in the instrument before submitting it through the MSA clearance process. Pending approvals and reflecting MSA feedback on the best point in each Academy calendar, the survey will be fielded in fall 2025 and readministered in fall 2026.

The purpose of the Company/Squadron Commander Surveys is to track participant progress (in the aggregate, not at an individual level, given staff re-assignments) regarding the following learning objectives:

1. How Company/Squadron Commander apply key principles of primary prevention in a leadership role.

2. How Company/Squadron Commander analyze the environment and apply measures to prevent harmful behaviors and create a professional unit climate.

3. How Company/Squadron Commander apply procedures and leverage resources to respond appropriately when cadets/midshipmen experience or engage in harmful behaviors.

MSA Focus Groups: NORC will conduct up to 36 focus groups with faculty, staff, alumni, and parents (of current cadets/midshipmen). Each focus group will have 10-12 participants and last up to 90 minutes. The goal is to conduct focus groups with faculty and staff in person on campus and virtual focus groups with off-site populations of interest (i.e., current parents and MSA alumni). NORC will work with the MSAs to find an appropriate time to conduct focus groups in 2026 through early 2027.

The purpose of the focus groups is to answer the following two questions:

1. How do faculty, staff, alumni, and parents serve as facilitators and/or barriers within the MSA social ecology to implementing CTTF activities at the MSAs?

2. How can faculty, staff, alumni, and parents be leveraged to support the goals of the CTTF activities?

1. Statistical methodologies for stratification and sample selection:

The CTTF evaluation involves the identification of Company/Squadron commanders at the MSAs. NORC expects to collect 252 surveys from current Company/Squadron Command Team members at each cross-sectional implementation. Participants will be invited to complete the survey via an email invitation to their military email address with a link to the survey questionnaire. NORC will host the survey, and the survey will be accessible via a mobile or laptop device with an internet connection. The survey will be anonymous. Participants will navigate through the survey and indicate their answer choices on the survey questionnaire. Participants may skip any question they do not wish to answer. To maintain anonymity, NORC will conduct a disclosure risk assessment review to identify steps needed to limit re-identification and attribution risk prior to delivery of the data to DoD. Any results generated from the data presented in reports will meet minimum-cell suppression requirements to protect against participant identification.

Focus groups will be conducted by a convenience sample of faculty, staff, alumni, and parents who volunteer to participate. NORC expects to conduct up to 36 focus groups across USMA, USNA, and USAFA. NORC will travel to each MSA to conduct in-person focus groups with faculty and staff. Additionally, NORC will work with the Evaluation Working Group (EWG) to identify strategies (such as collaborating with the MSA alumni offices, placing ads in MSA magazines and/or MSA newsletter, social media, and identifying key partners) to help recruit alumni and parent focus group participants. Participants will be recruited via email and referrals. Emails and other recruitment materials will include a summary of the purpose of the focus group and NORC contact information. NORC will work with the MSAs to schedule the focus groups in late 2026 through early 2027. Focus groups will be confidential, no audio or video recordings will take place. Participants will also be asked to keep names of participants confidential, i.e., not disclose names to those who did not participate in their focus group. Notes and analytic files will be saved in the project folder on NORC’s secure servers, which is only accessible to those who have signed a non-disclosure agreement for this project. Any files shared with the DoD will be de-identified and shared via a secure platform (e.g., DoD SAFE)

2. Estimation procedures

Not applicable.

3. Degree of accuracy needed for the Purpose discussed in the justification

To ensure the credibility of the evaluation findings, NORC has conducted statistical power calculations to determine the credibility of detecting significant effects at specific sample sizes.

We will recruit all Company/Squadron commanders aiming for 252 responses to each of the two cross-sectional surveys. Response rates and data quality will be monitored throughout the data collection period to ensure the target sample size is reached. With a 50% response rate at each wave, we could detect a significant difference of d=0.35 between subgroups (small effect size). NORC will work closely with DoD, USMA, USNA, and USAFA to encourage Company/Squadron commanders to participate in the survey. Participation rates and data quality will be monitored throughout the data collection period to ensure the collection meets the target sample size.

We will attempt to enroll 432 faculty, staff, alumni, and parents in the evaluation focus groups. The goal is to collect enough data from the focus groups to reach “saturation,” meaning no new themes or insights from additional participants emerge, indicating a sufficient understanding of the evaluation topics¹. Data quality will be monitored throughout the data collection period to ensure the collection meets saturation on the topics covered through the moderation guides.

4. Unusual problems requiring specialized sampling procedures

We do not anticipate a need for specialized sampling procedures given the study design.

5. Use of periodic or cyclical data collections to reduce respondent burden.

The administration of the cross-sectional survey to Company/Squadron Command team members at two data points is necessary to assess change in at the aggregate response in this population at each MSA, accommodating the reality of staff reassignments/new team members. Surveys will be administered Fall 2025 and Fall 2026. Given the need to identify change over time, at least two surveys are required.

According to the MSAs' scheduling requests, the focus groups will be administered at one-time point and scheduled to collect input from faculty, staff, alumni, and parents from summer 2026 to early 2027.

3.  Maximization of Response Rates, Non-response, and Reliability

There are several components of the CTTF evaluation design that will contribute to a strong response rate for this data collection. NORC has worked in collaboration with an Evaluation Working Group (EWG) that consists of evaluation and program experts from the USMA, USNA, USAFA, and the Department of Defense.

Addressing low response rates proactively will ensure that NORC produces a program evaluation that is trustworthy and useful in showing whether the CTTF MSA efforts have had their intended impact. NORC, in partnership with the MSAs, will work to encourage survey participation through reminder emails (to a military address) and digital communication encouragement from the MSA. NORC will encourage participation in the in-person focus groups through advance planning and as needed, scheduling a make-up session for recruited individuals who have unavoidable conflicts. NORC will provide reminder emails and technical support as needed to ensure adequate participation in the virtual focus groups. NORC may also use endorsement emails from MSA leadership to encourage participation, an approach that has worked in prior military studies.

To address missing data on survey responses, responders will be compared with basic aggregated demographic information available on all eligible personnel during the two survey periods. We will ask MSAs to provide aggregate demographic data to compare with respondent aggregated demographic information. To address item-level missing data (i.e., if respondents skip some questions), NORC will first assess the amount of missing data and whether missingness is random. If there is little missing data (e.g., under 10%), NORC will assess if using listwise deletion of these cases is statistically appropriate. If necessary, NORC will employ imputation methods to fill in missing values.

To minimize missing data for the focus groups, NORC will work with the MSAs to reschedule groups as needed to ensure representation from key groups of interest for the focus groups. Additionally, during in-person data collection, NORC will hold a daily open focus group to allow individuals who could not make their focus group time to participate in the evaluation.

4.  Tests of Procedures

The EWG and MSA leaders have reviewed survey language in addition to several (n=3) current Company and Squadron Commanders via one-on-one discussions. The Company and Squadron Commanders provided feedback to ensure the survey language was understandable and acceptable to the target population of Company and Squadron Commanders at the MSAs. NORC will also internally beta-test the survey instrument on the online platform Voxco to ensure that all recruited populations at the MSAs can access and complete the survey properly.

The MSA CTTF EWG reviewed the focus group interview guides to ensure they were appropriate for the population.

5.  Statistical Consultation and Information Analysis

a. Provide names and telephone numbers of individual(s) consulted on statistical aspects of the design.

• Elizabeth Mumford: 301-634-9435

• Cynthia Simko: 312-759-4066

• Bruce Taylor: 301-634-9512

2. Provide name and organization of person(s) who will actually collect and analyze the collected information.

• Elizabeth Mumford (NORC)

• Cynthia Simko (NORC)

• Bruce Taylor (NORC)

1 Hennick M, Kaiser B, Weber M. What Influences Saturation? Estimating Sample Sizes in Focus Group Research. Qual Health Res. Published online 29 2019. doi: 10.1177/1049732318821692.

3