Supporting Statement Part B

SUPPORTING STATEMENT – PART B

B.  COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Description of the Activity

The ultimate goal of this evaluation is to assess the extent to which the hiring of about 2000 new IPP personnel led to the implementation of quality prevention services for Service members. This evaluation will address six aims:

1. Assess how the DoD components’ Integrated Primary Prevention (IPP) policies and plans are aligned with the requirements set forth by the Department of Defense (DoD);

2. Assess the hiring progress (e.g., number and type of personnel hired) and qualifications of the Integrated Primary Prevention Workforce (IPPW);

3. Describe the structure and functioning of prevention infrastructure and whether the IPPW perceives having sufficient infrastructure support;

4. Assess the elements of leader support for the conduct of prevention (e.g., understanding of IPP, belief that IPP is important and works well), and assess the level of leader support perceived by the IPPW;

5. Assess the quality and comprehensiveness of the Comprehensive Integrated Primary Prevention (CIPP) Plans, what aspects of the plans are implemented as intended, and the facilitators and barriers to their implementation; and

6. Describe how the IPP workforce is building and maintaining prevention teams at the strategic, operational, and tactical levels and how these prevention teams are connected to leaders at each of these levels.

To answer these research questions, this evaluation will use multiple data sources specifically collected for this evaluation, described below.

Leader and Supervisor Interviews. RAND will conduct qualitative interviews to assess strategic and operational IPP personnel and DoD component leaders’ perceptions of progress hiring and integrating tactical-level IPP personnel. These interviews will ask questions about barriers to hiring and deploying tactical-level IPP personnel and their current and anticipated impacts. RAND will interview ~5 to 10 strategic and operational IPP personnel and DoD component leaders per service in Quarter 4 (Q4) Fiscal Year (FY) 24/Q1 FY25 and Q4 FY26/Q1 FY27. The end result of the Leader and Supervisor Interviews is that RAND will have an understanding of the barriers and facilitators that exist at each level of the hiring and deployment of IPPW. If IPPW face a great deal of barriers, that could help to explain the outcomes achieved by the entire initiative.

IPP Personnel Survey. Annually during Q1 of FY25, FY26, and FY27, we will administer an online survey to assess knowledge and experience of all IPP personnel (~n=2000) across strategic, operational, and tactical levels, perceptions of whether a supportive climate for IPP exists, and perceptions of barriers and facilitators to IPP. We are asking IPPW prevention knowledge and experience, climate, and barriers/facilitators questions because for this initiative to succeed, these personnel must have the knowledge and skills to perform their job. Further, the climate with which they operate must be receptive and supportive to prevention work, and IPPW must be able to navigate barriers and enhance facilitators. The end result of the IPP Personnel Survey is that RAND will have an understanding of the level of knowledge and experiences of IPP personnel, the degree of the support they received, and the most common barriers and facilitators to IPP. This data will be used to contextualize the ultimate outcomes of the evaluation—the level of knowledge and experience of IPP personnel, and the level of support and barriers/facilitators to IPP, could help explain why or why not DoD was able to implement high quality IPP activities.

Given that hiring will be rolling across the evaluation period, IPP personnel will take the survey a different number of times depending on when they are hired. We estimate the sample to cumulatively total 2,000 IPP personnel by the end of the evaluation period, with increasing numbers each year. This survey is intended to be a census—i.e., we are targeting all IPPW across all of DoD services and levels, not a sample. Based on previous RAND military projects, we anticipate the response rate to be ~65%.

Integrated Primary Prevention Activity Tracker (I-PACT). Starting in Q4 FY24 and occurring every six months thereafter through Q2 FY27, RAND will collect, via online, secure data collection portal developed by RAND, information about the IPP activities being implemented by all IPP personnel (i.e., I-PACT). The I-PACT will collect information on:

• Types of IPP activities

• Needs assessment and planning efforts, including collaboration and stakeholder engagement

• Target populations and participation

• Domains, behaviors, risk and protective factors targeted

• Evidence-basis, best practice adherence, and adaptations

• Evaluation efforts

This data collection is intended to be a census, including all locations where all IPP personnel work. The end result will provide us information about the comprehensiveness and quality of IPP activities.

RAND will contract with 3C Institute (3C; www.3CISD.com), a North Carolina S-Corporation, to build the I-PACT. 3C has extensive research and development (R&D) experience creating innovative technologies, including custom interactive websites, dynamic e-learning platforms, online and mobile applications for information sharing, data collection and analysis systems, and online distance networking and collaboration platforms for a wide array of purposes and audiences (see www.3Ctechnologies.com). They are leaders in the field of game-based platforms for intelligent assessment, adaptive learning, and personalized feedback. 3C’s technologies have been repeatedly tested to ensure maximum usability, security, and effectiveness with high fault tolerance for diverse users in diverse settings. Their products target areas in line with our subject matter expertise in social-emotional learning; mental health intervention/prevention, and implementation science. They have also extended our R&D experience into other healthcare realms through multiple external collaborations (e.g., chronic kidney disease with Dr. Maria Ferris of UNC-Chapel Hill). 3C has successfully commercialized over 50 evidence-based health assessment and intervention tools (see www.3Cmarketplace.com/catalog/products). Because of these efforts, 3C’s health promotion tools are currently being used by community agencies to help thousands of people across the U.S. and in 12 countries. Their interventions have won multiple awards, including the Substance Abuse and Mental Health Services Administration Science and Service Award for Mental Health Promotion.

Case Studies. The Case Studies represent an opportunity to study in-depth the process of integrating IPP personnel, the facilitators/barriers to doing so, and perceived impacts over time. Although focused on the tactical level, the Case Studies will yield important information about how the tactical level interfaces with the operational and strategic levels. Across 24 military organizations (12 with IPPW, 12 without), this approach will use virtual discussions with the same military organizations over two time points, and a site visit to observe an IPP activity. This method will provide a bottom-up perspective on the hiring, integration, and effectiveness of the IPP personnel, as well as critical context to the overall evaluation findings, including a clearer understanding of barriers and facilitators that IPP personnel face.

This method will rely on in-depth Case Studies, recruiting a sample of military organizations that have hired IPPW (6 high scoring military organizations or regions and 6 low scoring military organizations or regions, identified based on our data from Tracking IPPW billets and CIPP Plan Review) and a sample that have not hired IPPW (12 military organizations or regions). We anticipate stratifying the sample by DoD component to ensure representation from each of the Military Departments, Services, and National Guard Bureau (e.g., Army, Air Force, Navy, Marine Corps, Space Force).

Data collection for this method will take place virtually through discussions on Zoom or Teams with key stakeholders at each selected military organization (or region) in Q4 FY24 through Q2 FY25 and again in Q4 FY26 through Q2 FY27 of the evaluation. To the extent possible, there will also be a brief (i.e., one-to-two day) site visit coordinated with VPC, likely to take place between Q3 FY25 and Q3 FY26 of the evaluation, to observe an IPP activity. Questions during the virtual discussions will be about barriers and facilitators to hiring IPP personnel and implementing IPP activities. We will also ask questions related to organizational support for IPP. To supplement the data collected during the discussions, we will also visit each case study organization once during the course of the evaluation to observe a prevention activity being implemented and record our observations of key elements of the activity using a standardized form (e.g., type of activity, format, research-based). The end result of this data collection will be a qualitative analysis of the barriers and facilitators that IPP personnel face in their work as well as a quantitative rating of the data collected during discussions according to the Prevention Evaluation Framework, a measure RAND developed in collaboration with DoD that specifically assesses various markers of high quality IPP in military settings. Prevention activity observations will also provide information about the nature of activities, level of engagement of IPPW and participants, and markers of IPP quality.

Resume Review. Using the required IPP workforce skills and experiences from DoD components’ PDs and the DoD’s Prevention Workforce Model, RAND will rate the alignment of all IPPW resumes with these requirements.

RAND will collect resumes by asking IPPW taking the IPP Personnel Survey to share their resumes using a secure file transfer platform (e.g., Kiteworks). Each IPPW will only be asked to share their resume one time, most likely the first time they complete the IPP Personnel Survey. Once they share their resume, they will not be asked to share it again. To inform the analysis, each PD will be broken down into a list of requirements which will then be used to develop a data abstraction form for use with the resumes for IPPW hired for that PD. The skills and experiences related to each PD requirement will be abstracted from each resume submitted under the associated PD. Then each resume will be given a score as fully, partially or not at all meeting the PD requirements. We will repeat this process with requirements based on DoD component instructions and guidance. The end result of this analysis will be a Score by position level for IPPW by DoD component describing whether the actual qualifications of the IPPW they hired (as described on their resumes) are fully, partially, or not at all aligned with what is required by the DoD component and DoD policy/guidance.

2. Procedures for the Collection of Information

Leader and Supervisor Interviews. To identify who we will interview, we will start with IPPW at the strategic- and operational-levels to ask who their direct supervisors are until we get to a non-IPPW personnel and interview that person. We will refer to this non-IPP personnel as a DoD component leader. We will also interview members of the Prevention Collaboration Forum, a governance body to oversee the implementation of the integrated prevention policy (DODI 6400.09, 6400.11). We will conduct semi-structured interviews in two waves with these DoD component leaders and strategic- and operational-level IPPW to capture progress and barriers: early in the initiative (Q4 FY24/Q1 FY25) and after the initiative has matured (Q4 FY26/Q1 FY27). In the first administration, we will sample through ‘snowball’ referrals—i.e., using DoD stakeholders to guide us to the most appropriate individuals with whom to interview. We will ask the sponsor for points of contact in Office of the Under Secretary of Defense for Personnel and Readiness, the DoD components, and potentially the Joint Staff. Once identified, we will ask the following as part of the snowball sampling:

• Who in your organization will have responsibility for the hiring, supervision, and management of IPPW? Who will the IPPW report to?

• Who in your organization will have oversight for policy and resource allocation decision-making for the IPP personnel?

We will interview personnel across the DoD components. In each administration, we anticipate interviewing about 5 to 10 personnel (to include both uniformed and civilian, as relevant) across DoD components (Table 1), depending on the snowball sampling.

Table 1. Proposed DoD Components

Proposed Organizations for DoD Component Leaders

U.S. Air Force: Active HQ; Reserve HQ; National Guard HQ

U.S. Space Force: Active HQ

U.S. Army: Active HQ; Reserve HQ; National Guard HQ

U.S. Navy: Active HQ; Reserve HQ

U.S. Marine Corps: Active HQ; Reserve HQ

U.S. National Guard Bureau

*Contingent upon where the IPPW are placed (e.g., whether any are assigned to reserve installations)

It is also anticipated that we will interview leaders at OSD Personnel and Readiness organizations with a role in recruiting or coordinating with the IPP personnel and monitoring outcomes of prevention activities, as well as members of the Prevention Collaboration Forum. In the second administration, we will follow up with the same personnel from the first administration, or different ones if the personnel have changed positions or position responsibilities have changed. All interviews will be conducted using Zoom.gov.

Once we identify who we want to interview, we will obtain their email addresses and phone numbers from our sponsor. We will send up to 3 emails and make up to two phone calls to obtain participation. The email will contain a description of the project and its importance, including how the project was recommended by the Independent Review Commission on Sexual Assault in the Military. Given the saliency of the project, we estimate that there will be a high response rate, ~80%.

IPP Personnel Survey. RAND will obtain email addresses and phone numbers of all IPP personnel from our service points of contact or from our sponsor. The survey will be administered, via online platform (e.g., Confirmit), to all IPP personnel annually during Q1 of FY25, FY26, and FY27 of the evaluation. We will send IPP personnel an email containing a link to an online survey. The email will contain a description of the project and its importance, including how the project was recommended by the Independent Review Commission on Sexual Assault in the Military. The email will also mention that the respondent will receive a $40.00 gift card upon completion of the survey. We will email the respondents up to three times and call up to two times to attempt to obtain a response from each IPP personnel (either yes or no). This data collection is intended to be a census of all IPP personnel, not a sample.

We will offer $40 Amazon Gift Cards to each IPP personnel who completes a IPP Personnel Survey each year. These individuals will receive an email with a link to their Gift Card. There will be no requirements about the level of completion for each data collection. Once the survey is submitted online, the individual will be eligible for the Gift Card. These payments are being offered to compensate the participating individuals for their time.

Integrated Primary Prevention Activity Tracker. Starting in Q4 FY24 and occurring every six months thereafter through Q2 FY27, an email will be sent to all IPP personnel with a link to an online portal (i.e., the I-PACT) that RAND will create. RAND will send up to two follow up emails in order enhance response rate. This data collection is intended to be a census of all IPP personnel, not a sample. Based on previous RAND military projects, we anticipate the response rate to be ~70%.

Case Studies. Our selection of sites will involve recruiting a sample of military organizations that have hired IPPW (6 high scoring military organizations or regions and 6 low scoring military organizations or regions, identified based on our data from Tracking IPP billets and CIPP Plan Review) and a sample that have not hired IPPW (12 military organizations or regions). We anticipate stratifying the sample by DoD component to ensure representation from each of the Military Departments, Services, and National Guard Bureau (e.g., Army, Air Force, Navy, Marine Corps, Space Force).

Our sampling plan will also take into consideration that, for certain DoD components, a military organization may not be the ideal unit of analysis. For example, for the Navy, we may consider including one or more ships in the sample, as these often have their own prevention ecosystem and resources. We will also ensure our sampling plan includes not only Active Component organizations, but also Guard and Reserve organizations. We will also collaborate with DoD to identify whether there are any additional priorities that should be considered when selecting military organizations. For example, it is our understanding that each DoD component will establish criteria to determine which organizations will hire their IPP personnel first, and we will include that consideration as part of our selection process.

We will work with our service points of contact to identify appropriate sites for their respective service. We will coordinate our schedule with DoD to minimize disruption to military organizations.

Data collection for this method will take place virtually through discussions on Zoom or Teams with key stakeholders at each selected military organization (or region) in Q4 FY24 through Q2 FY25 and again in Q4 FY26 through Q2 FY27 of the evaluation. To the extent possible, there will also be a brief (i.e., one-to-two day) on-site visit coordinated with VPC to observe a prevention activity, likely to take place between Q3 FY25 and Q3 FY26 of the evaluation, to observe an IPP activity.

RAND will use a semi-structured discussion protocol to guide virtual discussions with the following groups at each military organization or region:

• IPP personnel (n=3-5). We will invite all IPP personnel to participate in these interviews.

• Other personnel that are collaborating closely with IPP personnel (or assisting with prevention efforts) at both intervention and comparison installations (n=7-8). We anticipate that this may include family readiness staff, family advocacy programs, sexual assault prevention and response staff, mental health staff, and chaplains. We will ensure that our selection of these other personnel is tailored to each DoD component (e.g., by including community action team members for Air Force installations).

• Military organization/command leadership at all military organizations (n=2-3). We anticipate that this may include the military organization commander, and potentially commanders from a sample of units.

• Front-line leaders or non-commissioned officers at all military organizations to provide a Service member perspective on IPP (n=5-6 but could be greater at some locations). We will work with personnel at each military organization to randomly select a sample of these front-line leaders

Resume Review. RAND will collect resumes by asking IPPW taking the IPP Personnel Survey to share their resumes using a secure file transfer platform (e.g., Kiteworks). Each IPPW will only be asked to share their resume one time, most likely the first time they complete the IPP Personnel Survey. Once IPPW share their resume, they will not be asked to share it again. We will offer $20 Amazon Gift Cards to each IPPW who uploads their resume.

3. Maximization of Response Rates, Non-response, and Reliability

Our sponsor has prepared a data collection support memo to be signed by a military leader within their organization which will be sent out to encourage all IPPW to participate in the evaluation activities. In addition, we will employ the following strategies for each activity.

Leader and Supervisor Interviews. To minimize non-response to the RAND data collection, we plan a robust series of follow-up data collection activities (up to three emails and two phone calls) for non-responding IPPW. In emails and phone calls we plan to emphasize the importance of the evaluation, how it represents an opportunity to have their information considered by DoD leadership, and how the project is endorsed at the highest levels of DoD.

IPP Personnel Survey. To minimize non-response to the RAND data collection, we plan to offer a modest monetary incentive ($40 gift card) to DoD participants, if approved. In addition, we plan a robust series of follow-up data collection activities (up to three emails and two phone calls) for non-responding IPPW. In emails and phone calls we plan to emphasize the importance of the evaluation, how it represents an opportunity to have their information considered by DoD leadership, and how the project is endorsed at the highest levels of DoD.

Integrated Primary Prevention Activity Tracker. To minimize non-response to the RAND data collection, we plan a robust series of follow-up data collection activities (up to three emails) for non-responding IPPW. In emails we plan to emphasize the importance of the evaluation, how it represents an opportunity to have their information considered by DoD leadership, and how the project is endorsed at the highest levels of DoD.

Case Studies. We do not anticipate non-response for this data collection method, as we are working directly with services and our sponsor to identify relevant sites and points of contact at each. RAND has led dozens of site visits at DoD installations before (i.e., the Onsite Installation Evaluation) and every site that was contacted participated in data collection.

Resume Review. Since the resumes will be collected via the IPP Personnel Survey, we will use the same series of follow-up data collection activities (up to three emails and two phone calls) for non-responding IPPW. In emails and phone calls we plan to emphasize the importance of the evaluation, how it represents an opportunity to have their information considered by DoD leadership, and how the project is endorsed at the highest levels of DoD.

4.  Tests of Procedures

For the Leader and Supervisor Interviews, Case Studies (which uses the interview protocols based on the Leader and Supervisor Interviews), IPP Personnel Survey, and Resume Review, we have pilot tested these instruments with a representative from each service branch, National Guard, and Reserves. The pilot test feedback we received was incorporated into the instruments. For the I-PACT, we are planning to pilot test that instrument when some IPP activity has begun, also with representative from each service branch, National Guard, and Reserves.

5.  Statistical Consultation and Information Analysis

a. Provide names and telephone number of individual(s) consulted on statistical aspects of the design.

(Statistical lead) Wenjing Huang, +1 310-393-0411 x7129, whuang@rand.org

b. Provide name and organization of person(s) who will actually collect and analyze the collected information.

Wenjing Huang*	Justin Lee*	Jessica Phillips*
*RAND Corporation.