Supporting Statement Part B

SUPPORTING STATEMENT – PART B

B.  COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Description of the Activity

The ultimate goal of this evaluation is to assess the extent to which the hiring of about 2500 new IPP personnel decreases harmful behaviors among Service members. The overall research questions, and their related method, are:

1. How are dedicated IPP personnel associated with a change in (1) Risk and protective factors (i.e., short-term outcomes)? (2) Self- and other-directed harmful behaviors (i.e., long-term outcomes, e.g., sexual assault, sexual harassment, family violence)?

2. How are IPP personnel, prevention processes, and prevention infrastructure associated with risk and protective factors and long-term outcomes? (Statistical Modeling)

3. What is the cost-benefit and return on investment (ROI) associated with a dedicated IPP personnel?

4. Is it possible to model future IPP impacts based on results from the outcome evaluation of the dedicated IPP personnel effort? If so, what is the cost-benefit and ROI on prevention of future self- and other-directed harmful behaviors per prevention FTE?

To answer these research questions, this evaluation will use multiple data sources specifically collected for this evaluation, described below.

Leader and Supervisor Interviews. RAND will conduct qualitative interviews to assess strategic and operational IPP personnel and DoD component leaders’ perceptions of progress hiring and integrating tactical-level IPP personnel. These interviews will ask questions about barriers to hiring and deploying tactical-level IPP personnel and their current and anticipated impacts. RAND will interview ~5 strategic and operational IPP personnel and DoD component leaders per service in study years 2, 4, and 6.

Tracking IPP Personnel Billets, Trainings, and Professional Development Activities. This data collection involves tracking the progress of filling IPP personnel billets (Hiring Ratio: Number of billets filled/Total number of billets), retaining IPP personnel (Retention Ratio: Number of billets actively filled/Total number of billets previously filled), and providing sufficient coverage across DoD (Number of billets filled/Total number of billets needed) and the completion of required IPP training, credentialing and for full-time IPPW annual professional development requirements.

IPP Personnel Survey. Annually for the entire evaluation period, starting in Year 2, we will administer an online survey to assess competencies of all IPP personnel (~n=2500) across strategic, operational, and tactical levels and perceptions of whether a supportive climate for IPP exists. We are asking IPP personnel competency and climate questions because for this initiative to succeed, these personnel must have the knowledge and skills (i.e., the competencies) to perform their job. Further, the climate with which they operate must be receptive and supportive to prevention work. The end result of the IPP Personnel Survey is that RAND will have an understanding of the level of competency of IPP personnel and the degree of the support they received. This data will be used to contextualize the ultimate outcomes of the evaluation—namely the level of competency of the IPP personnel could help explain why or why not DoD was able to reduce harmful behaviors.

Given that hiring will be rolling across the evaluation period, IPP personnel will take the survey a different number of times depending on when they are hired. See Table 1 for an estimated sample for the survey across the years of the evaluation. This survey is intended to be a census—i.e., we are targeting all IPP personnel across all of DoD services and levels, not a sample. Based on previous RAND military projects, we anticipate the response rate to be ~65%.

Table 1. Estimated unique number of new IPP personnel surveyed by FY

	FY 22-23	FY24	FY25	FY26	FY27	FY 28
DoD component	Study Y1	Study Y2	Study Y3	Study Y4	Study Y5	Study Y6
Army	338	458	672	886	980	980
Navy	88	99	128	128	128	128
Marine Corps	31	43	43	43	43	43
Air/Space Force	227	318	406	417	417	417
ARNG	170	325	325	325	325	325
ANG	71	136	136	136	136	136
Totals	925	1379	1710	1935	2029	2029

Integrated Primary Prevention Activity Tracker (I-PACT). Every six months, RAND will collect, via online, secure data collection portal developed by RAND, information about the IPP activities being implemented by all IPP personnel (i.e., Integrated Primary Prevention Activity Tracker, or I-PACT). The I-PACT will collect information on:

• Types of IPP activities

• Needs assessment and planning efforts, including collaboration and stakeholder engagement

• Target populations and participation

• Domains, behaviors, risk and protective factors targeted

• Evidence-basis, best practice adherence, and adaptations

• Evaluation efforts

This data collection is intended to be a census, including all locations where all IPP personnel work.

RAND will contract with 3C Institute (3C; www.3CISD.com), a North Carolina S-Corporation, to build the I-PACT. 3C has extensive R&D experience creating innovative technologies, including custom interactive websites, dynamic e-learning platforms, online and mobile applications for information sharing, data collection and analysis systems, and online distance networking and collaboration platforms for a wide array of purposes and audiences (see www.3CTechnologies.com). They are leaders in the field of game-based platforms for intelligent assessment, adaptive learning, and personalized feedback. 3C’s technologies have been repeatedly tested to ensure maximum usability, security, and effectiveness with high fault tolerance for diverse users in diverse settings. Their products target areas in line with our subject matter expertise in social-emotional learning; mental health intervention/prevention, and implementation science. They have also extended our R&D experience into other healthcare realms through multiple external collaborations (e.g., chronic kidney disease with Dr. Maria Ferris of UNC-Chapel Hill). 3C has successfully commercialized over 50 evidence-based health assessment and intervention tools (see www.3CMarketplace.com/catalog/products). Because of these efforts, 3C’s health promotion tools are currently being used by community agencies to help thousands of people across the U.S. and in 12 countries. Their interventions have won multiple awards, including the SAMHSA Science and Service Award for Mental Health Promotion.

Cohort Study. RAND will conduct a quasi-experimental study across DoD components to compare a sub-group of installations that have and have not hired IPP personnel on barriers to hiring, prevention activities, and support for IPP, at local organizational and DoD component levels, and provide context to the findings from the other methods. This method will provide a bottom-up perspective on the hiring, integration, and effectiveness of the IPP personnel, as well as critical context to the overall evaluation findings, including a clearer understanding of barriers and facilitators that IPP personnel face.

These installations will be recruited in two cohorts. Cohort A will comprise 16 installations (8 with their full complement of IPP Personnel and 8 without); Cohort B will comprise 10 installations, all with their full complement of IPP personnel.

The mode of data collection during these visits will be interviews with key IPP personnel. These interviews will follow closely the protocol for the Leader/Supervisor Interviews described above, adapted for the tactical/installation instead of the strategic or operational levels. For example, we will ask questions related to barriers and facilitators to hiring IPP personnel and implementing IPP activities using the Consolidated Framework for Implementation Research framework. We will also ask questions related to organizational support for IPP based on the Prevention Evaluation Framework, that are similar to questions on the IPP Personnel Survey. The end result of this data collection will be a qualitative analysis of the barriers and facilitators that IPP personnel face in their work as well as a quantitative rating of the interview data according to the Prevention Evaluation Framework, a measure RAND developed in collaboration with DoD that specifically assesses various markers of high quality IPP in military settings.

Resume Review. Using the required IPP workforce skills and competencies from DoD components’ PDs and the DoD’s Prevention Workforce Model, RAND will rate the alignment of a sub-set of full-time IPPW resumes with these requirements.

RAND will obtain from our sponsor a list of all IPP Position Descriptions (PD) across all the services. To inform the analysis, each PD will be broken down into a list of requirements which will then be used to develop a data abstraction form for use with the resumes for IPP personnel hired for that PD. The skills, competencies, and experiences related to each PD requirement will be abstracted from each resume submitted under the associated PD. Then each resume will be given a score as fully, partially or not at all meeting the PD requirements. We will repeat this process with requirements based on DoD component instructions and guidance. The end result of this analysis will be a Score by position level (for Levels 2-5) for IPP personnel by DoD component describing whether the actual qualifications of the IPP personnel they hired (as described on their resumes) are fully, partially, or not at all aligned with what is required by the DoD component and DoD policy/guidance.

2. Procedures for the Collection of Information

Leader and Supervisor Interviews. To identify who we will interview, we will start with IPP personnel at the strategic- and operational-levels to ask who their direct supervisors are until we get to a non-IPP personnel and interview that person. We will refer to this non-IPP personnel as a DoD component leader. We will also interview members of the Prevention Collaboration Forum, a governance body to oversee the implementation of the integrated prevention policy (DODI 6400.09, 6400.11). In Year 2, we will sample through ‘snowball’ referrals—i.e., using DoD stakeholders to guide us to the most appropriate individuals with whom to interview. We will ask the sponsor for points of contact in Office of the Under Secretary of Defense for Personnel and Readiness, the DoD components, and potentially the Joint Staff. Once identified, we will ask the following as part of the snowball sampling:

• Who in your organization will have responsibility for the hiring, supervision, and management of these new IPP personnel? Who will the new IPP personnel report to?

• Who in your organization will have oversight for policy and resource allocation decision-making for the IPP personnel?

We will interview personnel across the DoD components. We anticipate interviewing about 40 personnel (to include both uniformed and civilian, as relevant) across DoD components (Table 1), depending on the snowball sampling, in study years 2, 4, and 6.

Table 2. Proposed DoD Components

Proposed Organizations for DoD Component Leaders

U.S. Air Force: Active HQ; Reserve HQ; National Guard HQ

U.S. Space Force: Active HQ

U.S. Army: Active HQ; Reserve HQ; National Guard HQ

U.S. Navy: Active HQ; Reserve HQ

U.S. Marine Corps: Active HQ; Reserve HQ

U.S. National Guard Bureau

*Contingent upon where the IPPW are placed (e.g., whether any are assigned to reserve installations)

It is also anticipated that we will interview leaders at OSD Personnel and Readiness organizations with a role in recruiting or coordinating with the IPP personnel and monitoring outcomes of prevention activities, as well as members of the Prevention Collaboration Forum. In Years 4 and 6 we will follow up with the same personnel from Year 2, or different ones if the personnel have changed positions or position responsibilities have changed. All interviews will be conducted by Zoom.gov video calls.

Once we identify who we want to interview, we will obtain their email addresses and phone numbers from our sponsor. We will send up to 3 emails and make up to two phone calls to obtain participation. The email will contain a description of the project and its importance, including how the project was recommended by the Independent Review Commission on Sexual Assault in the Military. We will email the respondents up to three times and call up to two times to attempt to obtain a response from each IPP personnel (either yes or no). Given the saliency of the project, we estimate that there will be a high response rate, ~80%.

Tracking IPP Personnel Billets, Trainings, and Professional Development Activities.

This information will be provided either by service branch points of contact or from our sponsor, the Violence Prevention Cell (VPC). The schedule of data collection will depend on the specific services and VPC and will likely range from monthly to semi-annually. This data collection is intended to be a census of all IPP personnel.

IPP Personnel Survey. RAND will obtain email addresses and phone numbers of all IPP personnel from our sponsor. Annually for the entire evaluation period, starting in Year 2, we will send IPP personnel an email containing a link to an online survey. The email will contain a description of the project and its importance, including how the project was recommended by the Independent Review Commission on Sexual Assault in the Military. The email will also mention that the respondent will receive a $40.00 gift card upon completion of the survey. We will email the respondents up to three times and call up to two times to attempt to obtain a response from each IPP personnel (either yes or no). The survey will be conducted via secure online survey platform (e.g., Qualtrics). This data collection is intended to be a census of all IPP personnel, not a sample.

We will offer $40 Amazon Gift Cards to each IPP personnel who completes a IPP Personnel Survey each year. These individuals will receive an email with a link to their Gift Card. There will be no requirements about the level of completion for each data collection. Once the survey is submitted online, the individual will be eligible for the Gift Card. These payments are being offered to compensate the participating individuals for their time.

Integrated Primary Prevention Activity Tracker. Every six months, an email will be sent to all IPP personnel with a link to an online portal that RAND will create. RAND will send up to two follow up emails in order enhance response rate. This data collection is intended to be a census of all IPP personnel, not a sample. Based on previous RAND military projects, we anticipate the response rate to be ~70%.

Cohort Study. Our selection of intervention sites will be guided by the following criteria:

• DoD component: We will ensure representation from each of the Military Departments, Services, and National Guard Bureau (e.g., Army, Air Force, Navy, Marine Corps, Space Force). Our sampling plan will also take into consideration that, for certain DoD components, an installation may not be the ideal unit of analysis. For example, for the Navy, we will include one or more ships in the sample, as these often have their own prevention ecosystem and resources. We will also ensure our sampling plan includes not only Active Component installations, but also some Guard site and some Reserve sites.

• Baseline level of risk: We will recruit installations with varying levels of risk to determine if the effect of IPP personnel is larger at certain types of installations (e.g., higher-risk installations). We will work with the DoD to determine risk level, using data from the Defense Organizational Climate Survey (DEOCS).

• Installation characteristics: We will also use installation size and geography (e.g., rural/remote status), and whether an installation is joint or has tenant units.

Comparison sites will be selected from installations that have not yet hired their full complement of IPP personnel. To the extent possible, we will aim to select comparison sites that are matched to the intervention sites on the criteria used to select intervention sites, with a focus on match on DoD component and baseline risk, if possible.

Data collection for this method will take place during one-to-two-day site visits by RAND staff. As shown in Figure 1, Cohort A sites will be visited in Year 2, and then again in Year 4. Cohort B sites will be visited in Years 3 and 5.

Figure 1. Design of the Cohort Study

During the site visits, RAND will conduct semi-structured interviews with the following groups at each installation:

• IPP personnel at the intervention installations (n=3-5). We will invite all IPP personnel to participate in these interviews.

• Other personnel that are collaborating closely with IPP personnel (or assisting with prevention efforts) at both intervention and comparison installations (n=7-8). We anticipate that this may include family readiness staff, family advocacy programs, sexual assault prevention and response staff, mental health staff, and chaplains. We will ensure that our selection of these other personnel is tailored to each DoD component (e.g., by including community action team members for Air Force installations).

• Installation/command leadership at intervention and comparison installations (n=2-3). We anticipate that this may include the installation commander, and potentially commanders from a sample of units.

We plan to coordinate these site visits with the Onsite Installation Evaluation (OSIE) process, which also involves site visits. Once the sites are identified, we will work with the OSIE leads to make contact with the sites and arrange the visit.

Resume Review. RAND will collect resumes two ways. First, we will ask for those taking the IPP Personnel Survey to share their resumes using a secure file transfer platform (e.g., Kiteworks). We will also attempt to secure resumes through DoD’s new prevention credentialling system, which will be asking IPPW for their resumes. In that case, VPC will send resumes to RAND via Kiteworks. Our goal is to achieve a 10 percent sample of IPPW (each year on an ongoing basis as they are hired). We will purposefully select IPPW across DoD components, with estimated numbers and types based off the DoD component’s final hiring plans.

3. Maximization of Response Rates, Non-response, and Reliability

Leader and Supervisor Interviews. To minimize non-response to the RAND data collection, we plan a robust follow-up data collection scheme (up to three emails and two phone calls) for non-responding IPP personnel. In emails and phone calls we plan to emphasize the importance of the evaluation, how it represents an opportunity to have their information considered by DoD leadership, and how the project is endorsed at the highest levels of DoD.

Tracking IPP Personnel Billets, Trainings, and Professional Development Activities. It is not anticipated that there will be non-response for this data source. Our sponsor is tracking this data and has committed to providing that data to us.

IPP Personnel Survey. To minimize non-response to the RAND data collection, we plan to offer a modest monetary incentive to DoD participants, if approved. In addition, we plan a robust follow-up data collection scheme (up to three emails and two phone calls) for non-responding IPP personnel. In emails and phone calls we plan to emphasize the importance of the evaluation, how it represents an opportunity to have their information considered by DoD leadership, and how the project is endorsed at the highest levels of DoD.

Integrated Primary Prevention Activity Tracker. To minimize non-response to the RAND data collection, we plan a robust follow-up data collection scheme (up to three emails) for non-responding IPP personnel. In emails we plan to emphasize the importance of the evaluation, how it represents an opportunity to have their information considered by DoD leadership, and how the project is endorsed at the highest levels of DoD.

Cohort Study. We do not anticipate non-response for this data collection method. RAND led the first OSIE data collection effort and every site that was contacted participated. We will be collaborating with the new OSIE data collection team to arrange the visits.

Resume Review. We plan a robust follow-up data collection scheme (up to three emails and two phone calls) for non-responding IPP personnel. In emails and phone calls we plan to emphasize the importance of the evaluation, how it represents an opportunity to have their information considered by DoD leadership, and how the project is endorsed at the highest levels of DoD.

4.  Tests of Procedures

For the Leader interviews, Cohort Study (which uses the interview protocols based on the Leader interviews), IPP Personnel Survey, and Resume Review, we are planning on pilot testing with a representative from each service branch, national guard, and reserves. For the Integrated Primary Prevention Tracker, we are planning to pilot test that instrument when some IPP activity has begun, also with representative from each service branch, national guard, and reserves.

5.  Statistical Consultation and Information Analysis

a. Provide names and telephone number of individual(s) consulted on statistical aspects of the design.

(Simulation Modeling) Raffaele Vardavas, +1 310-393-0411 x6079, rvardava@rand.org

(Cost Analyses) Patricia Herman, +1 310-393-0411 x7129, pherman@rand.org

(Statistical Modeling) Wenjing Huang, +1 310-393-0411 x7129, whuang@rand.org

b. Provide name and organization of person(s) who will actually collect and analyze the collected information.

Andy Bogart*	Justin Lee*	Patricia Herman*
Wenjing Huang*	Jessica Phillips*	Raffaele Vardavas*
*RAND Corporation.

6