Prevention Workforce Evaluation

Case Study Protocols

INTRODUCTION/CONSENT TO BE USED BEFORE EACH PROTOCOL:

Hello, my name is ___________ and I am accompanied by [INTRODUCE OTHERS IN THE ROOM AND THEIR ROLES]. We’re researchers at RAND Corporation, which is a non-profit, policy research organization. We are working with Office of the Secretary of Defense for Personnel and Readiness and the Office of Force Resiliency’s Violence Prevention Cell, to evaluate the Department of Defense’s prevention workforce initiative. Thank you for taking the time to talk today about how [FILL IN SITE/MILITARY ORGANIZATION NAME] approaches integrated primary prevention. Our questions today are going to focus on how integrated primary prevention is organized at this [military organization]; your role and how you’ve been integrated into this [military organization’s] prevention work; how you plan and solicit feedback on prevention activities; and strengths and challenges related to integrated primary prevention here.

Before we begin, I want to assure you that your answers will be held in confidence by RAND to the extent allowed by law and DoD policy. [If more than one person participating in the discussion: However, there are others participating in this discussion so please do not say anything you do not want others to know.] We are having discussions with many folks who are involved in prevention at this [military organization]. Our team will also be reviewing documents, data, and other information collected from your [military organization]. Findings from across these sources will be reported in briefings and reports to DoD, but we would not be naming you specifically in any of those documents. We will be taking notes during all meetings during the discussions, but they will not have your name on them. There should be no discussion of open cases or investigations so as to respect the privacy of individuals who may participate in integrated primary prevention activities at this [military organization].

Your participation is voluntary, but may be subject to regulations and orders at your location. Refusal to participate will involve no penalty or loss of benefits to which you are otherwise entitled. You may discontinue participation at any time without penalty or loss of benefits, to which you are otherwise entitled. If you have questions about your rights as a participant or need to report a participation-related injury or concern, you can contact RAND's Human Subjects Protection Committee using the contact information provided to you in the materials shared by email in advance of this discussion.

Any questions before we begin?

Protocol #1: Integrated Primary Prevention Personnel

NOTE: Bolded language in parentheses maps questions onto the evaluation metrics.

Background

Before we dive in, I wanted to highlight that our interviews are focused specifically on primary prevention activities, rather than any role you might plan in intervention or response efforts. We’re especially interested in prevention of suicide, sexual assault, harassment, domestic violence, child abuse, substance use, and retaliation. And in particular, we’re interested in integrated primary prevention activities (meaning, activities that are meant to stop two or more harmful behaviors before they start). In these first few questions, I’ll be asking you more about your role, so you’ll be able to tell me about how your time is spent on prevention versus other activities – but then we’ll be switching gears to really focus on primary prevention.

1. Can you start by telling me a bit about yourself– what is your official title, and how long have you been in the role? (Background)

   1. What percentage of your time is spent on primary prevention-related activities?

      1. Potential probes: Percentage of time understanding integrated prevention; conducting needs assessments; planning activities; evaluating activities

Prevention Efforts at this Site/Military Organization

2. Now I’m going to ask some questions to learn more about how prevention efforts are organized at this [military organization]. (2.1.5)

1. How many new integrated primary prevention personnel are there at this [military organization] – that is, personnel whose work is focused on integrated primary prevention?

2. (For more senior preventionists) Have all the Integrated Primary Prevention personnel billets been filled at this [military organization]?

   1. If no, which ones are vacant? For how long?

   2. When those additional billets get filled, what duties will those personnel be responsible for?

3. Is the number of Integrated Primary Prevention personnel enough to properly accomplish the prevention goals here?

4. [If there are multiple Integrated Primary Prevention personnel] How do you work together with other Integrated Primary Prevention personnel?

   1. Probe: Do you meet regularly?

3. What types of other offices or positions are involved in prevention work? Which of these are involved in integrated primary prevention? (Background)

   1. How much do you work with them?

   2. In what ways do you work with them?

4. If there are multiple commands at this [military organization], who leads prevention efforts? For example, do Integrated Primary Prevention personnel from across commands collaborate, or do they focus on prevention for their command (rather than the [military organization] or installation)? (For joint installations) How does collaboration happen across branches? (Background)

5. Who oversees prevention efforts at this [military organization]? (Background)

1. What position does your direct supervisor hold – that is, the position of the person to whom you report? (Potential options: installation commander, other installation-level staff, prevention personnel or commander located at a different installation or higher level)

Note: If interviewee describes having a supervisor at this [military organization] who is someone other than the commander (e.g., someone within the Fleet and Family Support Center or another prevention/intervention program), we should interview that person using Protocol #4.

6. How prepared were you to start as an Integrated Primary Prevention personnel? (3.2.4)

   1. What types of trainings did you have, if any, as you started in your role? What topics did they cover?

   2. What topics do you wish had been covered to help you be more prepared in your role?

   3. What topics were most relevant or helpful in the work you’re now performing?

   4. What suggestions do you have for improving the training?

7. What are your responsibilities as part of the Integrated Primary Prevention Workforce? (2.1.5)

   1. What percentage of your role is focused on primary prevention?

   2. Potential probes: Develop integrated prevention plan, Support command climate assessment, Plan and execute prevention activities, Collect and analyze data to evaluate prevention efforts, Collaborate with key stakeholders

Readiness for Preventionists and Site/Military Organization Climate

8. When you were hired, how prepared did it seem [this [military organization]] was to incorporate Integrated Primary Prevention personnel into the [military organization]? (3.1.3)

   1. Was your role well understood by senior leaders?

   2. Was your role well understood by other prevention staff or leadership (e.g., from other programs, such as Family Programs, SAPR, etc.)?

   3. How were you oriented to your role?

   4. How well are IPP personnel currently integrated? How has this changed over time?

9. Have you had all the resources you need to plan high quality prevention activities (Resources can include having the budget, staff, materials, space, or time with service members needed for prevention) (3.1.3)

   1. What about to deliver high quality prevention activities within your normal duty hours? (Potential probes: Budget, installation resources or facilities, time availability)

   2. What types of resources have been most beneficial?

   3. What types of resources are missing? Are you going to be able to get these resources? Why or why not?

10. How do you interact with your [military organization]’s leadership -for example, [military organization] command? About what types of topics or situations? (6.2.3)

    1. For example, do they provide input or feedback on the [military organization]’s prevention needs?

       1. If so, what is their input based on? (Potential probes: Data-informed, observation, opinion)

    2. Do you brief [leader] on the progress of your prevention efforts? Do they review evaluation findings?

    3. Has the relationship between [military organization] leadership and Integrated Primary Prevention personnel changed over time? How so?

11. How supportive has the leadership of this [military organization] been of prevention activities? (6.2.3)

    1. What types of leaders have been most supportive of prevention activities?

    2. What kind of support, if any, have they given you? Can you provide specific examples?

    3. Have they put up any barriers? If yes, what are examples?

    4. Have leaders been receptive to the prevention activities that have occurred since you entered this role?

12. How have you and your efforts been incorporated into any prevention processes that were already in place at this [military organization] before you started? For example, this could include meetings of all prevention staff, running prevention efforts, collection of data related to prevention, or things like that? (3.1.3)

13. What type of communication and/or support do you receive from Integrated Primary Prevention personnel at other levels – for example, those at the operational or strategic level [major command or headquarters level]? (4.3.4)

    1. How do they communicate with you?

    2. What types of things do they communicate with you?

    3. Do you feel like you have enough support from Integrated Primary Prevention personnel at those levels? Why or why not?

14. How receptive are service members at this [military organization] to prevention activities? (3.1.4)

    1. Can you give an example of a time that they were particularly receptive? Can you give an example of a time that it seemed that they were not receptive?

    2. Are there certain types of activities that seem to be better received? Can you give me an example?

Overarching Barriers and Facilitators

15. Has the hiring of the Integrated Primary Prevention personnel been implemented according to plan at this [military organization]? Has your role been consistent with the guidance you’ve received? (5.5.1)

    1. What has made implementation easier? What has made implementation successful?

    2. What has gotten in the way?

    3. What hasn’t gone to plan?

    4. How has the initiative evolved over time at this [military organization]?

16. What are the most significant challenges to providing high-quality integrated primary prevention activities at this [military organization]? (5.5.1)

17. What have been the most important factors facilitating your efforts to provide high-quality integrated primary prevention activities (e.g., leadership support, centralized preventionist resources, collaborations with other [military organization] staff (5.5.1)

Additional Questions for Sites/Military Organizations/Commands Without Full Complement of Integrated Primary Prevention Personnel

18. What do you know about the plans to hire the remaining prevention staff? (Potential probes: Number of people to be hired, levels, timeline) (2.1.5)

19. What types of things are the [military organization]/command doing to prepare for hiring those additional Integrated Primary Prevention personnel? (2.1.5)

20. How do you expect your role will change after the remaining Integrated Primary Prevention personnel have been hired, if at all? (2.1.5)

21. Are there things you haven’t been able to accomplish because the [military organization] does not have all its prevention roles filled? What are those things? What do you expect to be able to accomplish once all the Integrated Primary Prevention personnel have been hired? (2.1.5)

Conclusion

22. Is there anything that I haven’t asked already that you think would be helpful for me to know about your work or about prevention at this [military organization]?

Protocol #2: Other Prevention Staff (e.g., Family Advocacy, SAPR-VA)

NOTE: Bolded language in parentheses maps questions onto the evaluation metrics.

Background

Before we dive in, I wanted to highlight that our interviews are focused specifically on integrated primary prevention activities (meaning, activities that are meant to stop two or more harmful behaviors before they start) – so please keep that framing in mind when answering these questions.

1. Can you start by telling me a bit about yourself – what is your role, and how long have you been in the role? (Background)

2. How is prevention organized at this [military organization]? (2.1.5)

1. How does integrated primary prevention fit into this?

3. How many Integrated Primary Prevention personnel are there at this [military organization] – that is, preventionists whose work is focused on integrated primary prevention? (2.1.5)

1. Is the number of integrated primary preventionists enough to properly accomplish the integrated prevention goals here?

2. How do you work together?

4. What types of other offices or positions are involved in prevention work? (2.1.5)

5. What is your role in prevention? (2.1.5)

6. Has your role in prevention changed since the hiring of the Integrated Primary Prevention personnel? Do you expect your role in prevention will change with the hiring of the Integrated Primary Prevention personnel? How so? (2.1.5)

7. Did the Integrated Primary Prevention personnel replace any of the responsibilities or roles that used to be held by other prevention staff? If so, what? (2.1.5)

8. Will you be/Have you been collaborating with the Integrated Primary Prevention personnel in any way? Can you give me some examples? (Background)

9. How have the Integrated Primary Prevention personnel and their efforts been incorporated into current processes here? (3.1.3)

1. Do they have an opportunity to collaborate with other prevention staff at this [military organization], if there are any?

2. Are they invited to regular meetings of prevention staff, if those exist? For example, we’re thinking about meetings that are used to plan events or evaluate prevention needs, not simply case management meetings.

Readiness for Preventionists and Site/Military Organization Climate

10. How prepared is/was [military organization] to integrate Integrated Primary Prevention personnel into this [military organization? (3.1.3)

11. Is there a comprehensive integrated prevention plan in place? (3.1.3)

12. Does the [military organization] have all the resources it needs to incorporate the Integrated Primary Prevention personnel and host high quality prevention activities? These can include having the budget, staff, materials, space, or time with service members needed for prevention. (3.1.3)

1. Potential probe: What about integrated prevention? Are there any additional resource needs specific to that type of prevention?

13. What types of communication have there been about the Integrated Primary Prevention personnel and their role to other prevention and intervention staff at the [military organization]? (3.1.3)

14. What types of communication have there been about the Integrated Primary Prevention personnel and their role to service members at this [military organization]? (3.1.3)

15. How supportive has the leadership of your [military organization] been of prevention activities? (6.2.3)

    1. Has this varied by type of leader – for example, installation leadership versus [program] leadership?

    2. What kind of support, if any have they given you? Can you provide specific examples?

    3. Have they put up any barriers? If yes, what are examples?

    4. Has leadership provided any more or less support for prevention since the hiring of the Integrated Primary Prevention personnel?

    5. Does leadership provide different types of support for integrated primary prevention compared to other prevention activities?

16. 

    l (

    f

    How receptive are service members at this [military organization] to prevention activities? (3.1.4)

    1. Are there certain types of activities that seem to be better received? What are they?

    2. Have you noticed any change in the service members’ receptiveness of prevention since the hiring of the Integrated Primary Prevention personnel?

17. Is there a strong need for dedicated prevention personnel at this [military organization] – that is, prevention staff whose roles are fully focused on primary prevention? (3.1.5)

    1. Why or why not?

    2. Do others see a need for it?

Perceptions of Preventionists and Prevention Activities

18. How effective do you think the Integrated Primary Prevention personnel have been? (4.4.1)

    1. How do you know if they’ve been an effective addition to this location? What sources of data?

    2. What has been the most effective impact of the Integrated Primary Prevention personnel at this [military organization]?

    3. The least effective?

19. Are there still gaps related to prevention that are not being addressed? What are those? Are there barriers that are getting in the way? If yes, what are they? (5.5.1)

20. What are the most significant challenges to providing high-quality prevention activities at this [military organization]? (5.5.1)

Protocol #3 : Installation Command/Unit Commanders

NOTE: Bolded language in parentheses maps questions onto the evaluation metrics.

With this interview, we’re interested in asking some questions about prevention efforts at this [military organization], as well as some questions about the Integrated Primary Prevention personnel who have been hired/are being hired. Before we dive in, if I ask any questions that you don’t know the answer to because they’re too “in the weeds,” let me know and we can skip those. I also wanted to highlight that when we refer to “prevention,” we are focused specifically on primary prevention activities, meaning activities intended to stop harmful behaviors before they start. And when we talk about “integrated primary prevention,” we’re specifically referring to activities that are meant to stop two or more harmful behaviors before they start.

1. Can you start by telling me what your role is/will be in overseeing the work of the Integrated Primary Prevention personnel at this [military organization]? (Background)

   1. For example, are you responsible for setting prevention priorities, working with Integrated Primary Prevention personnel to ensure their activities are meeting the needs of service members, evaluating the effectiveness of their activities?

   2. Do you have any role in their hiring?

Implementation Climate

2. How much of a need is there for full-time prevention personnel at this [military organization]? (3.1.5)

1. Why or why not?

2. Do others see a need for it?

3. Are any prevention activities required? Or strictly voluntary?

4. Have there been any changes in the need for prevention staff over time over the last couple of years?

5. Have the Integrated Primary Prevention personnel filled a gap on this [military organization] when it comes to the prevention strategy? Why or why not?

3. How essential are the Integrated Primary Prevention personnel and their prevention activities, in terms of meeting the needs of service members at this [military organization] and helping to maintain their readiness? (3.1.5)

1. Are the Integrated Primary Prevention personnel filling a role/will they fill a role that wasn’t previously filled at this [military organization]? Who was previously responsible for the activities that that the Integrated Primary Prevention personnel are now filling?

2. What aspects of readiness do you think will be affected by having these new roles at this [military organization]?

4. How have the prevention activities been received? (3.1.4)

1. How do you think service members will respond/are responding to the Integrated Primary Prevention personnel and prevention activities?

2. Are the Integrated Primary Prevention personnel and their activities addressing needs that weren’t previously being met at this [military organization]?

Readiness for Implementation

5. Do you feel you have enough resources to support the Integrated Primary Prevention personnel? (3.4.3, 6.2.3)

   1. What resources do you need?

6. Do you feel you have enough information to support the Integrated Primary Prevention personnel? (6.2.3)

1. Has there been any briefing or training to prepare you?

2. [If yes] What training?

7. How are you supporting the Integrated Primary Prevention personnel and their efforts to becoming incorporated into existing prevention processes and workflows at this [military organization] – for example, planning of events or evaluating the [military organization]’s prevention needs? (6.2.3)

8. What kinds of information and materials about the Integrated Primary Prevention personnel have been made available to you/were made available to you? (3.1.3)

1. Has it been timely? Relevant? Sufficient?

Feedback and Evaluation

9. How effective do you think that the Integrated Primary Prevention personnel have been? (4.4.1)

   1. How do you know if they’ve been an effective addition to this [military organization]? Has there been any evaluation of the prevention activities? What sources of data?

   2. What has been most effective?

   3. What has been least effective?

10. Are there still gaps related to prevention that are not being addressed? What are those? Are there barriers getting in the way? If yes, what barriers? (5.5.1)

11. What are the most significant challenges to providing high-quality prevention activities at this [military organization]? (5.5.1)

Protocol #4: Site/Military Organization-Level Supervisors/Managers, non-Command

NOTE: Bolded language in parentheses maps questions onto the evaluation metrics.

With this interview, we’re interested in understanding the roles of the Integrated Primary Prevention personnel at this [military organization], and how their role fits with other prevention activities at the [military organization]. When we talk about “integrated primary prevention,” we’re specifically referring to activities that are meant to stop two or more harmful behaviors before they start.

1. Can you start by telling me a bit about yourself – what is your role, and how long have you been in the role? (Background)

2. How many Integrated Primary Prevention personnel do you have here now and how long have they been here? (2.1.5)

1. Do you think there are enough Integrated Primary Prevention personnel here to accomplish the prevention goals here?

2. Are you planning to hire more? If yes, how many and by when?

3. What factors have influenced your [military organization]’s hiring efforts? (Probes: DoD policy, availability of qualified candidates)

3. What type of role do you have in overseeing the work of the Integrated Primary Prevention personnel and/or collaborating with them? (2.1.5)

   1. For example, are you responsible for setting prevention priorities, working with Integrated Primary Prevention personnel to ensure their activities are meeting the needs of service members, evaluating the effectiveness of their activities?

   2. Do you make budgeting or resource allocation decisions for Integrated Primary Prevention personnel or their activities (e.g., facilities, supplies, licenses, money)? If yes, what is the budget for them?

4. Has your role in prevention changed since the hiring of the Integrated Primary Prevention personnel /do you expect your role in prevention will change with their hiring? How so? (2.1.5)

5. Did the Integrated Primary Prevention personnel replace any of the responsibilities or roles that used to be held by other prevention staff? If so, what? (2.1.5)

Readiness for Implementation

6. Given [INFO FROM #2—i.e., based on their experience with their Integrated Primary Prevention personnel], do you think there is a need for dedicated prevention personnel at this [military organization]? (3.1.5)

1. Why or why not?

2. Do others see a need for it?

7. How ready was/is your [military organization] to hire Integrated Primary Prevention personnel? (3.1.3)

8. What have you done (or what do you plan to do) to get a plan in place to prepare for their hiring? (3.1.3)

9. What role, if any, have you had in orienting or onboarding Integrated Primary Prevention personnel? (3.1.3)

1. Are there any standard orientation/onboarding processes or materials for Integrated Primary Prevention personnel? If so, what was the approach in designing them, and are there materials we may review?

10. Do you have enough resources to support the Integrated Primary Prevention personnel? (3.1.3)

11. Do you feel you have enough information about the purpose of the Integrated Primary Prevention personnel or the scope of their role to support them? (6.2.3)

12. What kinds of information and materials about the Integrated Primary Prevention personnel have been made available to you/were made available to you? (6.2.3)

1. Has it been timely? Relevant? Sufficient?

13. How have the Integrated Primary Prevention personnel and their efforts been incorporated into current processes? (3.1.3)

1. Do they have an opportunity to collaborate with other prevention staff at this [military organization], if there are any?

2. Are they invited to regular meetings?

3. Do they brief you on progress of their prevention efforts?

4. What kinds of changes or alterations do you think you will need to make to the prevention workforce initiative so it will work effectively at this [military organization]?

14. Have you encountered any differing perspectives between the [military organization] and HQ level management and oversight on incorporating the Integrated Primary Prevention personnel and their activities? (4.3.4)

1. If so, what were they? Were they complementary or at odds? How have those differences been addressed (if they have)?

15. How do service members on this installation feel about current prevention activities? (3.1.5)

1. To what extent did previous prevention activities fail to meet existing needs? Are the Integrated Primary Prevention personnel and their efforts meeting these needs? If so, how?

Impact

16. What do you anticipate the impact of the Integrated Primary Prevention personnel will be? (4.4.1)

17. What impact have you seen thus far? How were you able to identify that impact? (4.4.1)

1. Probe for types of impacts: Was any kind of evaluation conducted? Was any data collected on frequency and quality of prevention efforts, reach, ability to integrate violence prevention efforts, improved data collection and monitoring, impact on harmful behaviors