2023 Military Service Academies on Site Installation Evaluation

2023_MILITARY_SERVICE_ACADEMIES_ON_SITE_INSTALLATION_EVALUATION_REPORT.PDF

DoD-wide Data Collection and Analysis for the Department of Defense Qualitative and Quantitative Data Collection in Support of the Independent Review Commission on Sexual Assault Recommendations

2023 Military Service Academies on Site Installation Evaluation

OMB: 0704-0644

Document [pdf]

Download: pdf | pdf

2023 Military Service
Academies On-Site Installation
Evaluation Report

2023 Military Service Academies On-Site Installation Evaluation Report

Table of Contents
Executive Summary........................................................................................................................................... 4
Introduction........................................................................................................................................................ 9
Methodology ...................................................................................................................................................... 9
Findings and Recommendations ..................................................................................................................... 14
Conclusion....................................................................................................................................................... 23
Appendix A: Consolidated Cross-MSA Recommendations ............................................................................. 24
Appendix B: MSA Site Profiles and Supporting Data ...................................................................................... 26
Appendix C: OSIE Process, Integrated Prevention Metric Development, Validation, & Scoring ....................... 36
Appendix D: Acronyms List .............................................................................................................................. 53

Executive Summary
In February 2021, Secretary of Defense Austin directed that the Office of the Secretary of Defense (OSD)
conduct on-site installation evaluations (OSIEs) at installations throughout the Department. Based on the
value of the 2021 OSIEs, Secretary Austin directed the Under Secretary for Personnel and Readiness to
conduct OSIEs in 2023 and biennially thereafter. On March 10, 2023, Secretary Austin directed OSIEs be
conducted at the military service academies (MSAs) after an increase in the estimated prevalence of unwanted
sexual contact (USC), sexual harassment (SH), and other concerning climate issues at the MSAs during the
2021-2022 academic year. As a result, the OSIE methodology was applied to each MSAs to gain insight on
shared risk and protective factors. As a critical leadership tool, these visits provide the Department of Defense
(DoD) with ways to comprehensively improve its prevention efforts, better support efforts to advance the
approved recommendations of the Independent Review Commission on Sexual Assault (IRC) in the Military
and the Suicide Prevention and Response Independent Review Commission (SPRIRC) and inform future
policy development. The MSA OSIEs were to be completed no later than April 30, 2023.

Methods:
The MSAs were identified for OSIE visits based on increased rates of USC and other climate indicators on the
scientific survey (Service Academy Gender Relations) that included a majority of the cadet/midshipman
population. Consistent with other OSIEs, site visits were scoped to those units that had elevated scores on
multiple risk factors or elevated scores on multiple protective factors based on their most recent Defense
Organizational Climate Survey (DEOCS). These outlier units provide indicators of what factors may be
contributing to the increased rates and may inform necessary preventative measures.
Utilizing best practices and lessons learned from the 2021 OSIE visits, the MSA OSIEs assessed prevention
capabilities and climate of the MSAs and units of interest through focus groups with cadets/midshipmen,
leaders and prevention staff, surveys, document review and data evaluation. Site visit teams (SVT) collected
data to inform nine OSIE metrics. Additional information on the OSIE framework and domains are found in
Table 3.
This report summarizes findings and recommendations for the three MSAs:
•
•
•

United States Military Academy (USMA) - OSIE conducted March 12 – 18, 2023
United States Naval Academy (USNA) - OSIE conducted March 5 – 11, 2023
United States Air Force Academy (USAFA) - OSIE conducted March 19 – 25, 2023

Findings and Recommendations:
OSIEs assess prevention and climate factors. Department of Defense Instruction (DoDI) 6400.09, “DoD Policy
on Integrated Primary Prevention of Self-Directed Harm and Prohibited Abuse or Harm,” defines “primary
prevention” as the act of stopping self-directed harm and prohibited abusive or harmful acts from occurring. An
optimal prevention system, including those programs and personnel with equity in prevention of harmful
behaviors will sustain prevention-specific knowledge and skills, productive and collaborative relationships,
facilitate and institutionalize effective planning, execution, evaluation, and quality improvement of the
prevention system and activities. DoDI 6400.11 defines “climate” as the collection of shared attitudes and
perceptions of people within an organization or unit. Within the military, it often reflects leadership efforts to
build cohesion or trust among personnel.
At the MSAs, practices that may have once been highly effective in developing and implementing prevention
systems or activities have, in some cases, not kept pace with the changing characteristics of incoming students
at the academies. The OSIE teams observed that these practices may be having unintended consequences or
may be exacerbating unhealthy climate. While the MSAs have been diligent in adding prevention and support
resources over time, harmful behaviors will continue to increase until the climates and environment contributing
to that increased risk are modified. Although some common themes were identified across MSAs, the severity
4

or degree to which these findings were observed varied; thus, the recommendations draw out this nuance in
some cases and implementation of the recommendations will need to be tailored to the specific Academy
context.
The overarching findings and recommendations are divided into three areas based on the metrics in Table 3
with a consolidated list of cross-MSA recommendations in Appendix A and MSA specific findings and
recommendations provided in Appendix B: MSA Site Profiles and Supporting Data. Importantly, in some
cases, the MSA’s own internal assessments identified similar findings as the OSIEs and after the OSIE
visits MSAs moved out on actions to address the findings. In that event, implementation plans may
reflect those already implemented or planned efforts.

Prolonged Stress (assessed through metrics measuring protective environments)
Findings
•

Cadets and midshipmen are expected to fix
and police themselves regarding harmful
behaviors, but do not feel empowered or
prepared to do so

•

“Zero tolerance” is at odds with perceived
lack of accountability through the military
justice process and other means to address
inappropriate behaviors

•

Traditional peer leadership hierarchy has
unintended, unhealthy consequences (e.g.,
bullying, hazing, lack of connectedness)

•

Tactical Officer (TAC), and Air Officer
Commanding (AOC) skillsets are limited
which leads to cadets and midshipmen
seeking support elsewhere (Note:
Addressed through action directed by
Secretary Austin in March 10, 2023 memo;
thus, no corresponding OSIE
recommendation)

Recommendations
Immediate:
•

Allow cadets and midshipmen the time and
privacy required to seek and use mental health
care or other helping resources, as appropriate

•

Encourage and promote a range of mental
health and non-medical support services
available to cadets and midshipmen, such as
training, skill building, or other support services
that could be available prior to needing mental
health services

Intermediate:
•

Identify opportunities to increase transparency
of actions taken to prevent and hold individuals
appropriately accountable, where possible

•

Identify prominent misperceptions and mixed
messages; develop and disseminate countermessages supported by reinforcing actions to
address a perceived lack of accountability and
ensure that, to the extent possible, any
command or leadership communication align
with actions taken

Long Term:
•

Strengthen peer leadership structure
o [USAFA] Adjust the fourth-class system
and continue to deliberately develop
cadets throughout their 4-year journey
to stop instances of cadet hazing and
mistreatment
o

complementary leadership model
already exists, expand it to ensure
greater saturation of officers and
enlisted leaders amongst cadets and
midshipmen
o

[USMA] Review and enhance
preparation of peer leaders

Ensure MSA leadership have diversified
experiences in different Service training
environments to enhance cadet and
midshipman leadership development and
broaden their skillsets. [long term]

•

Cynicism, Distrust, and Stigma for Help-Seeking (assessed through metrics measuring stakeholder
engagement)
Findings
•

Concerns about collateral misconduct limit
reporting of sexual assault or harassment

•

There is stigma around behavioral health due
to the perceived impact on commissioning or
career field assignments

•

Distrust of systems and processes impacts
reporting and the use of resources

•

Cynicism is shaped by the perceived lack of
transparency in decisions and cyber
misinformation or bullying that threatens
protective environments

Recommendations
Immediate:
•

Review and expand, as necessary, current
Safe to Report policies to address collateral
misconduct when reporting egregious
violations and implement measures to ensure
all individuals are aware of Safe to Report
policies

•

Provide education or tools to cadets and
midshipmen that explain the facts of helpseeking including how and when it may affect
commissioning and career field assignments

Intermediate:
•

[OSD] Develop charter and institute a working
group for MSAs to address primary
prevention efforts, formulate interventions,
share evaluation outcomes, and strengthen
connections to other existing working groups

•

Develop and provide tools and information to
MSA leadership, cadets, and midshipmen to
identify and protect against cyber
misinformation and bullying that threaten
protective environments.

•

Evaluate long-standing traditions, systems,
processes, or internal policies that impede
necessary evidence-informed best practices
in prevention and address influencers who
seek to limit necessary changes,
cohesiveness, connectedness, and trust

•

[OSD] Examine commissioning standards
and accession waiver processes and develop
communication tools for educating cadets and
6

midshipmen on the policy and accession
waiver process, and dispel myths, where
possible, on the negative impact of mental
health treatment

Prevention Through the MSA Lifecycle (assessed through metrics measuring integrated prevention)
Findings
•

Efficacy of prevention activities can be
greatly diminished by an unhealthy climate

•

There is a lack of integration across
prevention, character development, and
leadership development efforts

•

Core curriculums do not include graded or
structured courses on the prevention of
harmful behaviors or developing leadership
practices to address harmful behaviors.

Recommendations
Immediate:
•

[Military Departments] Prioritize and
expedite the hiring of integrated prevention
personnel at MSAs and direct a “whole of
installation” approach that ensures continuity
with broader force structure and utilization of
all available prevention assets

Intermediate:
•

Fully integrate prevention, character
development, and leadership development
efforts within each MSA

•

Ensure services available at the MSAs are
integrated to support the entire military
community and not only the cadets and
midshipmen

Long Term:
•

[Military Departments] In collaboration with
the MSA, the Secretary of the Military
Department concerned will ensure all
incoming cadets and midshipmen receive a
deliberate and sustained education in
financial readiness, workplace
professionalism, time management, goal
setting, stress management, disappointment
tolerance, and other key skills to achieve the
leadership competencies outlined in DoDI
6400.11

Introduction
Every year each DoD MSA admits between 1,100 and 1,350 cadets and midshipmen with the goal of
developing those incoming cadets and midshipmen into valuable leaders of character committed to service and
defending the Nation. According to survey data from the 2021 Service Academy Gender Relations Survey
(SAGR) the estimated prevalence rates of unwanted sexual contact (USC) and sexual harassment (SH) have
increased steadily and significantly since 2014. According to a report released by RAND (2021), “Effects of
Sexual Assault and Sexual Harassment on Separation from the U.S. Military,” experiences of abuse and harm
early in cadet and midshipmen’s careers may have negative and lasting impacts and more must be done to
prevent harmful behaviors at the MSAs.
Secretary of Defense Lloyd Austin has made preventing these harmful behaviors one of his priorities. On
March 10, 2023, in response to the most recent survey data from the SAGR, Secretary Austin, directed the
Under Secretary for Personnel and Readiness (USD(P&R)) to conduct OSIEs at the MSAs to better evaluate
policy and capabilities intended to prevent self-directed harm and prohibited abusive or harmful acts and make
specific recommendations targeted to the local climate. This report reflects the results of the MSA OSIEs.

Methodology
The OSIEs focus on integrated primary prevention for the military community, and specifically for the purposes
of this report, the community at the MSAs. The definitions in Table 1 guided the methods used to identify sites
and develop metrics.
Table 1: Risk and Protective Factors in DEOCS 5.0

Primary
Prevention

Integrated
Prevention

Stopping harmful acts before they occur. Can be implemented for an entire group or
population without regard to risk (universal primary prevention) or can be implemented for
individuals, groups, or a population that is at risk (selected primary prevention).
Primary prevention activities can target:
1. Influencers, such as leaders who set a climate and shape norms, but may not be
present when harmful acts occur can be those outside of the agency who also set or
maintain traditions, influence policy, or funding (i.e. alumni groups, senior leaders)
2. Bystanders, who may be present when harmful acts occur;
3. Individuals, who may commit harmful acts; or,
4. Individuals who may be affected by harmful acts.
Taking action to decrease harmful behaviors and lessen the chances of these behaviors
negatively impacting readiness and retention in a way that:
1. Incorporates values of inclusivity, connectedness, dignity and respect (access, equity,
rights, and participation)—including the elevation of Service member and family
member voice—to inform plans, processes, and trainings;
2. Recognizes and adjusts plans, processes, and trainings to consider and be responsive
to climate issues and populations that have been disproportionately impacted by
harmful acts;
3. Intentionally seeks to align and find common operating principles across prevention
efforts and offices (e.g., equal opportunity, suicide, SA); and,
4. Incorporates multiple lines of effort across individual, interpersonal, organizational
ecological levels.

Identification of OSIE Sites
The DoD annually assesses the MSAs for progress made toward combating USC and SH1. In the academic
year (APY) 2021-2022 DoD Annual Report on SH and Violence at Military Service Academies, utilizing the
SAGR, an estimated 21.4 percent of MSA women and 4.4 percent of MSA men indicated experiencing
unwanted sexual contact in the year prior to the Service Academy Gender Relations (SAGR) survey. Based
on these rates, the DoD estimates that 1,136 MSA men and women may have experienced some form of
unwanted sexual contact in the year prior to being surveyed. Figure 1 represents the estimated rates of
unwanted sexual contact in the year prior to the survey by MSA.
The SAGR identified a similar disturbing trend in the prevalence of SH at the MSAs with 63 percent of MSA
women and 20 percent of MSA men who may have experienced an incident of SH in APY 2021-2022. This
equates to 2,127 women and 1,813 men who experienced SH in APY 2021-2022. This was a significant
increase from 50 percent and 16 percent, respectively, from APY 2017-2018. Figure 2 represents the change
in estimated prevalence rates of SH from 2016 - 2022. In addition to population-based surveys, more recent
MSA DEOCS indicate more cadets and midshipmen at the MSAs may have experienced sexually harassing
behaviors than respondents at most other DoD installations and ships.
Figure 1: Unwanted Sexual Contact Prior Year Prevalence Estimates by MSA 2006-2022

DoD Annual Report on SH and Violence at Military Service Academies, Academic Program Year 2021-2022, March 10, 2023.

Figure 2: Estimated Rates of SH at MSAs

The primary data source for selecting units of focus for the MSA OSIEs was the DEOCS. The MSAs
completed the DEOCS in late 2022, which provided the most timely and sensitive measure of command
climate available. The redesigned DEOCS is comprised of 19 factors, nine of which depict risk factors and 10
of which depict protective factors for readiness detracting behaviors, such as sexual assault, harassment, and
suicide. For the purposes of this analysis ratings for transformational leadership, passive leadership, and toxic
leadership were treated as separate risk factors for the unit/organization leader, commander, and the Senior
NCO, if applicable. As a result, this analysis includes the 22 total factors found in Table 2.
Table 2: Risk and Protective Factors in DEOCS 5.0

DEOCS 5.0 Risk Factors

DEOCS 5.0 Protective Factors

Alcohol Impairing Memory

Cohesion

Binge Drinking

Connectedness

Stress

Engagement and Commitment

Passive Leadership

Fairness

Toxic Leadership

Inclusion

Racially Harassing Behaviors

Morale

Sexually Harassing Behaviors

Safe Storage for Lethal Means

Sexist Behaviors

Work-Life Balance

Workplace Hostility

Leadership Support
Transformational Leadership

These factors are the same for MSAs and other military installations; however, the leadership referenced for
this analysis reflects the MSA population. MSA leadership in “leadership support” and “toxic leadership” refers
to the first cadet or midshipman within an individual’s chain of command, and leadership in “transformational
leadership” and “passive leadership” refers to the company or squadron permanent party MSA command
team.

Integrated Prevention Assessment Methods:
In 2021, OUSD(P&R), in collaboration with RAND, identified nine dimensions to guide the assessment of
prevention capabilities for the OSIEs. These dimensions were identified by an analysis of the focus areas not
11

covered by existing DoD compliance checklists and DoD assessment tools to enforce relevant prevention
policies and the OSIE framework outlined in the 2021 OSIE Report.
OUSD(P&R) prioritized three domains of focus for the development of new metrics:
•

Healthy & Protective Environment: Research shows that command climates can positively or
negatively impact behaviors such as SA and harassment.

•

Integrated Prevention: Effective prevention targets a mix of risk and protective factors that are both
common across problem areas as well as unique to specific harmful behaviors.

•

Stakeholder Engagement: Outcomes can be improved when multiple stakeholders have genuine
involvement in prevention activities.

Three additional domains were added from the OSIE framework:
•

Priority: Higher-level leadership sets the tone and sustains consistent focus on harmful behaviors

•

Preparation: Prevention personnel and intermediate leadership are equipped with the ability, and exist
within a structure, that incentivizes and supports addressing harmful behaviors

•

Implementation: Approach aligns with best practices and is done well (i.e., with high quality)

Crossing the three domains from OSIE framework with the three domains (i.e., focus area) in existing
compliance checklists and assessment tools yielded a matrix of nine dimensions in Table 3 to be included in
the assessment.
Table 3: Prevention Capabilities Assessed in OSIEs

PRIORITY

PREPARATION

HEALTHY &
PROTECTIVE
ENVIRONMENT

Leaders prioritize
fostering a
protective
environment by
their actions and
communications.

Leaders have the
requisite knowledge,
skills, abilities (KSAs)
and access to training to
develop those KSAs.

Leaders employ practices
known to support a
protective environment

INTEGRATED
PREVENTION

Leaders prioritize
prevention
activities.

Leaders and prevention
personnel have the
requisite KSAs to carry
out prevention
successfully.

Prevention activities that
target risk and protective
factors across multiple
negative behaviors are
evaluated.

SERVICE
MEMBER
ENGAGEMENT

FOCUS AREAS

OSIE FRAMEWORK AREA
IMPLEMENTATION

Leaders prioritize
engaging
stakeholders.

Prevention personnel
have the resources and
requisite KSAs to
engage stakeholders
effectively.

Stakeholders are
genuinely engaged in
prevention activities
across multiple planning
stages.

To assess these nine dimensions, SVTs collected information from various personnel during each OSIE.
Using all information collected, the site team made binary ratings on a series of data elements (present or
absent), which were combined to establish whether various subdimensions were sufficient. A maturity score
was then calculated for each dimension. A maturity score represents a progression and achievement in a
particular domain or discipline so that a higher score suggests more advanced practice on agreed upon
standards. The maturity scores on the nine dimensions were informed by the number of sufficient
corresponding subdimensions. More details on the development, validation, and application of these metrics
are found in Appendix C. Scores on the nine dimensions and sub-dimensions for each site are found in Table
14 in Appendix B.

On-Site Evaluations
Teams comprised of DoD civilian employees, military members, and contracted research assistants conducted
site visits that spanned several days and included focus group interviews and surveys across all helping
agencies (integrated prevention personnel workforce, faculty, staff, administration personnel, etc.), leadership,
and students (cadets and/or midshipmen). These focus groups and surveys collected data on prevention
efforts related to harmful behaviors such as discrimination, sexual assault, harassment, retaliation, suicide, and
intimate partner violence.
For the 2023 MSA OSIEs, several changes/improvements were made to the OSIE based on lessons learned
from the inaugural OSIE conducted in 2021 at installations across the military, as identified by the Secretary of
Defense. Pre-visit process improvements included shifting the OSIE timeframe to late winter/early spring to
reduce the impact of staff rotation (PCS), reducing the complexity of the pre-visit data collection requests, and
providing earlier and more detailed advance notice of the site visit and more collaborative planning with the
local site visit teams.
Several improvements and modifications were made from the 2021 OSIEs to better pertain to the 2023 MSA
OSIEs which included adjusting the language and flow of the discussion protocolss to improve clarity and
simplicity. Improvements were also made to scoring and reporting, including the development of processes to
organize notes to better align with scoring. Finally, two additional discussion protocols were developed for the
MSAs: one for fourth- and third-class year cadets/midshipmen and one for second- and first-class year cadets
and midshipmen. The scenario used for the tabletop exercise (TTX) was adapted to increase relevance to the
MSA mission.
Units were selected if they had elevated scores on multiple risk factors or elevated scores on multiple
protective factors. Both high risk and high protective units were selected to identify best practices as well as
gaps. Consistency of findings across these groups also provides clues about whether issues highlighted by
the units were widespread across the MSA, such that evidence for issues was found in both at risk and
protective units; or, if issues may be localized and evidence of specific issues was found only in those groups.
Each OSIE multi-disciplinary evaluation team included a Senior Executive or GS-15 team lead and seven staff
representing the Office of the Under Secretary of Defense for Personnel and Readiness (Office of Force
Resiliency, Office of Diversity, Equity and Inclusion, Defense Suicide Prevention Office, Sexual Assault
Prevention and Response Office, Office of People Analytics, and the Diversity Management Operation Center).
The teams also included representation from the Military Services (military and civilian employees), and the
National Guard Bureau. The Service members assigned to each OSIE team acted as a Senior Subject Matter
Expert (SME) and came from the same Military Department with jurisdiction over the MSA being evaluated.
This allowed for a mixture of military perspectives and insight into MSA culture. Research assistants were also
present during focus group interviews to collect interviewee responses allowing team members to engage with
focus group participants in a fluid manner.

Findings and Recommendations
The OSIE team developed an overarching conceptual framework as context for the findings and
recommendations. As outlined in Figure 5, this conceptual framework describes the changing needs of the
newest cohorts of cadets and midshipmen and how existing leadership structures need to evolve to meet these
needs and mitigate risk for harmful behaviors.
Survey data from the Centers for Disease Control and Prevention (CDC) suggests that young people are
experiencing or showing signs of harmful behaviors prior to entering the MSAs. Specifically, to the 2021 CDC
Youth Risk Behavior Survey 2 found that 8.5 percent of high school students had been physically forced to have
sexual intercourse at some point in their lives while 11 percent had experienced sexual violence 3 in the 12
months prior to the survey. The survey also found that, during the 12 months prior to the survey, 42.3 percent
had felt sad or hopeless, 22.2 percent had seriously considered attempting suicide, 10.2 percent had
attempted suicide, and 2.9 percent had an attempt that resulted in an injury, poisoning, or overdose that
required medical treatment. According to the SAGR, the number of incoming cadets and midshipmen who
indicated that they may have experienced USC at any time prior to entering the MSA was 33 percent, up from
25.1 percent in 2018, and the number of those incoming cadets and midshipmen who indicated that they may
have experienced SH in the past year was 63%, up from 50% in 2018.
A prior sexual assault or SH elevates the risk of experiencing USC in the future and previous trauma can affect
how individuals assess and respond to stressors and harmful behaviors 4. The MSAs have established
prevention resources and systems at their institutions to help cadets and midshipmen deal with and prevent
harmful behaviors; however, the 2023 MSA OSIEs demonstrated that the training environment and overall
climate across the MSAs can affect the success of prevention activities. For example, if cadets and
midshipmen receive conflicting messages from their leadership, peers, or other members of the MSA
community about accountability, help-seeking, and expectations/norms related to how they treat each other
from social media, the impact of their senior leader’s communications could be undermined. Such mixed
messages fuel distrust, cynicism, and stigma for help-seeking. Even if this unhealthy climate is generated by
misperceptions, it influences cadets and midshipmen behavior, what and how they seek help, and how they
behave and hold each other accountable. The goal of the MSAs is to educate, train, and inspire
cadets/midshipmen to become leaders of character; however, some aspects of the current training
environment and climate need to be adjusted to meet this goal given changing needs of incoming students.

Centers for Disease Control and Prevention. (2021). High School Youth Risk Behavior Survey. Department of Health and Human
Services. https://nccd.cdc.gov/Youthonline/App/Default.aspx
3
Defined as “such things as kissing, touching, or being forced to have sexual intercourse that they did not want to do one more
times during the 12 months before the survey.”
4
Rosellini, A. J., Street, A. E., Ursano, R. J., Chiu, W. T., Heeringa, S. G., Monahan, J., Naifeh, J. A., Petukhova, M. V., Reis, B. Y.,
Sampson, N. A., Bliese, P. D., Stein, M. B., Zaslavsky, A. M., & Kessler, R. C. (2017). Sexual Assault Victimization and Mental Health
Treatment, Suicide Attempts, and Career Outcomes Among Women in the US Army. American journal of public health, 107(5), 732–
739. https://doi.org/10.2105/AJPH.2017.303693
2

Figure 3: MSA Leadership Development

Incoming students: Per a national shift among young people 5, incoming cadets and midshipmen are more
likely to have trauma that occurred before entering the MSAs. This trauma increases the risk of harmful
behaviors even before the cadets begin the highly stressful training environment at the MSAs. As cited above,
there is a greater focus on identifying the behavioral health needs of young people and highlighting the
differences in perceivable healthy coping skills compared to previous years and generations.
Prolonged stress training environments: While some amount of stress is inherent to the MSA, military
training, and educational experiences, the structure within the MSAs and the skills of those in leadership
positions must be equipped to mitigate that stress. Our visits indicate that in some cases peer leaders (i.e.,
company officers) were unprepared to support the cadets and midshipmen they led and in other cases, the
peer leadership structure created unhealthy power dynamics in which cadets and midshipmen are hesitant or
fearful of intervening in or reporting harmful behaviors. Additionally, in some cases, inadequately prepared
AOCs and TAC officers also fail to meet the needs of incoming students.
Climate of cynicism, distrust, and stigma: Unchecked influencers and social media communications, to
include the chat app Jodel, send mixed messages among the cadet/midshipmen population and can lead to
misperceptions about the MSAs prevention efforts, the judicial system, the value of reporting, and prevention
support resources, and dissuade those at the MSA from seeking the help they need.
Leaders of character: Cadets and midshipmen experiencing these risk factors and climate throughout their
time in a MSA may embark on their military careers with unaddressed trauma and an unhealthy perspective of
military training and the importance of prevention.

MSA Strengths
During the MSA OSIEs several strengths were found at each institution. These strengths demonstrate that the
MSAs are positioned to make progress if the factors causing prolonged stress and the climate of cynicism are
adequately addressed. These strengths are broken out between MSA climate and MSA prevention efforts in
Table 4.

U.S. Surgeon General. Protecting Youth Mental Health. 2021.

Table 4: MSA Strengths

USMA
• Leaders generally
model and reward
good behavior
• Cadets, Service
members, faculty,
and staff recognize
and work to address
shared risk factors
Climate

• Broad appreciation
and trust for Tactical
NCOs supports
problem solving and
helps prevent crises
• Embedded Military
and Family Life
Counselors are
widely viewed as an
asset

• Referrals to help for
those in need occur
on a regular basis

Prevention

• Active data and
information sharing
forums support
integration and
awareness
• Addressing SH/SA;
Creating Healthy
Climates; and
Tackling Holistic
Health (ACT) cadets
are helpful peer-level
prevention assets

USNA
• Working groups (WG)
including midshipmen

•

o Prevention WG

o Midshipmen Affairs
Team
o Midshipmen
Leadership Team

• Current Commandant
and Deputy
Commandant

•

USAFA
Peer groups (i.e.,
Teal Ropes) and
affinity and culture
clubs (i.e., Hispanic
heritage) support
connection and
belongingness
Faculty’s
commitment to the
cadets

o Culture of listening
o Mailbox for
anonymous
communication

o Promoting unity and
cohesion

• Prevention personnel
take their roles seriously
o Interested in better
integration
o Interested in
furthering their
knowledge of
prevention subject
areas

• Numerous researchbased prevention
activities underway
• Highly regarded athletic
department Healthy
Relationships training
• Reduced access to
alcohol and other factors
that increase risk

MSA Maturity Scores
Each MSA was scored based on nine dimensions. As implemented, this maturity model serves three
purposes: it allows DoD and others to understand the current capabilities of the sites, it helps sites identify
ways to strengthen their prevention efforts, and it permits comparison, both within and across sites. The
scores are based on a scale of 0 to 5, with a higher score indicating that the SVT found consistent evidence of
the metric across engagements and other data sources.
The scores for each MSA across the nine dimensions are included in Table 5 below. Additional subdimension
and dimension data is available in Table 14 in Appendix B.
16

Table 5: MSA Maturity Scores

Priority
Does leadership
consistently prioritize…

Protective
Environment
Integrated
Prevention
Stakeholder
Engagement

Preparation
Are leaders and
prevention staff
equipped and
empowered to…
USMA USNA USAFA USMA USNA USAFA

Implementation
Are efforts implemented
with quality and
seamlessly integrated…
USMA USNA USAFA

Findings and Recommendations Across Three MSAs
Despite the strengths that each of the MSAs demonstrate in its respective climates and prevention activities,
OSIE teams identified areas for improvement. Consistently the OSIE teams found that cadets and
midshipmen had misperceptions about their senior leader’s intent, policy, processes, and acceptable norms.
The misperceptions seemed to be generated by uninformed influencers 6; and the leaders best positioned to
dispel misperceived norms – peer leaders (e.g., company officers) and TAC officers/AOCs – were unprepared
to adequately address the misinformation or in some cases were the source of the misinformation themselves
(e.g., lack of established norms across units result in misinformation being shared amongst peers). Even if this
unhealthy climate is generated by misperceptions, it influences cadet and midshipmen behavior, what and how
they seek help, and how they behave and hold each other accountable.
While clear communication and understanding was identified in the 2021 OSIEs as a consistent challenge
across sites, at the MSAs it is imperative to ensure those leaders closest to the cadets and midshipmen are
equipped and modeling healthy norms and behaviors. The findings and recommendations focus on these
areas and are steps the MSAs can take to address these areas.
Of note, although some common themes were identified across MSAs, the severity or degree to which these
findings were observed varied; thus, the recommendations draw out this nuance in some cases and
implementation of the recommendations will need to be tailored to the specific Academy context. Importantly,
in some cases, the MSA’s own internal assessments identified similar findings as the OSIEs and after
the OSIE visits MSAs moved out on actions to address the findings. In that event, implementation
plans may reflect those already implemented or planned efforts.

Prolonged Stress (assessed through metrics measuring protective environments)
Some level of stress in the MSA training environment is to be expected; however, to help cadets and
midshipmen effectively handle that stress, leaders, particularly those directly overseeing cadets and
midshipmen – peer leaders (e.g., company officers), TAC officers/AOCs –must be adequately prepared to
address issues relating to prevention of harmful behaviors. The following table shows the findings and
recommendations that relate to the feeling of prolonged stress and contributing factors within the MSA
environment. These findings were consistent across all three MSAs and were assessed through the metrics
measuring protective environments in Table 5 and Appendix C .

Influencers is defined in Table 1 above “such as leaders who set a climate and shape norms, but may not be present when harmful
acts occur can be those outside of the agency who also set or maintain traditions, influence policy, or funding (i.e. alumni groups,
senior leaders)”

Table 6: Findings & Recommendations for Prolonged Stress Across All Three MSAs

PROLONGED STRESS ACROSS ALL THREE MSAS
Findings
Recommendations
Strengthen peer leadership structure [long
term]

•

• The traditional peer leadership hierarchy has
unintended, unhealthy consequences (e.g.,
bullying, hazing, lack of connectedness)
• TAC and AOC skillsets are limited which
pushes cadets and midshipmen to seek
support elsewhere (Note: Addressed through
action directed by Secretary Austin in March
10, 2023 memo; thus, no corresponding
OSIE recommendation)

[USAFA] Adjust the fourth-class system
and continue to deliberately develop
cadets throughout their four-year journey
to stop instances of cadet hazing and
mistreatment

[USNA/USAFA] In order to provide more
supervision and learning/modeling
opportunities, complement the peer
leadership structure with additional noncadet/midshipman leaders, including
officers and non-commissioned officers
(NCOs) who have experience leading
entry level Service members; where this
complementary leadership model already
exists, expand it to ensure greater
saturation of officers and enlisted leaders
amongst cadets and midshipmen

o [USMA] Review and enhance preparation
of peer leaders

• “Zero tolerance” is at odds with the perceived
lack of accountability in the military justice
process, to include cases that do not
necessarily meet the threshold for a criminal
case, and other methods meant to address
inappropriate behavior

• Cadets and midshipmen are expected to fix
and police themselves regarding harmful
behaviors, but do not feel empowered or
prepared to do so

•

Ensure MSA leadership have diversified
experiences in different Service training
environments to enhance cadet and
midshipman leadership development and
broaden their skillsets. [long term]

•

Identify opportunities to increase
transparency of actions taken to prevent
and hold individuals appropriately
accountable, where possible [intermediate]

•

Allow cadets and midshipmen the time and
privacy required to seek and use mental
health care or other helping resources, as
appropriate [immediate]

•

Encourage and promote a range of mental
health and non-medical support services
available to cadets and midshipmen, such
as training, skill building, or other support
services that could be available prior to
needing mental health services
[immediate]

•

Identify prominent misperceptions and
mixed messages; develop and disseminate
counter-messages supported by reinforcing
actions to address a perceived lack of
accountability and ensure that, to the extent
possible, any command or leadership
communication align with actions taken
[intermediate]

Cynicism, Distrust, Stigma for Help-Seeking (assessed through stakeholder engagement metrics)
The following table shows the findings and corresponding recommendations that relate directly to the cynicism,
distrust, and stigma for help-seeking widespread in the MSA environment. These findings were consistent
across all three MSAs and were assessed through the metrics measuring stakeholder engagement in Table 5
and Appendix C. Recommendations are for all MSAs unless otherwise noted.
Table 7: Findings & Recommendations to Address Cynicism, Distrust, and Stigma for Help-Seeking Across All Three MSAs

CYNICISM, DISTRUST AND STIGMA FOR HELP-SEEKING ACROSS ALL THREE MSAS
Findings

Recommendations

• Distrust of systems and processes impact the
use of prevention resources and reporting of
harmful behaviors

• Evaluate long-standing traditions, systems,
processes, or internal policies that impede
necessary evidence-informed best practices in
prevention and address influencers who seek to
limit necessary changes, cohesiveness,
connectedness, and trust [intermediate]

• Concerns about collateral misconduct limit
reporting of harmful behaviors

• Review and expand, as necessary, current Safe
to Report policies to address collateral
misconduct when reporting egregious violations
and implement measures to ensure all
individuals are aware of Safe to Report policies
[immediate]

• Cynicism is shaped by the perceived lack of
transparency in decisions and cyber
misinformation or bullying that threatens
protective environments

• Develop and provide tools and information to
MSA leadership, cadets, and midshipmen to
identify and protect against cyber misinformation
and bullying that threaten protective
environments [intermediate]

• Stigma around behavioral health due to the
perceived impact on commissioning and career
field assignments

• [OSD] Examine commissioning standards and
accession waiver processes and develop
communication tools for educating cadets and
midshipmen on the policy and accession waiver
process, and dispel myths, where possible, on
the negative impact of mental health treatment
[intermediate]

Prevention Throughout the MSA Lifecycle (assessed through integrated prevention metrics)
While prolonged stress and cynicism amongst the MSA population hinder MSA prevention efforts from
reaching their full potential, it is still critical to ensure that prevention efforts are constantly improving, aligned
with best practice, and meet the needs of the MSA population. The following table shows the findings and
recommendations that relate to prevention efforts throughout the entire MSA lifecycle. These findings were
consistent across all three MSAs and were assessed through the metrics measuring integrated prevention in
Table 5 and Appendix C.
Table 8: Findings & Recommendations to Improve Prevention Throughout the MSA Lifecycle at All Three MSAs

PREVENTION THROUGHOUT THE MSA LIFECYCLE
Findings

Recommendations
• Ensure services available at the MSAs are
integrated to support the entire military community
and not only the cadets/midshipmen
[intermediate]

• Efficacy of prevention activities can be greatly
diminished by an unhealthy climate

• [Military Departments] Prioritize and expedite the
hiring of integrated prevention personnel at MSAs
and direct a “whole of installation” approach that
ensures continuity with broader force structure
and utilization of all available prevention assets
[immediate]

• There is a lack of integration across prevention,
character development, and leadership
development efforts

• Fully integrate prevention, character development,
and leadership development efforts within each
MSA [intermediate]

• Core curriculums do not include graded or
structured courses on the prevention of harmful
behaviors or developing leadership practices to
address harmful behaviors.

• [Military Departments] In collaboration with the
MSA, the Secretary of the Military Department
concerned will ensure all incoming cadets and
midshipmen receive a deliberate and sustained
education in financial readiness, workplace
professionalism, time management, goal setting,
stress management, disappointment tolerance,
and other key skills to achieve the leadership
competencies outlined in DoDI 6400.11 [long
term]

Additional Findings and Recommendations for Individuals MSAs
While the findings and recommendations in the above section cut across the three MSAs, there were certain
findings that applied more specifically to individual MSAs with individualized recommendations to address
them. These findings and recommendations are based on metrics measuring protective environments,
stakeholder engagement, and integrated prevention in Table 5. Additional context for each MSA can be found
in Appendix B.

USMA
The following table includes the OSIE findings and recommendations for the USMA. All recommendations in
this section are for the USMA unless otherwise noted. Findings without an accompanying recommendation
are addressed in the cross-MSA recommendations in Tables 6-8.
20

Table 9: USMA Specific Findings and Recommendations

Findings

Recommendations

• There is inadequate physical security of barracks
rooms which increases opportunities for harmful
behaviors to occur
• ACT cadets are good sources of information and
referral, but cadets expressed a desire for more
subject matter expert delivered training that
reflects the seriousness of prevention issues
• NCOs are a strength

• Repair and maintain barracks rooms locks to
ensure physical safety [immediate]
• Provide evidence-informed small group training led
by subject matter experts on harmful behaviors
and prevention [long-term]
• Ensure NCOs are sufficient in number and
represent diverse backgrounds will continue to
encourage cadet engagement [long-term]

USNA
The following table includes the OSIE findings and recommendations for the USNA. All recommendations are
for the USNA unless otherwise noted. Findings without an accompanying recommendation are address in the
cross-MSA recommendations in Tables 6-8.
Table 10: USNA Specific Findings & Recommendations

Findings
• Subjective peer ranking system gives
midshipmen the perception that midshipmen
rankings are based on popularity rather than
capability, contribution, or achievement

Recommendations
• Address misperceptions of class ranking criteria
[intermediate]

• Multiple midshipmen interviewed during the site
visit believed they need an escort to seek care
• Multiple midshipmen interviewed during the site
visit reported transportation to care outside of
USNA was either unavailable or difficult to
access in a timely fashion
• Early intervention mental health programs are
understaffed, and traditional business hours
negatively impact use by midshipmen
• Perception among midshipmen that influencers
outside of USNA, including USNA alumni, have
unspoken control over attitudes, behaviors, and
outcomes
• There is a general lack of comprehensive,
integrated, research-based prevention

USAFA
The following table includes the OSIE findings and recommendations for the USAFA. All recommendations
are for the USAFA unless otherwise noted. Findings without an accompanying recommendation are
21

addressed in the cross-MSA recommendations in Tables 6-8 above and many recommendations are also
captured in USAFA’s newly launched “Let’s Be Clear” campaign.
Table 11: USAFA Specific Findings & Recommendations

Findings
• Cadets who participated in the focus groups
characterize the USAFA campus as a prison
environment

Recommendations
• [Department of the Air Force] Enhance
oversight to be commensurate with other Air
Force major commands [intermediate]

Multiple fourth-class cadets interviewed
during the site visit believed they needed an
escort to seek physical or mental health care
(including the Peak Performance Center)

• Review evaluation of the Enhanced Access,
Acknowledge, Act Program and recommend how
to address countering the negative experience of
female cadets [intermediate]

Multiple cadets interviewed during the site
visit reported transportation to care outside
of USAFA was either unavailable or difficult
to access in a timely fashion

• Evaluate if the athletic department’s Healthy
Relationships training would be valuable for the
entire cadet wing [intermediate]

Multiple cadets interviewed perceived AOC
priorities as not appropriately considering
cadet needs (e.g., sick days, mental health
days, outside assistance)

•

Cadets are becoming, or have become,
desensitized to conditions likely to be perceived
as unacceptable in other military contexts (i.e.,
unprofessional leadership styles, hazing,
harassment, or other behaviors that disrupt
training events)

•

There is a misperception that programing
through the Military and Family Readiness
Center (MFRC) does not apply to the cadet
wing
o

• [OSD] Conduct a follow-up visit at USAFA in 2024
[intermediate]

The full gamut of support services at the
MFRC are not offered to cadets, are
identified as unnecessary for cadets (i.e.
Family Advocacy Program relationship
trainings), or a duplicative effort is instituted
in the cadet wing (i.e., financial readiness
program)

OSD Recommendations
To help address the findings from the MSA OSIEs and to assist the Military Departments and MSAs in
enacting meaningful change to reduce risk factors and enhance prevention efforts, OSD should:
•

•

Hold Deputy Secretary of Defense meetings twice per year to track and enforce implementation of the
approved Independent Review Commission on Sexual Assault in the Military and the Suicide
Prevention and Response Independent Review Commission. This will ensure implementation of
actions to address real and perceived lack of accountability and victim support across the MSAs.
The Under Secretary of Defense for Personnel and Readiness convene a working group to develop
implement plans on the actions directed by the SD in response to the 2023 MSA OSIEs. This working
22

group should consist of representatives from all three MSAs and should meet regularly until final
implementation plans are finalized and adopted.

Conclusion
The 2023 MSA OSIEs provided an opportunity for OSD to gain further insight into the risk and protective
factors at each MSA, including identifying what is working and what is not. While the MSAs have made
advancements in their efforts to prevent harmful behaviors, there remain significant hurdles to achieving
progress that leadership at all levels is determined to achieve. Changes are necessary to address the issues
that are prolonging and intensifying cadet and midshipman stress beyond what would be considered
acceptable in many other military contexts. Some of these changes, such as addressing the leadership
structure and better preparing peer leaders will mitigate challenges real or perceived, experienced by the MSA
communities.
The climate of cynicism, distrust, and stigma must be addressed at all levels of the MSA environment.
Effective prevention efforts and investments will be thwarted if the cadets and midshipmen continue to have
pervasive distrust of those efforts and cynicism that seeking help or reporting harmful behaviors will lead to
meaningful positive change. Certain changes, like greater communication and transparency, may be changes
that can be quickly implemented, while others, such as strengthening the peer leadership structure will require
bold, longer-term action. All parties, from the cadets and midshipmen themselves to the highest levels of
leadership, benefit from an environment built on trust and healthy behaviors. This ensures that leaders of
character are the next generation of military leadership and positive healthy behaviors will be rewarded and
expected at all levels.

Appendix A: Consolidated Cross-MSA Recommendations
Tables six to eight of the report identify findings and accompanying recommendations for three focus areas.
The following consolidates those cross-MSA recommendations and groups them by the recommended
implementation timeline. The recommendations apply to all three MSAs unless otherwise noted.

Immediate
•

[OSD] Hold Deputy Secretary of Defense meetings twice per year that will track and enforce
implementation of the approved Independent Review Commission on Sexual Assault in the Military and the
Suicide Prevention and Response Independent Review Commission. This will ensure implementation of
actions to address real and perceived lack of accountability and victim support across DoD and the MSAs.

•

[OSD] The Undersecretary of Defense for Personnel and Readiness convene a working group to develop
implement plans on the actions directed by the SD in response to the 2023 MSA OSIEs. This working
group should consist of representatives from all three MSAs and should meet regularly until final
implementation plans are finalized and adopted.

•

[Military Departments] Prioritize and expedite the hiring of integrated prevention personnel at MSAs and
direct a “whole of installation” approach that ensures continuity with broader force structure and utilization
of all available prevention assets

•

Allow cadets and midshipmen the time and privacy required to seek and use mental health care or other
helping resources, as appropriate

•

Encourage and promote a range of mental health and non-medical support services available to cadets
and midshipmen, such as training, skill building, or other support services that could be available prior to
needing mental health services

•

Review and expand, as necessary, current Safe to Report policies to address collateral misconduct when
reporting egregious violations and implement measures to ensure all individuals are aware of Safe to
Report policies

•

[USMA] Repair and maintain barracks rooms locks to ensure physical safety

Intermediate
•

[OSD] Examine commissioning standards and accession waiver processes and develop communication
tools for educating cadets and midshipmen on the policy and accession waiver process, and dispel myths,
where possible, on the negative impact of mental health treatment

•

Identify opportunities to increase transparency of actions taken to prevent and hold individuals
appropriately accountable, where possible.

•

Identify prominent misperceptions and mixed messages; develop and disseminate counter-messages
supported by reinforcing actions to address a perceived lack of accountability and ensure that, to the extent
possible, any command or leadership communication align with actions taken

•

Evaluate long-standing traditions, systems, processes, or internal policies that impede necessary evidenceinformed best practices in prevention and address influencers who seek to limit necessary changes,
cohesiveness, connectedness, and trust

•

Develop and provide tools and information to MSA leadership, cadets, and midshipmen to identify and
protect against cyber misinformation and bullying that threaten protective environments

•

Fully integrate prevention, character development, and leadership development efforts within each MSA

•

Ensure services available at the MSAs and their home installations are integrated to support the entire
military community

•

[OSD] Conduct a follow-up visit at USAFA in 2024

•

[Department of the Air Force] Enhance oversight to USAFA to be commensurate with other Air Force
major commands

•

[USAFA] Review evaluation of the Enhanced Access, Acknowledge, Act Program and recommend how to
address countering the negative experience of female cadets

•

[USAFA] Evaluate if the athletic department’s Healthy Relationships training would be valuable for the
entire cadet wing

•

[USNA] Address misperceptions of class ranking criteria

Long Term
•

[Military Departments] In collaboration with the MSA, the Secretary of the Military Department will ensure
all incoming cadets and midshipmen receive a deliberate and sustained education in financial readiness,
workplace professionalism, time management, goal setting, stress management, disappointment tolerance,
and other key skills to achieve the leadership competencies outlined in DoDI 6400.11

•

Strengthen peer leadership structure
o

[[USAFA] Adjust the fourth-class system and continue to deliberately develop cadets
throughout their four-year journey to stop instances of cadet hazing and mistreatment

[USNA/USAFA] In order to provide more supervision and learning/modeling opportunities,
complement the peer leadership structure with additional non-cadet/midshipman leaders,
including officers and non-commissioned officers (NCOs) who have experience leading entry
level Service members; where this complementary leadership model already exists, expand it to
ensure greater saturation of officers and enlisted leaders amongst cadets and midshipmen

[USMA] Review and enhance preparation of peer leaders

•

Ensure MSA leadership have diversified experiences in different Service training environments to enhance
cadet and midshipman leadership development and broaden their skillsets

•

[USMA] Provide evidence-informed small group training led by subject matter experts on harmful
behaviors and prevention

•

[USMA] Ensure NCOs are sufficient in number and represent diverse backgrounds will continue to
encourage cadet engagement

Appendix B: MSA Site Profiles and Supporting Data
Sites Identified
The following table summarizes the sites identified by OSIEs and the units of interest that participated in the
OSIE. Units of interest are the units within the MSA that had the highest risk or protective percentile scores
among units at each MSA. The most recent DEOCS scores drove the risk and protective factors which
included contextual factors and command climate analysis. These units were selected to provide a better
understanding of the factors contributing to increased USC, SH, and suicide rates and necessary preventative
measures to mitigate those factors.
Table 12: MSA Units Identified for OSIEs
Military Service Academy
U.S. Military Academy*

Units with High DEOCS Protective Factors
- C Co., 4th Regiment
- I Co., 3rd Regiment
- U.S. Corps of Cadets Faculty and Staff

Units with High DEOCS Risk Factors
- A Co., 4th Regiment
- B Co., 2nd Regiment
- G Co., 3rd Regiment
- F Co., 1st Regiment
U.S. Naval Academy
- Co. 13, 3rd Battalion
− Co. 2, 1st Battalion
− Co. 11, 3rd Battalion
- Co. 20, 4th Battalion
- Co. 22, 5th Battalion
− Co. 19, 4th Battalion
− Co. 27, 6th Battalion
U.S. Air Force Academy
- Cadet Squadron 15
- Cadet Squadron 7
- Cadet Squadron 30
- Cadet Squadron 20
- Cadet Squadron 24
- Cadet Squadron 40
*Units from the garrison were also assessed at USMA; findings and recommendations for those units are not included in this report.

MSA Risk Factors
USC Factors
Figure 4 incorporates estimates from the SAGR and compares those estimates to the number of reports made
by cadets and midshipmen for events occurring during that APY.

Figure 4: Estimated Cadets and Midshipmen Experiencing Unwanted Sexual Contact Based on Past-Year Prevalence Rates versus
Number of Cadets and Midshipmen in Reports of Sexual Assault Made during the APY, for Incidents Occurring During Military Service

SH Factors
The number of cadets and midshipmen who may have experienced unwanted sexual contact and SH has
increased over time. Cadets and midshipmen are more likely to experience unwanted sexual contact and SH,
compared to active component Service members of the same age. Figure 5 represents the total number of SH
complaints by MSA while Figure 6 represents the estimated past year SH rate for each MSA compared to rates
in 2016 and 2018.

Source: Defense Sexual Assault Incident Database and SAGR data

Figure 5: SH Complaints by APY - All MSAs

Figure 6: Estimated Past Year SH Rates - All MSAs

An estimated 63% of DoD MSA women (2,127 cadets/midshipmen) and 20% of DoD MSA men (1,813
cadets/midshipmen) experienced SH in the past APY. These estimates reflect a significant increase
compared to 2018 when the rate for women was 50% and 16% for men.

8
9

•

USMA - an estimated 60% of women (611 cadets) and 19% of men (613 cadets) may have
experienced SH in the past APY. These estimates reflect a significant increase compared to 2018
when the rate for women was 48% and 17% for men.

•

USNA - an estimated 67% of women (824 midshipmen) and 22% of men (683 midshipmen) may
have experienced SH in the past APY. These estimates reflect a significant increase compared to
2018 when the rate for women was 57% and 17% for men.

Source: SAGR data
Source: SAGR data

•

USAFA an estimated 60% of women (693 cadets) and 19% of men (518 cadets) may have
experienced SH in the past APY. These estimates reflect a significant increase compared to 2018
when the rate for women was 46% and 13% for men.”

Suicide
Table 13 includes the number of total suicide deaths at each MSA from CY 2012 through April 2023. These
numbers are inclusive of both cadets/midshipmen and cadre.

Table 13: Number of Suicide Deaths CY 2012-April 2023 at MSAs

2012
USAFA
USMA
USNA
Total

2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023* Total

0
0
2

0
0
1

0
1
1

1
1
3

1
0
1

0
1
1

0
0
0

1
0
1

1
1
3

0
0
0

0
1
2

0
0
1

4
5
16 10

Source: Defense Suicide Prevention Office

United States Military Academy (West Point, NY)
USMA was established in 1802 in West Point, New York. The Academy has a population of approximately
4,300 cadets, 1,200 active-duty soldiers, 3,000 family members and 5,000 personnel. Eight DoD personnel
and 3 research support staff engaged with more than 200 participants during 41 sessions and 1 TTX.
Available risk index data or the U.S. Military Academy is summarized in the table below.

USMA Integrated Prevention Maturity Scoring
The following scoring dials represent the SVT’s assessment of the current integrated prevention capabilities of
the USMA. Given the early nature of the Department’s integrated prevention policies and practices, these
scores are not expected to be high and are in line with other sites visited during the 2023 round of OSIEs. This
round of scoring will be used to help form a baseline for measuring future progress.
Figure 7: USMA Prevention Maturity Dials

United States Naval Academy (Annapolis, MD)
USNA was established in 1845 in Annapolis, Maryland. The Academy has a population of approximately
4,400 midshipmen and 580 faculty members – both military and civilian. Over 5 days 8 DoD personnel and 3
research support staff engaged with more than 300 participants during 40 sessions and one TTX.

USNA Integrated Prevention Maturity Scoring
The following scoring dials represent the SVT’s assessment of the current integrated prevention capabilities of
the USNA. Given the early nature of the Department’s integrated prevention policies and practices, these
scores are not expected to be high and are in line with other sites visited during the 2023 round of OSIEs. This
round of scoring will be used to help form a baseline for measuring future progress.

Figure 8: USNA Prevention Maturity Dials

United States Air Force Academy (Colorado Springs, CO)
The U.S. Air Force Academy was established in 1959 in Colorado Springs, Colorado. The academy has a
population of approximately 4,000 cadets and a civilian workforce of approximately 1,600 personnel. Over 4
days, 8 DoD personnel and 3 research support staff engaged with approximately 265 participants during 36
sessions and one TTX.

USAFA Integrated Prevention Maturity Scoring
The following scoring dials represent the SVT’s assessment of the current integrated prevention capabilities of
the USAFA. Given the early nature of the Department’s integrated prevention policies and practices, these
scores are not expected to be high and are in line with other sites visited during the 2023 round of OSIEs. This
round of scoring will be used to help form a baseline for measuring future progress.

Figure 9: USAFA Prevention Maturity Dials

Combined Data Used to Score Prevention Capabilities
Detailed Data Used to Score the MSA’s Prevention Capability – USMA, USNA, USAFA
On-site Installation Evaluation (OSIE) team members provided individual scores across each sub-dimension
using a Qualtrics online secure survey platform. Once all individual scores were complete, OSIE teams for a
given MSA met together and consolidated their ratings into one team score for that MSA. The tables below
reflect the team scores for each data element and sub-dimension that led to the overall dimension scores. The
sub-dimensions were then scored to have sufficient or not sufficient evidence of being true. Sub-dimension
scores were derived by using scoring rules that incorporate data element ratings.

= Sufficient;

= Not Sufficient

Table 14: Prevention Capability Dimension/Subdimension Scoring:

DIMENSION: HEALTHY AND PROTECTIVE ENVIRONMENT—
PRIORITY
Leaders consistently emphasize the importance of a healthy
protective environment.

USMA
Score

USNA
Score

USAFA
Score

Leaders consistently monitor progress on harmful behaviors and
climate.
Leaders reinforce positive behaviors.
DIMENSION: HEALTHY AND PROTECTIVE ENVIRONMENT—
PREPARATION
32

Leaders are knowledgeable and skilled in building a protective
environment.
Established or systematic processes/structure to support healthy
climate.
Leaders and subordinates maintain connections.
DIMENSION: HEALTHY AND PROTECTIVE ENVIRONMENT—
IMPLEMENTATION
Subordinates and peers are referred to appropriate resources when
at-risk for harmful behaviors.
Leaders clearly communicate expectations for benchmarks, roles,
and responsibilities for improving/maintaining protective
environments to subordinates.
Leaders proactively monitor the stress levels of subordinates.
Leaders and Service members are held accountable for harmful
behaviors in a consistent manner (e.g., through standard operating
procedure).
Positive behaviors are rewarded/recognized.

DIMENSION: INTEGRATED PREVENTION—PRIORITY

USMA
Score

USNA
Score

USAFA
Score

Leaders hold prevention personnel accountable for sustained
integrated prevention.
Leaders reinforce best practice prevention processes (i.e., present
dose, theory-based, evaluated, trained deliverers, interactive
content).
Leaders prioritize data and evaluation related to prevention.
Leaders communicate integrated primary prevention is a consistent
and enduring priority to subordinates.
DIMENSION: INTEGRATED PREVENTION—PREPARATION
Prevention personnel receive ongoing and systematic training and
professional development to continually improve their approach to
integrated prevention.
Leaders are knowledgeable and skilled in primary prevention.
Prevention personnel are dedicated, knowledgeable and skilled in
primary prevention.
Continuity of prevention staff and effective prevention activities are
maintained over time.

DIMENSION: INTEGRATED PREVENTION—IMPLEMENTATION
Prevention approach is integrated (use common messages,
consistent collaboration, and common operating procedures).
Prevention approach is comprehensive.
Prevention approach is evaluated.
Prevention approach is continuously improved.
Resistance to the prevention approach is monitored and addressed.

DIMENSION: STAKEHOLDER ENGAGEMENT—PRIORITY
Leader communications stress the importance of stakeholder
engagement.

USMA
Score

USNA
Score

USAFA
Score

2–
Involve

1 – Inform

Leaders and prevention personnel provide positive reinforcement for
stakeholder engagement.
DIMENSION: STAKEHOLDER ENGAGEMENT—PREPARATION
Leaders have the knowledge and skills needed to conduct
stakeholder engagement.
Prevention personnel are dedicated, knowledgeable, and skilled in
conducting stakeholder engagement.
Stakeholders are knowledgeable about prevention.
Present resources exist to conduct stakeholder engagement.
DIMENSION: STAKEHOLDER ENGAGEMENT—
IMPLEMENTATION
Level of Collaboration: Score the level of stakeholder engagement
using a modified version of the IAP2 spectrum of public participation:
o NONE (0): Feedback from stakeholders is neither sought nor
used by leaders or prevention personnel.
o INFORM (1): Leaders and prevention personnel share
information in a variety of ways with key stakeholder groups
(“We will keep you informed”). No effort is made to get input.
o INVOLVE (2): Leaders and prevention personnel seek input
from stakeholders AFTER decisions are made.
o PARTICIPATE (3): Leaders and prevention personnel see
input BEFORE decisions are made.
o COLLABORATE (4): Leaders and prevention personnel work
with stakeholders to jointly frame the problem and the
solutions. Leaders and prevention personnel regularly circle
back with stakeholders to update them on progress

COLLABORATE PLUS (5): Leaders and prevention
personnel work with stakeholders to jointly frame the problem
and the solutions, using a standing group of stakeholders.
This includes leaders and prevention personnel regularly
circling back with stakeholders to update the group on
progress.

Appendix C: OSIE Process, Integrated Prevention Metric Development,
Validation, & Scoring
Participants
At each MSA, students (cadets and/or midshipmen, first through fourth year), active-duty Service members,
administration, faculty, staff, and dedicated prevention personnel participated in focus group interviews.
Across the MSAs, 116 focus groups (including TTX) were conducted and 915 total individuals attended and
provided data. Table 15 shows the number of focus group participants from each MSA.
Table 15: Focus Groups Completed and Number of Participants

USAFA
# Groups
# Participants
USMA
# Groups
# Participants
USNA
# Groups
# Participants

DP1

DP2

DP3

DP4

DP5

DP6

DP7

DP8

DP9

TTX

Total

2
22

3
23*

9
45

4
14

9
57

9
97

1
34*

37
292

3
8

2
5

2
10

7
45

4
17

9
38

5
21

2
14

4
47

1
26

39
231

1
5

1
13

8
19

7
26

14
189*

8
129

1
11

40
392 11

Student leaders (first-and second-class), faculty leaders, and prevention staff completed pen and paper
surveys following the focus groups. Table 16 shows the number of individuals who completed surveys at each
MSA.
Table 16: Leadership and Prevention Surveys Completed

MSA
USAFA
USMA
USNA

Number of Surveys Completed
Leadership Surveys
Prevention surveys
35
28
68
39
142
17

Measures
Based on an analysis of the requirements in DoDI 6400.09 and the elements of the OSIE Framework (priority,
preparation, and implementation), the OSD, in coordination with RAND, developed nine new metrics to assess
prevention capabilities associated with specific focus areas in DoDI 6400.09: Healthy and protective
environments, integrated prevention, and stakeholder engagement. 12
Three main tools (DPs, TTXs, and surveys) were used to collect data that would inform maturity scores that
capture MSAs’ overall capacity for integrated prevention. This section presents the dimensions upon which
MSAs were scored, the data collection tools used, and the scoring process that teams used to develop
dimension maturity scores from the data. Nine dimensions are used to guide the assessment of prevention
capabilities for the OSIE MSA report. These dimensions were identified by an analysis of the focus areas not

*Indicates that estimated (instead of precise) number of participants were provided for focus group(s) in this column
Information collection for these metrics were approved by Office of Management and Budget (OMB Control Number
0704-0610).
11

covered by existing DoD compliance checklists and DoD assessment tools to enforce relevant prevention
policies and the OSIE framework described in Table 17.
OSD prioritized three domains of focus:
1) Healthy & Protective Environment: Research shows that command climates can positively or negatively
impact behaviors such as SA and harassment.
2) Integrated Prevention: Effective prevention targets a mix of risk and protective factors that are both
common across problem areas as well as unique to specific harmful behaviors.
3) Stakeholder Engagement: Outcomes can be improved when multiple stakeholders have genuine
involvement in prevention activities.
Three additional domains are also included in OSIE framework:
1) Priority: Higher-level leadership sets the tone and sustains consistent focus on harmful behaviors.
2) Preparation: Prevention personnel and intermediate leadership are equipped with the ability, and exist
within a structure, that incentivizes and supports addressing harmful behaviors.
3) Implementation: Approach aligns with best practices and is done well (i.e., with high quality).

Table 17: Prevention Capabilities Assessed at MSA OSIEs

PREPARATION

Leaders prioritize
fostering a protective
environment by their
actions and
communications.

Leaders have the requisite
knowledge, skills, abilities
(KSAs) and access to
training to develop those
KSAs.

INTEGRATED
PREVENTION

Leaders prioritize
prevention activities.

Leaders and prevention
personnel have the requisite
KSAs to carry out prevention
successfully.

Prevention activities that
target risk and protective
factors across multiple
negative behaviors are
evaluated.

Leaders prioritize
engaging
stakeholders.

Prevention personnel have
the resources and requisite
KSAs to engage
stakeholders effectively.

Stakeholders are genuinely
engaged in prevention
activities across multiple
planning stages.

HEALTHY &
PROTECTIVE
ENVIRONMENT

PRIORITY

SERVICE
MEMBER
ENGAGEME
NT

FOCUS AREAS

OSIE FRAMEWORK AREA
IMPLEMENTATION

Leaders employ practices
known to support a protective
environment

These areas are referred to as core dimensions. Given the breadth of these nine dimensions, each one was
divided into multiple subdimensions, which are narrower in focus. These subdimensions were worded as
positive statements (e.g., Leaders consistently deter negative behaviors) so they would represent a highquality standard to which installations should aspire. Under each subdimension are even narrower “data
elements.” An overall score for each of the nine dimensions starts at the data element level. Each data
element, also worded as a positive standard to achieve, is judged to be either “present” or “absent” by
considering multiple data sources collected at the site. A scoring rubric was created so that a certain number
of data elements rated as “present” are needed for the subdimension to be considered “present.” The number
of data elements varies for each subdimension and thus the number of “present” data elements needed also
varies by subdimension. Figure 10 shows an example for Core Dimensions 1 (Healthy & Protective
Environment – Priority) and its subdimensions. This dimension has five subdimensions and the two data
elements are shown for Subdimension 1.2. In the scoring rubric, both data elements (1.2.1 and 1.2.2) need to
be rated as present for Subdimension 1.2 to be present.
Figure 10: Example of the Link between Data Elements, Subdimensions, and Core Dimensions

Once it is determined which subdimensions are present and absent, then a maturity score is used to determine
the final score for the Core Dimension. Table 18 below shows the maturity scoring for each core dimension.
Although a six-point scale is used to reflect the range of maturity, the exact makeup of the scoring rubric for
each core dimension varies by the number of subdimensions. Typically, the highest level of maturity not only
has all the subdimensions present, but also an additional requirement for a more robust presence of those
subdimensions.

Background on Maturity Scoring
RAND developed a structured maturity scoring system tailored to each core dimension. In its simplest form, a
maturity model is a set of characteristics, attributes, indicators, or patterns that represent progression and
achievement in a particular domain or discipline. The artifacts that make up the model are typically agreed
upon by the domain or discipline and are validated through application and iterative recalibration. A maturity
model allows an organization or industry to have its practices, processes, and methods evaluated against a
clear set of artifacts that establish a benchmark. These artifacts typically represent best practice and may
incorporate standards or other codes of practice that are important in a particular domain or discipline. By
having the ability to benchmark, organizations can use maturity models to determine their current level of
achievement or capability and then apply these models over time to drive improvement. However, when used
in a broader sense, maturity models can also help organizations benchmark their performance against other
38

organizations in their domain or industry, and help an industry determine how well it is performing by
examining the achievement or capability of its member organizations. Architecturally, maturity models typically
have “levels” along an evolutionary scale that defines measurable transitions from one level to another. The
corresponding attributes define each level; in other words, if an organization demonstrates these attributes, it is
said to have achieved both that level and the capabilities that the level represents. Having measurable
transition states between the levels enables an organization to use the scaling to:
•
•
•

Define its current state
Determine its future, more “mature” state; and
Identify the attributes it must attain to reach that future state

RAND tailored the general maturity approach, developing a specific scoring method for each individual
dimension (see Table 18). Thus, rather than one overall, generic scoring system, the maturity approach
focused on the specifics of each dimension. This approach was based on an assessment process OSD and
RAND used in a Department of Defense project rating the SA prevention capabilities of U.S. Military Service
Academies (Acosta et al., 2022).
In general, for each dimension, a higher maturity rating indicated a greater number of subdimensions that were
rated as present (which were driven by the number of data elements present). For example, there are five
subdimensions for Dimension 1 (Healthy & Protective Environment – Priority). A site could achieve a Maturity
Score of 2 by having any three subdimensions present. This scoring method was chosen because it assigns a
higher score for more subdimensions present, while also allowing sites to express their level of maturity in
different ways. For many of the dimensions, to obtain the highest score, a site needs to show consistent
evidence that the subdimensions (and their underlying data elements) have been maintained over the past two
years despite competing priorities.
As implemented, the maturity model can serve three purposes: it will allow DoD and others to understand the
current capabilities of the sites, it may help sites identify ways to strengthen their prevention efforts, and it may
permit comparison, both within and across sites.
Table 18: Link between Data Elements, Subdimensions, and Maturity Scoring
Dimension Maturity Scoring

Subdimensions
(total # of data elements needed to rate Subdimension as ‘present’/total #
data elements)

1. Healthy & Protective Environment – Priority
Maturity Score:
1.1. Consistently emphasize the importance of a healthy protective environment
5-Present in all 5 and consistent evidence that
(3/4)
1.2. Consistently deters negative behaviors (2/2)
presence has been maintained over the past
two years despite competing priorities
1.3. Leaders hold subordinates accountable for timely action (2/2)
4-Present in all 5 subdimensions
1.4. Leaders reinforce positive behaviors (1/1)
3-Present in 4 out of 5 subdimensions
1.5. Leader’s role model positive behaviors (1/1)
2-Present in 3 out 5 subdimensions
1-Present in 1 or 2 out of 5 subdimensions
0-None are Present
2. Integrated Prevention – Priority
Maturity Score:
2.1. Leaders see integrated primary prevention as a consistent and enduring
5-Present in all 4 subdimensions and
priority and communicate it to subordinates (2/2)
2.2. Leaders hold prevention personnel accountable for sustained integrated
consistent evidence that sufficiency has been
prevention (2/2)
maintained over time despite competing
2.3. Leaders reinforce best practice prevention processes (i.e., sufficient dose,
priorities
4-Present in all 4 subdimensions
theory-based, evaluated, trained deliverers, interactive content) (2/2)
3-Present in 3 out of 4 subdimensions
2.4. Leaders prioritize data and evaluation related to prevention (2/2)
2-Present in 2 out 4 subdimensions
1-Present in 1 out of 4 subdimensions
3. Stakeholder Engagement – Priority
Maturity Score:
3.1. Leaders and prevention personnel use stakeholder engagement to inform
5-Present in 3 out of 3 subdimensions,
priorities (1/1)
including support from the data call, and

consistent evidence that presence has been
3.2. Leader communications stress the importance of stakeholder engagement
maintained over time despite competing
(1/1)
3.3. Leaders and prevention personnel provide positive reinforcement for
priorities
4-Present in 3 out of 3, including support from
stakeholder engagement (2/2)
the data call
3-Present in 3 out of 3 subdimensions
2-Present in 2 out of 3 subdimensions
1-Present in 1 out of 3 subdimensions
0-None are Present
4. Healthy & Protective Environment – Preparation
Maturity Score:
4.1 Leaders are knowledgeable and skilled in building a protective environment**
5-Present in all 4 subdimensions, plus mean of 4.2 Established or systematic processes/structure to support healthy climate
data element 2.1.1 is greater than 4.0
4.3 Leaders and subordinates maintain present connections (3/4)
4-Present in 4 out 4 subdimensions
4.4 Leaders monitor climate-related efforts and behaviors and consider them in
3-Present in 3 out of 4 subdimensions
performance evaluations (2/2)
2-Present in 2 out of 4 subdimensions
**This data element is scored via a survey = overall mean score above 3.0 for the
1-Present in 1 out of 4 subdimensions
eleven leader survey items
0-None are Present
5. Integrated Prevention – Preparation
Maturity Score:
5.1. Prevention personnel receive ongoing and systematic training and
5-Present in all 5 subdimensions
professional development to continually improve their approach to integrated
4-Present in 4 of the 5 subdimensions
prevention (2/3)
3-Present in 3 out of 5 subdimensions
5.2. Leaders are knowledgeable and skilled in primary prevention**
2-Present in 2 out of 5 subdimensions
5.3. Prevention personnel are dedicated, knowledgeable and skilled in primary
1-Present in 1 out of 5 subdimensions
prevention (2/2)
0-None are Present
5.4. Collaborative structure exists to support integrated primary prevention (2/2)
5.5. Continuity of prevention staff and effective prevention activities are maintained
over time (2/2)
**This data element is scored via a survey = overall mean score above 3.0 for the
eight leader survey items
6. Stakeholder Engagement - Preparation
Maturity Score:
6.1. Leaders have the knowledge and skills needed to conduct stakeholder
5-Present in all 4 subdimensions and mean of
engagement**
6.2. Prevention personnel are dedicated, knowledgeable and skilled in conducting
8.1 OR 8.2 is greater than 4
4-Present in all 4 subdimensions
stakeholder engagement^^
3-Present in 3 out of 4 subdimensions
6.3. Stakeholders are knowledgeable about prevention (2/2)
2-Present in 2 out of 4 subdimensions
6.4. Sufficient resources exist to conduct stakeholder engagement (1/1)
1-Present in 1 out of 4 subdimensions
**This data element is scored via a survey = overall mean score above 3.0 for the
0-None are Present
four leader survey items
^^ This data element is scored via a survey = overall mean score above 3.0 for the
six prevention survey items
7. Healthy & Protective Environment – Implementation
Maturity Score:
7.1. Subordinates and peers are referred to appropriate resources when at-risk for
5-Present in all 5 subdimensions
harmful behaviors (2/2)
4-Present in 4 of the 5 subdimensions
7.2. Leaders clearly communicate expectations for benchmarks, roles, and
3-Present in 3 out of 5 subdimensions
responsibilities for improving/maintaining protective environments to
2-Present in 2 out of 5 subdimensions
subordinates (2/3)
1-Present in 1 out of 5 subdimensions
7.3. Leaders proactively monitor the stress levels of subordinates (2/2)
0-None are Present
7.4. Leaders and Service members are held accountable for harmful behaviors in a
consistent manner (e.g., through standard operating procedure) (2/2)
7.5. Positive behaviors are rewarded/recognized (1/1)
8. Integrated Prevention - Implementation
Maturity Score:
8.1. Prevention approach is integrated (use common messages, consistent
5-Present in all 5 subdimensions
collaboration, common operating procedures) (3/4)
4-Present in 4 of the 5 subdimensions
8.2. Prevention approach is comprehensive (3/4)
3-Present in 3 out of 5 subdimensions
8.3. Prevention approach is evaluated (3/3)
2-Present in 2 out of 5 subdimensions
8.4. Prevention approach is continuously improved (2/2)
1-Present in 1 out of 5 subdimensions
8.5. Resistance to the prevention approach is monitored and addressed (2/3)
0-None are Present
9. Stakeholder Engagement - Implementation
Maturity Score:
9.1 Level of collaboration
Score based on the following scale:
•
NONE (0): Feedback from
stakeholders is neither sought nor
used by leaders or prevention
personnel.

•

•
•

INFORM (1): Leaders and prevention
personnel share information in a
variety of ways with key stakeholder
groups (“We will keep you informed”).
No effort is made to get input.
INVOLVE (2): Leaders and
prevention personnel seek input from
stakeholders AFTER decisions are
made.
PARTICIPATE (3): Leaders and
prevention personnel see input
BEFORE decisions are made.
COLLABORATE (4): Leaders and
prevention personnel work with
stakeholders to jointly frame the
problem and the solutions. Leaders
and prevention personnel regularly
circle back with stakeholders to
update them on progress

Subdimensions
Core dimensions were designed to be broad categories. In contrast, subdimensions were designed to address
narrower topics. Striking a balance between breadth and simplicity, there are three to five subdimensions in
each core dimension, except for Core Dimension nine (Stakeholder Engagement-Implementation), which has
one subdimension. Subdimensions were chosen for their theoretical connection to the dimension, their
support in the research literature, and their focus on a narrower aspect of the core dimension. Below is a
summary of the subdimensions used to assess each of the nine core dimensions and relevant references
supporting their inclusion.

Subdimensions for Dimension 1: Healthy & Protective Environment-Priority
This dimension contains five subdimensions that aim to assess the extent to which leaders prioritize a healthy
and protective environment and sets the tone to sustain a focus on a protective environment.
Subdimensions

References

1.1

Leaders consistently emphasize the
importance of a healthy protective environment

Crittendon & Hope, 2017, pp.18-21; Hoover,
Randolph, Elig, & Klein, 2001, pp. 31-33;
Ratcliff, Key-Roberts, Simmons, & JiménezRodríguez, 2018, pp. 4-18

1.2

Leaders consistently deter negative behaviors

Cook, Jones, Lipari, & Lancaster, 2005; Ratcliff,
Key-Roberts, Simmons, & Jiménez-Rodríguez,
2018, pp. 4-16

1.3

Leaders hold subordinates accountable for
timely action

Jones & Bullis, 2003, pp. 24-25

1.4

Leaders reinforce positive behaviors

Jones & Bullis, 2003, pp. 21-40

1.5

Leaders role model positive behaviors

Ratcliff, Key-Roberts, Simmons, & JiménezRodríguez, 2018, pp.2

Subdimensions for Dimension 2: Integrated Prevention-Priority
This dimension contains four subdimensions that aim to assess the extent to which leaders prioritize integrated
primary prevention and sets the tone to sustain a focus on a prevention.
41

Subdimensions

References

2.1

Leaders see integrated primary prevention as a
consistent and enduring priority and
communicate it to subordinates

Noonan et al., 2009; Kreuter, Lezin, & Young,
2000; McCartan, Kemshall, & Tabachnick, 2015;
Campbell & Wasco, 2005; Patton, 2010

2.2

Leaders hold prevention staff accountable for
sustained integrated prevention

Thompson, Taplin, McAfee, Mandelson, &
Smith, 1995; Nation et al., 2003; McIntosh, Filter,
Bennett, Ryan, & Sugai, 2010

2.3

Leaders reinforce best practice prevention
processes (sufficient dose, theory-based,
evaluated, trained deliverers, interactive
content)

Kratochwill, Volpiansky, Clements, & Ball, 2007;
Hawkins, Shapiro, & Fagan, 2010; Mihalic &
Irwin, 2003; McDonald, Charlesworth, &
Graham, 2015; Murnieks, Allen, & Ferrante,
2011

2.4

Leaders prioritize data and evaluation related to
prevention

DeGue et al., 2012; Brubaker, 2009; Provost &
Fawcett, 2013; Mandinach, 2012; Sable, Danis,
Mauzy, & Gallagher, 2006

Subdimensions for Dimension 3: Stakeholder Engagement-Priority
This dimension contains three subdimensions that aim to assess the extent to which leaders prioritize
stakeholder engagement and sets the tone to sustain a focus on stakeholder engagement to inform primary
prevention.
Subdimensions

References

3.1

Leaders and prevention personnel use
stakeholder engagement to inform priorities

Ahmed & Palermo, 2010; Dills, Fowler, & Payne,
2016; Goodman et al., 2017; Hood et al., 2010

3.2

Leader communications stress the importance
of stakeholder engagement

Ahmed & Palermo, 2010; Jolibert & Wesselink,
2012

3.3

Leaders and prevention staff provide positive
reinforcement for stakeholder engagement

Hood et al., 2010

Subdimensions for Dimension 4: Healthy & Protective Environment-Preparation
This dimension contains four subdimensions that aim to assess the extent to which leaders and prevention
staff are equipped—with skills and knowledge—and empowered with a clear line of sight across the chain of
command to maintain a healthy and protective environment.

Subdimensions

References

4.1

Leaders are knowledgeable about and skilled at
building a protective environment

Cook, Jones, Lipari, & Lancaster, 2005, pp. 9-10

4.2

Established or systematic processes/structure
support a protective environment

Crittendon & Hope, 2017, pp. 20-29

4.3

Leaders and subordinates maintain sufficient
connections

Ratcliff, Key-Roberts, Simmons, & JiménezRodríguez, 2018, pp. 4 & 17

4.4

Leaders monitor climate-related efforts and
behaviors and consider them in performance
evaluations

Hoover, Randolph, Elig, & Klein, 2001, pp. 32-33

Subdimensions for Dimension 5: Integrated Prevention-Preparation
This dimension contains five subdimensions that aim to assess the extent to which leaders and prevention staff
are equipped—with skills and knowledge—and empowered with a clear line of sight across the chain of
command to sustain high-quality integrated primary prevention.
Subdimensions

References

5.1

Prevention personnel receive ongoing and
systematic training and professional
development to continually improve their
approach to integrated prevention

Kratochwill, Volpiansky, Clements, & Ball, 2007;
Hawkins, Shapiro, & Fagan, 2010; Mihalic &
Irwin, 2003; McDonald, Charlesworth, & Graham,
2015; Murnieks, Allen, & Ferrante, 2011

5.2

Leaders are knowledgeable and skilled at
primary prevention

Kratochwill, Volpiansky, Clements, & Ball, 2007;
Hawkins, Shapiro, & Fagan, 2010; Mihalic &
Irwin, 2003; McDonald, Charlesworth, & Graham,
2015; Murnieks, Allen, & Ferrante, 2011

5.3

Prevention personnel are dedicated,
knowledgeable and skilled in primary prevention

Kratochwill, Volpiansky, Clements, & Ball, 2007;
Hawkins, Shapiro, & Fagan, 2010; Mihalic &
Irwin, 2003; McDonald, Charlesworth, & Graham,
2015; Murnieks, Allen, & Ferrante, 2011

5.4

Collaborative structure exists to support
integrated primary prevention

DeGue et al., 2012; Brubaker, 2009; Provost &
Fawcett, 2013; Mandinach, 2012; Sable, Danis,
Mauzy, & Gallagher, 2006

5.5

Continuity of prevention staff and effective
prevention activities are maintained over time

Dills, Fowler, & Payne, 2016; Wandersman &
Florin, 2003; Lundgren & Amin, 2015; Bond &
Hauf, 2004; McMahon, Postmus, & Koenick,
2011

Subdimensions for Dimension 6: Stakeholder Engagement-Preparation
This dimension contains four subdimensions that aim to assess the extent to which leaders and prevention
staff are equipped—with skills and knowledge—and empowered with a clear line of sight across the chain of
command to sustain stakeholder engagement efforts to inform primary prevention.
Subdimensions

References

6.1

Leaders have the skills and knowledge needed
to conduct stakeholder engagement

SAMHSA, 2021

6.2

Prevention staff are dedicated, knowledgeable
and skilled in conducting stakeholder
engagement

Scaccia et al., 2015; Powell et al., 2015;
SAMHSA, 2021

6.3

Stakeholders are knowledgeable about
prevention

Desai, 2018

6.4

Sufficient resources exist to conduct
stakeholder engagement

Noonan et al., 2009; Krug, Mercy, Dahlberg, &
Zwi, 2002; García-Moreno et al., 2015; Hawkins,
Shapiro, & Fagan, 2010

Subdimensions for Dimension 7: Healthy & Protective Environment-Implementation
This dimension contains five subdimensions that aim to assess the extent to which actions taken by leaders
and prevention staff are aligned with best practices for building a healthy and protective environment and are
done well (i.e., with high quality).
Subdimensions

References

7.1

Subordinates and peers are referred to
appropriate resources when at-risk for harmful
behaviors

Crittendon & Hope, 2017, pp.18-21

7.2

Leaders clearly communicate expectations for
benchmarks, roles, and responsibilities for
improving/maintaining protective environments
to subordinates

Ratcliff, Key-Roberts, Simmons, & JiménezRodríguez, 2018, pp.4-16, 18

7.3

Leaders proactively monitor the stress level of
subordinates

Hoover, Randolph, Elig, &. Klein, 2001, pp. 4

7.4

Leaders and Service members are held
accountable for harmful behaviors in a
consistent manner (e.g., through standard
operating procedure)

Cook, Jones, Lipari, & Lancaster, 2005
Ratcliff, Key-Roberts, Simmons, & JiménezRodríguez, 2018

7.5

Positive behaviors are rewarded/recognized

Jones & Bullis, 2003, pp. 21-40

Subdimensions for Dimension 8: Integrated Prevention-Implementation
This dimension contains five subdimensions that aim to assess the extent to which actions taken by leaders
and prevention staff are aligned with best practices for integrated primary prevention and are done well (i.e.,
with high quality).
Subdimensions

References

8.1

Prevention approach is integrated (use common
messages, consistent collaboration, common
operating procedures)

Gidycz, Wyatt, Galbreath, Axelrad, & McCone,
2018

8.2

Prevention approach is comprehensive

Brofenbrenner, 1992, 2005; Casey & Lindhorst,
2009; Banyard, Eckstein, & Moynihan, 2010;
Prochaska & Prochaska, 2011; Vladutiu, Martin,
& Macy, 2011

8.3

Prevention approach is evaluated

Chinman et al., 2016; 2018; Francisco, Paine, &
Fawcett, 1993

8.4

Prevention approach is continuously improved

Chinman et al., 2016; 2018; Francisco, Paine, &
Fawcett, 1993
44

8.5

Resistance to the prevention approach is
monitored and addressed

Nation et al., 2003; Rich, Utley, Janke, &
Moldoveanu, 2010

Subdimension for Dimension 9: Stakeholder Engagement-Implementation
This dimension contains one subdimension that aims to assess the extent to which actions taken by leaders
and prevention staff are aligned with best practices for stakeholder engagement and are done well (i.e., with
high quality).
Subdimensions
9.1

References

Level of collaboration ranging from none, to inform (sharing
information, lowest level) to collaborate (sharing decision
making and implementation, highest level)

International Association for Public
Participation, 2018

Focus Group Discussion Protocols
In the initial OSIE process, seven DPs were developed to measure integrated prevention and prevention
capacity. Two more DPs were created for MSA OSIEs (DPs 8/9) -- derived from the original DP2/3 -- using
language more inclusive to cadets and/or midshipmen. The seven DPs were designed to collect responses
across different levels of students (cadets and/or midshipmen, first through fourth year), active-duty Service
members (leadership), administration, faculty, and staff (Table 19). OSIE teams facilitated each focus groups,
using the appropriate DP depending on the focus group audience. Focus groups were scheduled either inperson or over a virtual platform for one hour and informed consent information was provided orally prior to the
start of the groups. Participants in focus groups for Service members, leaders, and students were split into
separate groups for men and women.
Table 19: Discussion Protocols and Target Participants

Discussion
Protocol
DP1
DP2
DP3
DP4
DP5
DP6
DP7
DP8
DP9

Target Participants
MSA Command
Service Members (SMs) – E1-E4, O1-O3
SMs – E1-E4, E5-E6
Leaders – O4-O5, O6
Leaders – E7-E9
Prevention Personnel
Prevention Support Personnel
Cadets/MSM – 1/C & 2/C
Cadets/MSM – 3/C & 4/C 13

Surveys
Following focus groups for Leaders, MSA Command, and upperclassmen (classes 2 and 1), participants were
asked to complete “Leadership Surveys.” The leadership surveys ask participants to rate how knowledgeable
they are about 23 prevention-related items on a scale of 1 (no knowledge) to 5 (extensive knowledge).
Participants are also asked to rate how relevant each of these 23 items is to their job on a scale of 1 (not
relevant) to 5 (central to my job). Survey items cover topics across the dimensions (healthy and protective
environment, integrated prevention, and stakeholder engagement).

Note that DP8 and DP9 are derived from DP2 and DP3 respectively and were used at the MSAs to target cadets and midshipmen
rather than Service members.
13

Following the prevention personnel and prevention support focus groups, prevention-related participants are
asked to complete “Prevention Surveys.” These surveys ask respondents to rate their knowledge and the
relevance of 24 items related to primary prevention and Service member engagement using the same scale as
the leadership survey. Both surveys include demographic questions (gender, age, ethnicity, education, pay
grade, service, component). Surveys were generally completed by hand (using pen and paper) and sent via
mail to the Miami University research team who entered the survey responses and provided average scores
that were used in the overall site scoring process. The leadership survey leads to mean knowledge scores for
each dimension and the prevention survey provides mean knowledge scores for integrated prevention and
stakeholder engagement.

Tabletop Exercise (TTX)
The TTX is an activity that prompts prevention personnel to complete a prevention-related activity. The TTX
used at most military installations usually consists of an exercise involving deployment/redeployment
prevention readiness based off a real-world scenario. Leadership and prevention personnel are given a
scenario and asked to explain how prevention efforts would be assigned and/or utilized in response to the
scenario. Participants are observed and scored based on their responses and collaboration throughout the
activity. Because the MSAs cannot be activated for deployment status or return from a deployment, the TTX
used at the MSAs was edited to assess integrated prevention in relation to cadet/midshipman summer field
experiences.
Together, the DPs, TTX observations, and Leadership/Prevention Surveys are used to create a final
assessment of the integrated primary prevention capabilities at each MSA. Figure 19 below expresses how all
materials mentioned above (DPs, surveys, and TTX) do not capture each of the nine dimensions individually,
but span across the nine dimensions to capture a more complete understanding of an MSAs integrated primary
prevention capabilities and ultimately informing final OSIE maturity scores.
Table 20: Tools Used to Create Final Assessment

Data Collection and Scoring Procedures
Each OSIE multi-disciplinary evaluation team included a Senior Executive or GS-15 team lead and seven staff
representing from the Office of the Under Secretary of Defense for Personnel and Readiness (Office of Force
Resiliency, Office of Diversity, Equity and Inclusion, Defense Suicide Prevention Office, Sexual Assault
46

Prevention and Response Office, Office of People Analytics, and the Diversity Management Operation Center).
The teams also included representation from the Military Services (military and civilian employees), and the
National Guard Bureau. The teams were comprised of both Service members and civilian employees. The
Service members assigned to each OSIE team acted as a Senior Subject Matter Expert (SME) and served
with the associated military branch respective to the MSA. This allowed for a mixture of military perspectives
and insight into MSA culture. Research assistants were also present during focus group interviews to collect
interviewee responses allowing team members to engage with focus group participants in a fluid manner.
Several improvements and modifications were made from the 2021 OSIEs to better pertain to the 2023 MSA
OSIEs which included adjusting the language and flow of the discussion protocols to improve clarity and
simplicity. Improvements were also made to scoring and reporting, including development of processes to
organize notes to better align with scoring. Finally, two additional discussion protocols were developed for the
MSAs: one for fourth-class and third-class cadets/midshipmen and one for second-class and first-class cadets
and midshipmen. The scenario used for the TTX was adapted to increase relevance to the MSA mission.
At the end of the MSA OSIE, research assistants from Miami University, through an Intergovernmental
Personnel Act agreement with DoD, compiled responses from the DPs and TTX into a single document using
the Qualtrics (2023) online secure survey platform. The compiled document and survey mean scores were
sent to the SVT for scoring. Individually, team members used the data to derive a maturity score for each
dimension. To ensure inter-rater reliability, the individual scores were provided to the team leader to measure
the consistency of scoring across team members. The team lead was responsible for validating scores. When
team member scores differed, the team conferred to assign and validate scores. Only validated teams scores
were used for the final assessment of the MSAs.

Appendix C References
Acosta, J., Chinman, M., Tharp, A., Baker, J., Flaspohler, P., Fortson, B., Kerr, A., Lamont, A., Meyer, A.,
Smucker, S., Wargel, K., & Wandersman, A. (2022). Development and pilot test of criteria defining best
practices for organizational sexual assault prevention. Preventive Medicine Reports, 26, 101723.
https://doi.org/10.1016/j.pmedr.2022.101723
Agboola, F., McCarthy, T., & Biddinger, P. D. (2013). Impact of emergency preparedness exercise on
performance. Journal of Public Health Management and Practice 19 Suppl 2, S77–S83.
https://doi.org/10.1097/PHH.0b013e31828ecd84
Ahmed, S. M., & Palermo, A. G. S. (2010). Community engagement in research: frameworks for education and
peer review. American Journal of Public Health, 100(8), 1380-1387.
Ayuso, S., Rodríguez, M. A., García-Castro, R., & Ariño, M. A. (2014). Maximizing stakeholders’ interests: An
empirical analysis of the stakeholder approach to corporate governance. Business & Society, 53(3), 414-439.
Banyard, V. L., Eckstein, R. P., & Moynihan, M. M. (2010). Sexual violence prevention: The role of stages of
change. Journal of Interpersonal Violence, 25(1), 111-135.
Batorowicz, B. & Shepherd, T.A. (2008). Measuring the quality of transdisciplinary teams, Journal of
Interprofessional Care, 22: 612-620.
Bernoff, J., & Schadler, T. (2010). Empowered: unleash your employees, energize your customers, transform
your business. Boston, Mass., Harvard Business Press.
Bond, L. A., & Hauf, A. M. C. (2004). Taking stock and putting stock in primary prevention: Characteristics of
effective programs. Journal of Primary Prevention, 24(3), 199-221.
Bronfenbrenner, U. (1992). Ecological systems theory. London: Jessica Kingsley Publishers.
Bronfenbrenner, U. (2005). Ecological systems theory (1992). In U. Bronfenbrenner (Ed.), Making human
beings human: Bioecological perspectives on human development (pp. 106–173). Sage Publications Ltd.
Bronfenbrenner, U. (Ed.). (2005). Making human beings human: Bioecological perspectives on human
development. Sage Publications Ltd.
Brubaker, S. J. (2009). Sexual assault prevalence, reporting and policies: Comparing college and university
campuses and military service academies. Security Journal, 22(1), 56-72.
Campbell, R., & Wasco, S. M. (2005). Understanding rape and SA: 20 years of progress and future directions.
Journal of Interpersonal Violence, 20(1), 127-131.
Cartmill, C., Soklaridis, S., & Cassidy, J. (2011). Transdisciplinary teamwork: the experience of clinicians at a
functional restoration program, Journal of Occupational Rehabilitation, 21: 1-8.
Casey, E. A., & Lindhorst, T. P. (2009). Toward a multi-level, ecological approach to the primary prevention of
SA: Prevention in peer and community contexts. Trauma, Violence, & Abuse, 10(2), 91-114.
Center for the Army Profession and Leadership, Building and Maintaining a Positive Climate Handbook, July
2020.
Chandra, A., Williams, M. V., Lopez, C., Tang, J., Eisenman, D., & Magana, A. (2015). Developing a Tabletop
Exercise to Test Community Resilience: Lessons from the Los Angeles County Community Disaster Resilience
Project. Disaster Medicine and Public Health Preparedness, 9(5), 484–488.
https://doi.org/10.1017/dmp.2015.99

Chinman, M., Acosta, J., Ebener, P. et al. Can implementation support help community-based settings better
deliver evidence-based sexual health promotion programs? A randomized trial of Getting To Outcomes®.
Implementation Sci 11, 78 (2015). https://doi.org/10.1186/s13012-016-0446-y
Chinman, M., Ebener, P., Malone, P. S., Cannon, J., D'Amico, E. J., & Acosta, J. (2018). Testing
implementation support for evidence-based programs in community settings: a replication cluster-randomized
trial of Getting To Outcomes®. Implementation Science: IS, 13(1), 131. https://doi.org/10.1186/s13012-0180825-7
Cook, P., Jones, A., Lipari, R. & Lancaster, A. (2005). Service academy 2005 sexual harassment and assault
survey. Defense Manpower Data Center, Arlington, VA: Survey and Program Evaluation Division.
Crittendon, D. & Hope, R.O. (2017). An assessment of FY2016 locally developed questions from the DEOMI
Organizational Climate Survey: Recommendations and potential implications, No. 10-17, 2017, Defense Equal
Opportunity Management Institute, Patrick Air Force Base, Florida.
DeGue, S., Holt, M. K., Massetti, G. M., Matjasko, J. L., Tharp, A. T., & Valle, L. A. (2012). Looking ahead
toward community-level strategies to prevent sexual violence. Journal of Women's Health, 21(1), 1-3.
Dills, J., Fowler, D., & Payne, G. (2016). Sexual violence on campus: Strategies for prevention. National
Center for Injury Prevention and Control (U.S.). Division of Violence Prevention.
Desai, V. M. (2017). Collaborative stakeholder engagement: An integration between theories of organizational
legitimacy and learning. Academy of Management Journal, 61, 2018, 220-244.
Dyer, J.A. (2003). Multidisciplinary, interdisciplinary, and transdisciplinary: educational models and nursing
education. Nursing Education Perspectives, 24: 186-188.
Frahm, K. A., Gardner, P. J., Brown, L. M., Rogoff, D. P., & Troutman, A. (2014). Community-Based Disaster
Coalition training. Journal of Public Health Management and Practice, 20 Suppl 5, S111–S117.
https://doi.org/10.1097/PHH.0000000000000058
Francisco, V. T., Paine, A., & Fawcett, S. B. (1993). A methodology for monitoring and evaluating community
health coalitions. Health Education Research: Theory and Practice, 8, 403-416.
García-Moreno, C., Zimmerman, C., Morris-Gehring, A., Heise, L., Amin, A., Abrahams, N., ... & Watts, C.
(2015). Addressing violence against women: a call to action. The Lancet, 385(9978), 1685-1695.
Gidycz, C.A., Wyatt, J., Galbreath, N.W., Axelrad, S. & McCone, D.R. (2018). Sexual assault prevention in the
military: Key issues and recommendations, Military Psychology, 30.3, 2018, 240-251.
Goodman, M. S., Thompson, V. L. S., Arroyo Johnson, C., Gennarelli, R., Drake, B. F., Bajwa, P., ... & Bowen,
D. (2017). Evaluating community engagement in research: quantitative measure development. Journal of
Community Psychology, 45(1), 17-32.
Hawkins, J. D., Shapiro, V. B., & Fagan, A. A. (2010). Disseminating effective community prevention practices:
Opportunities for social work education. Research on Social Work Practice, 20(5), 518-527.
Hood, N. E., Brewer, T., Jackson, R., & Wewers, M. E. (2010). Survey of community engagement in NIH‐
funded research. Clinical and Translational Science, 3(1), 19-22.
Hoover, E. C., Randolph, J.S., Elig, T.W. & Klein, P.M. (2001). Overview of the 2000 Military Exit Survey, No.2
2001-001. Defense Manpower Data Center, Arlington, VA: Survey and Program Evaluation Division.
International Association for Public Participation. (2018). Public Participation Pillars. Available online at
www.iap2.org.
49

Jolibert, C., & Wesselink, A. (2012). Research impacts and impact on research in biodiversity conservation:
The influence of stakeholder engagement. Environmental Science & Policy, 22, 100-111.
Jonas, J.M., Boha, J., Sörhammar, D. and Moeslein, K.M. (2018). Stakeholder engagement in intra- and interorganizational innovation: Exploring antecedents of engagement in service ecosystems. Journal of Service
Management, Vol. 29 No. 3, pp. 399-421. https://doi.org/10.1108/JOSM-09-2016-0239
Jones, S. M., & Bullis, C. (2003). Improving Accountability for Effective Command Climate: A Strategic
Imperative. United States Army War Colleges, Carlisle, Pennsylvania.
King, G., Strachan, D., Tucker, M., Duwyn, B., Desserud, S., & Shillington M. (2009). The application of a
transdisciplinary model for early intervention services. Infants and Young Children, 22: 211-223.
Klima, D. A., Seiler, S. H., Peterson, J. B., Christmas, A. B., Green, J. M., Fleming, G., Thomason, M. H., &
Sing, R. F. (2012). Full-scale regional exercises: closing the gaps in disaster preparedness. The journal of
Trauma and Acute Care Surgery, 73(3), 592–598. https://doi.org/10.1097/TA.0b013e318265cbb2
Kratochwill, T. R., Volpiansky, P., Clements, M., & Ball, C. (2007). Professional development in implementing
and sustaining multitier prevention models: Implications for response to intervention. School Psychology
Review, 36(4).
Kreuter, M.W., Lezin, N.A., & Young, L.A. (2000). Evaluating community-based collaborative mechanisms:
Implications for practitioners. Health Promotion Practice, 1, 49-63.
Krug, E. G., Mercy, J. A., Dahlberg, L. L., & Zwi, A. B. (2002). The world report on violence and health. The
Lancet, 360(9339), 1083-1088.
Kujala, J. & Sachs, S. (2019). The practice of stakeholder engagement (Chapter 14), The Cambridge
Handbook of Stakeholder Theory, 227.
Lipnack, J., & Stamps, J. (1997). Virtual teams: Reaching across space, time, and organizations with
technology. New York: John Wiley & Sons.
Andrzej, L. (2016). The manifestations of positive leadership strategies in the doctrinal assumptions of the
U.S. army leadership concept. Journal of Corporate Responsibility and Leadership, 2 (51) 51, 2016.
Lundgren, R., & Amin, A. (2015). Addressing intimate partner violence and sexual violence among
adolescents: emerging evidence of effectiveness. Journal of Adolescent Health, 56(1), S42-S50.
Mandinach, E. B. (2012). A perfect time for data use: Using data-driven decision making to inform practice.
Educational Psychologist, 47(2), 71-85.
Matthews, M., Morral, A.M, Schell, T.L., Cefalu, M., Snoke, J., & Briggs, R.J. (2020). Organizational
Characteristics Associated with Sexual Assault Risk in the U.S. Marine Corps, Santa Monica, Calif.: RAND
Corporation.
McIntosh, K., Filter, K. J., Bennett, J. L., Ryan, C., & Sugai, G. (2010). Principles of sustainable prevention:
Designing scale‐up of school‐wide positive behavior support to promote durable systems. Psychology in the
Schools, 47(1), 5-21.
McCartan, K. F., Kemshall, H., & Tabachnick, J. (2015). The construction of community understandings of
sexual violence: Rethinking public, practitioner and policy discourses. Journal of Sexual Aggression, 21(1),
100-116.
McDonald, P., Charlesworth, S., & Graham, T. (2015). Developing a framework of effective prevention and
response strategies in workplace SH. Asia Pacific Journal of Human Resources, 53(1), 41-58.
50

McMahon, S., Postmus, J. L., & Koenick, R. A. (2011). Conceptualizing the engaging bystander approach to
sexual violence prevention on college campuses. Journal of College Student Development, 52(1), 115-130.
Mihalic, S. F., & Irwin, K. (2003). Blueprints for violence prevention: From research to real-world settings—
factors influencing the successful replication of model programs. Youth Violence and Juvenile Justice, 1(4),
307-329.
Morral, A.R., Schell, T.L., Cefalu, M., Hwang, J. & Gelman, A. (2021). Sexual Assault and SH in the U.S.
Military: Volume 5. Estimates for Installation- and Command-Level Risk of Sexual Assault and SH from the
2014 RAND Military Workplace Study, Santa Monica, Calif.: RAND Corporation
Murnieks, C. Y., Allen, S. T., & Ferrante, C. J. (2011). Combating the effects of turnover: Military lessons
learned from project teams rebuilding Iraq. Business Horizons, 54(5), 481-491.
Nation, M., Crusto, C., Wandersman, A., Kumpfer, K. L., Seybolt, D., Morrissey-Kane, E., & Davino, K. (2003).
What works in prevention: Principles of effective prevention programs. American Psychologist, 58(6-7), 449.
Noonan, R. K., Emshoff, J. G., Mooss, A., Armstrong, M., Weinberg, J., & Ball, B. (2009). Adoption, adaptation,
and fidelity of implementation of sexual violence prevention programs. Health Promotion Practice, 10(1_suppl),
59S-70S.
O'Neill, A. S., Acosta, J. D., Chinman, M., Tharp, A. L., & Fortson, B. L. (2023). Development and pilot test of
the competency assessment for sexual assault prevention practitioners. Health Promotion Practice, 24(3),
514–522. https://doi.org/10.1177/15248399221084228
Patton, M. Q. (2010). Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and
Use. Guilford Press.
Powell, A., Watson, J., Staley, P., Patrick, S., Horn, M., Fetzer, L., ... & Verma, S. (2015). Blending Learning:
The Evolution of Online and Face-to-Face Education from 2008-2015. Promising Practices in Blended and
Online Learning Series. International association for K-12 online learning.
Prochaska, J. J., & Prochaska, J. O. (2011). A review of multiple health behavior change interventions for
primary prevention. American Journal of Lifestyle Medicine, 5(3), 208-221.
Provost, F., & Fawcett, T. (2013). Data science and its relationship to big data and data-driven decision
making. Big Data, 1(1), 51-59.
Ratcliff, N. J., Key-Roberts, M., Simmons, M.J & Jiménez-Rodríguez, M. (2018). Inclusive Leadership Survey
Item Development, No. 2018-03. Consortium of Universities, Washington DC.
Rich, M. D., Utley, E.A., Janke, K. & Moldoveanu, M. (2010). I'd rather be doing something else: male
resistance to rape prevention programs, The Journal of Men’s Studies 18, 268-288.
Rosenfield, P.L. (1992). The potential of transdisciplinary research for sustaining and extending linkages
between the health and social sciences. Social Science and Medicine, 35: 1343-1357.
Sable, M. R., Danis, F., Mauzy, D. L., & Gallagher, S. K. (2006). Barriers to reporting SA for women and men:
Perspectives of college students. Journal of American College Health, 55(3), 157-162.
Sadler, A. G., Lindsay, D.R., Hunter, S.T., & Day, D.V. (2018). The impact of leadership on sexual harassment
and sexual assault in the military. Military Psychology, 30(3).
Scaccia, J. P., Cook, B. S., Lamont, A., Wandersman, A., Castellow, J., Katz, J., & Beidas, R. S. (2015). A
practical implementation science heuristic for organizational readiness: R = MC2. Journal of Community
Psychology, 43(4), 484–501. https://doi.org/10.1002/jcop.21698
51

Stepans, M.B., Thompson, C.L. & Buchanan, M.L. (2002). The role of the nurse on a transdisciplinary early
intervention assessment team, Public Health Nursing, 19: 238-245.
Substance Abuse and Mental Health Services Administration (SAMHSA). (2021). Prevention Core
Competencies. Publication No. PEP20-03-08-001. Rockville, MD: Substance Abuse and Mental Health
Services Administration.
Thompson, R. S., Taplin, S. H., McAfee, T. A., Mandelson, M. T., & Smith, A. E. (1995). Primary and
secondary prevention services in clinical practice: twenty years' experience in development, implementation,
and evaluation. JAMA, 273(14), 1130-1135.
Vladutiu, C. J., Martin, S. L., & Macy, R. J. (2011). College-or university-based SA prevention programs: A
review of program outcomes, characteristics, and recommendations. Trauma, Violence, & Abuse, 12(2), 67-86.
Wandersman, A., & Florin, P. (2003). Community interventions and effective prevention. American
Psychologist, 58(6-7), 441.

Appendix D: Acronyms List
ACT
AOC
DEOCS
DoD
DP
MSA
NCO
OSD
OSIE
SAGR
SVT
TAC
TTX
USAFA
USC
USMA
USNA

CUI

Addressing Sexual Harassment/Sexual Assault; Creating Healthy Climates; and
Tackling Holistic Health
Air Officers Commanding
Defense Organizational Climate Survey
Department of Defense
Discussion protocol
Military Service Academy
Non-Commissioned Officer
Office of Secretary of Defense
On-site installation evaluation
Sexual Assault Gender Relations Survey
Site visit team
Tactical Officer
Table-top exercise
United States Air Force Academy
Unwanted Sexual Contact
United States Military Academy
United States Naval Academy
Acronyms List

Appendix
E:

File Type	application/pdf
File Title	2023 Military Service Academies On-Site Installation Evaluation Report
Subject	2023 Military Service Academies On-Site Installation Evaluation Report
Author	OSDPA
File Modified	2023-08-17
File Created	2023-08-17