SUPPORTING STATEMENT – PART B
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
Description of the Activity
The respondent universe is female sailors ages 17 or older, as well as EAAA program facilitators located at the Naval Air Technical Training Center (NATTC) in Pensacola, FL. This is a new data collection and evaluation effort. Eligible participants for this evaluation include all newly enlisted female sailors as they arrive at NATTC during the time period of the baseline data collection effort. As part of base indoctrination class (INDOC) schedule, female sailors (expected rank of E1-E4) will be provided the opportunity to participate in the EAAA training, as well as the EAAA evaluation.
The EAAA outcome evaluation design involves both self-selection by and randomization of participants. In other words, eligible participants self-select both into the evaluation of EAAA as well as the EAAA training through a unique hybrid self-selection trial (SST) and randomized control trial (RCT) study design. Sailors who express interest in the EAAA training will be randomized into either the Training Group (to attend the EAAA training) or Comparison Group A (interested in EAAA training and randomized into control group). Sailors who are not interested in participating in the EAAA training, but express interest in the overall evaluation, will be placed into Comparison Group B. Comparison Group B is imperative to provide Navy and NATTC with the perspective and experience of individuals not interested in the EAAA training, given the training is provided on a voluntary basis currently. Because the Baseline Survey is facilitated in-person, the group is calculated at 100% for those who are interested in the evaluation effort. As attrition at follow-up is a common phenomenon in survey data collection, NORC expects a strong response rate at follow-up through use of incentives.
NORC will enroll up to 1,200 total female sailors across all three evaluation groups (see Table 1 for more information). Based on power analyses (see section 2c below), the evaluation requires a minimum of 400 female sailors to complete the baseline and six-month follow-up surveys, with 150 sailors each in the Program Group and Comparison Group A, and 100 sailors in Comparison Group B. To achieve the necessary responses during the evaluation, assuming a 40% response rate, 1200 sailors will need to complete the baseline survey. Each EAAA training session has a maximum capacity of approximately 25 sailors. Historically, because the EAAA team at NATTC has experienced some no-shows, the team is allowed to recruit up to 30 sailors for each training. Therefore, NORC anticipates recruitment for the evaluation will be needed for a minimum of 15 EAAA training sessions over the course of the evaluation period to meet the number of participants for the sample goal, equivalent to recruiting at 30 INDOC sessions, at a minimum, to meet the necessary sample. See Table 2 below for more information on INDOC sessions and building the sample pool.
A Training Facilitator Fidelity Checklist will be administered to track the fidelity of the content delivered. A CNRSE IPPW staff member will sit in on six 2-day EAAA training sessions and complete the training facilitator fidelity checklist to capture how facilitators are adhering to the training curriculum. Each observation will cover a full delivery of the EAAA training. The CNRSE IPPW staff member will record what curricular content was missed or covered differently and any content that was added because of participant questions or other reasons.
Table 1. Data Collection Components by Evaluation Group with recruitment ranges
Groups |
Data collection component |
Training Group: Female sailors interested in EAAA who receive the complete EAAA training. |
|
Comparison Group A: Female sailors who will not receive the complete EAAA training but expressed interest in training and are willing to participate in the evaluation. |
|
Comparison Group B: Female sailors not interested in EAAA but willing to participate in the evaluation. |
Table 2. NATTC INDOC Session – EAAA Interest and Potential Sample Pool
Average Weekly INDOC Attendees |
33.5 |
Average Expressed Interest in EAAA |
32 (96%) |
Maximum Sailors randomized into Program Group per INDOC |
15 |
Maximum Sailors randomized into Program Group per EAAA Cohort |
30 |
Target EAAA Training Group at Baseline |
450 |
Minimum # INDOC Session Required |
30 |
Minimum # EAAA Cohorts Needed |
15 |
Procedures for the Collection of Information
Describe any of the following if they are used in the collection of information:
Statistical methodologies for stratification and sample selection.
The EAAA outcome evaluation design involves both self-selection and randomization. This unique hybrid design is a combination of a self-selection trial (SST) and randomized control trial (RCT). Female sailors at NATTC will receive invitations to the evaluation and can self-select whether to volunteer for the opportunity to participate in the EAAA training or only to participate in the evaluation. Then, female sailors interested in the EAAA training will be randomized into two groups: those who receive the opportunity to complete the EAAA training over one weekend at NATTC and those who expressed interest but were not randomized to receive the training. This randomization is necessary because there is not enough capacity or resources at NATTC to offer EAAA to all female sailors who express interest, and it enables the most robust evaluation design (an RCT) for detecting training effectiveness. Depending on female sailors’ interest in participating in the evaluation without participation in the EAAA training, the research team is also prepared to randomize sailors' opportunities to be a part of Comparison Group B to manage the size of that cohort of respondents as needed.
Six months after the Baseline Survey, all respondents (Training Group, Comparison A, Comparison B) will be invited to participate in the 6-Month Follow-up Survey via an email invitation. Those who do not respond to this invitation will receive periodic reminder emails (up to 5) and text messages (up to 2) for a maximum of 8 weeks after the initial invitation to improve response rates.
Target sample sizes for all three evaluation groups across each data collection component are described in Table 1 above.
Estimation procedures.
Not applicable.
Degree of accuracy needed for the Purpose discussed in the justification.
To ensure the credibility of the evaluation findings, NORC has conducted statistical power calculations to determine the credibility of detecting a significant program effect at specific sample sizes. NORC calculated power estimates for Cohen’s f using G*Power 3.1 software, assuming 80% power and an alpha level of 0.05. Cohen’s f is the standardized effect measure for ANOVA and regression models, where 0.10 is a small effect, 0.25 is medium, and 0.40 is large. NORC proposes recruiting between 1,050-1,200 total participants to the evaluation at baseline, with roughly 40% recruited each to the Program and Comparison A groups and the remaining 20% recruited to the Comparison B group. NORC expects to retain 40% of the baseline sample at follow-up, and we will be able to detect small effects (f = 0.15) with the minimum expected analytic sample (480). To the extent that a greater proportion of the baseline sample is retained, NORC will be able to detect smaller effects. Power for subgroup analyses (such as prior sexual assault victimization) will vary depending on the final sample size and subgroup of interest.
Unusual problems requiring specialized sampling procedures.
We do not anticipate a need for specialized sampling procedures given the study design.
Use of periodic or cyclical data collections to reduce respondent burden.
Data collection efforts and corresponding burden depend on which group the sailor is randomized to or selects into. Two data collection points at minimum are necessary to identify change over time. The first quantitative survey (Baseline Survey) is administered in person prior to the intervention (EAAA training) and the second quantitative survey (6-Month Follow-up Survey) is administered online 6 months after the intervention. Without a follow-up quantitative survey for all groups, NORC is unable to assess the outcomes associated with the EAAA training. Given the need to identify change over time, a minimum of two quantitative surveys is required.
Training Group: Sailors who participate in the EAAA training will complete two quantitative surveys.
Comparison Group A: Sailors who participate in the evaluation but are not randomized into the EAAA training will complete two quantitative surveys.
Comparison Group B: Sailors who participate in the evaluation but are not interested in the EAAA training will complete two quantitative surveys.
Maximization of Response Rates, Non-response, and Reliability
The findings of the EAAA evaluation will be generalizable to female sailors at NATTC, because the sampling frame includes all female sailors as they start at NATTC during the data collection time period. We will use attrition analysis to assess non-response, comparing baseline respondents to follow-up respondents on key demographics. Additionally, we will use propensity score weighting to identify a statistically equivalent subsample of respondents for our analyses to minimize bias.
There are several facets of the research design that will contribute to a strong response rate for this data collection and thus overall reliability and validity of the EAAA training evaluation effort. The research team has been engaged with the CNRSE IPPW staff and team overseeing EAAA at NATTC over the past year to understand the context of the EAAA programming as well as approaches to EAAA implementation. The regional prevention coordinator team overseeing EAAA at NATTC is fully informed of and approves the training evaluation design. The recruitment protocols and survey language have been carefully reviewed with CNRSE IPPW staff, EAAA facilitators, SAPRO staff, and internal NORC experts in the field of sexual harassment and sexual assault prevention; and with a small group of female sailors at NATTC to assure that the recruitment and survey language is understandable, relatable, and acceptable to the target population at NATTC. Additionally, the EAAA training has, separately, been externally evaluated through a published randomized control trial (RCT) conducted in a civilian University environment, further informing which validated and reliable scales would be acceptable to this age group, once modified for the NATTC experience. NORC has developed the data collection protocols to be consistent with the best practices in the field on survey design and implementation. NORC has tested the offline survey environment on the tablets to be used during the evaluation at baseline for in-person recruitment and data collection to ensure the recruited participants will encounter a user-friendly design without technical glitches, enabling easy survey participation at baseline. At the 6-month follow-up period, the sailors will no longer be located at NATTC and will be contacted via email and text message to participate in the online follow-up survey. NORC has tested the online survey link on different web browser platforms using different NORC laptops, personal computing devices, and personal mobile devices to ensure participants at follow-up will also encounter a user-friendly design for sailors to complete when they have access to the internet.
Respondents will also be offered modest financial incentives for participation based on guidance from the EAAA training team at NATTC following established practices in the field of survey research or reliable and valid data collection. There are incentives associated with each component of the EAAA evaluation to bolster participation. As noted above, while attrition at the follow-up stage is a common phenomenon in survey data collection, sailors are not regularly offered tokens of appreciation for survey participation, and thus, the current data collection is expected to achieve a strong response rate. Tokens of appreciation have been tailored to the local context and geared to the needs of the sailors. At both Baseline and the 6-Month Follow-up, sailors will have the ability to choose between receiving an incentive to the Navy Exchange (NEX), which is usable at all Navy installations, or Amazon. At the time of follow-up, the participating sailors may not be located near a NEX and may prefer a different option.
Tests of Procedures
The evaluation recruitment and survey language were reviewed during several (n<9) one-on-one discussions on instrument and implementation feedback with current female sailors familiar with the environment and culture of the evaluation site (NATTC). The sailors provided feedback to ensure the recruitment and survey language is understandable and acceptable to the target population of NATTC sailors.
5. Statistical Consultation and Information Analysis
Provide names and telephone number of individual(s) consulted on statistical aspects of the design.
Provide name and organization of person(s) who will collect and analyze the collected information.
Elizabeth Mumford (NORC)
Bruce Taylor (NORC)
Cynthia Simko (NORC)
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Patricia Toppings |
| File Modified | 0000-00-00 |
| File Created | 2026-01-07 |