Information Collection Request
New
Healthy Schools Program Evaluation
Supporting Statement B
Program Official/Contact
Yulia Chuvileva, PhD, MA, MSc
Health Scientist
National Center for Chronic Disease and Health Promotion
Centers for Disease Control and Prevention
770/488-0027
July 10, 2024
Supporting Statement B
Collections of Information Employing Statistical Methods
Respondent Universe and Sampling Methods
The Respondent Universe for the evaluation of the CDC-RFA-DP-23-002 Healthy Schools Program (referred to as the 2302 Healthy Schools Program) includes four respondent types (Table 1):
(1) Staff contacts in 20 state level cooperative agreement recipient organizations funded by CDC-RFA-DP-23-002 Healthy Schools Program. Cooperative agreement recipients include state education and health agencies, universities, and a tribal nation. One designated staff member within each recipient organization will submit required monthly progress reports to CDC. In addition, up to three staff contacts per recipient organization will participate in key informant interviews in years 2 and 4 (Attachment X).
(2) Staff contacts in 20 priority local education districts (priority LEA) located within the recipient states. Each state identified/selected one priority district for concentrated implementation and evaluation efforts. Districts range in size from 1 to 52 schools. Up to three staff contacts in each priority LEA will participate in key informant interviews in years 2 and 4.
(3) Staff contacts in approximately 250 participating schools located within the priority LEAs (also referred to as corresponding schools). A census of participating schools (1 contact per school) will submit an annual school level questionnaire. We anticipate a 75% response rate from participating schools.
(4) Students in grades 4-12 attending participating schools located within the priority LEAs (estimated 13,150 annually). Schools and districts were selected by virtue of their participation in the program. To obtain a representative sample of students in each priority LEA, a random sample of students in participating schools will complete the student questionnaire. Inferences will be made only to the priority LEAs.
Table 1. Maximum Number of Respondents Annually per Information Collection/Respondent Type
Information Collection |
Respondent Type |
Maximum Number of Respondents for Each Data Collection Year |
Monthly Progress Reporting Form |
Recipient personnel |
20 |
Total |
20 |
|
Key informant interviews with recipient staff |
Recipient personnel |
40 |
Total |
40 |
|
Key informant interviews with priority LEA staff |
Priority LEA Personnel |
40 |
Total |
40 |
|
School questionnaire |
School personnel |
250 |
Total |
250 |
|
Student questionnaire |
Students |
13,150 |
Total |
13,500 |
Sampling
Methods
We plan to obtain a census of participants for the monthly progress reporting form and key informant interviews with recipient staff and priority LEA staff. We intend to administer the school questionnaire to a convenience sample of corresponding schools within the priority LEAs that have agreed to participate in 2302 school health activities. We also plan for the student questionnaires to reach students in each of the corresponding schools in a sample that is representative of each priority LEA’s set of corresponding schools. the school questionnaire, and the student questionnaire. To ensure student questionnaires reach a representative sample, we plan to estimate the sample sizes needed to generate precise estimates for schools of different sizes at 95% confidence intervals within +/- 5 percentage points. We expect to need 850 students from 22 small schools to complete the survey (we plan to invite all students to participate). We calculate the need for 3,500 from 35 medium schools, and 8,800 students from large schools to complete the student survey (sample approach that is adjusted for size of school district). We plan to survey a total of 13,150 students each year across the 20 priority LEAs.
Sample size calculations are premised on expected student participation rates of 80% in schools with passive consent but as low as 20% in schools where active consent is needed. We plan to use a systematic sampling approach to randomly select classrooms from a list of eligible classrooms generated by each school; the sampling interval will be determined after the school prepares the list of eligible classes. This sampling method similar to that employed by CDC’s national and state Youth Risk Behavior Surveys (YRBS).
Information Collection Procedures
ICF is contracted to conduct the evaluation on behalf of CDC. Information collection procedures are described in detail below for each data collection instrument. All data collection activities have been reviewed and approved by ICF’s IRB (federal-wide assurance #00002349, expires June 15, 2028). IRB approval notification is included in Attachment 20.
Monthly Progress Reporting Form
The Monthly Progress Reporting Form (Attachment 3) will be used to monitor funded recipients’ annual and monthly indicators, annual performance measures, and overall program barriers and successes. The Monthly Progress Reporting is required for all organizations funded by CDC-RFA-DP-23-0002 Healthy Schools Program.
Each month, funded recipients will receive an invitation email (Attachment 4) with their unique URL link to access the Monthly Reporting Form programmed in Qualtrics. Screenshots of the programmed form are included in Attachment 5. After reports are submitted, ICF will conduct monthly data quality control checks and follow up with recipients to verify data as needed. All data will be stored in a secure password protected Qualtrics account only accessible to the team. Once exported from Qualtrics, all data will be stored on ICF’s secure network servers and access will be restricted to approved team members identified by user ID and password.
Key Informant Interviews
Virtual key informant interviews will be conducted with funded recipient organizations (Attachment 6) and priority LEA staff (Attachment 7) to understand how they are using 2302 Healthy Schools funding to implement strategies, activities, and achieve outcomes. The interviews will be conducted by the ICF evaluation team and will follow semi-structured interview guides (Attachments 6 and 7).
An initial email invitation (Attachment 8) will be sent to each funded recipient and priority LEA partner in the Spring semester of interview data collection years 2 and 4 (2025, 2027). The initial email invitation will be sent at least 30 days prior to data collection. This initial email invitation will describe the purpose of the interviews, the expected time commitment, the way the information collected will be used, and the next steps. ICF will then work with each funded recipient and priority LEA partner to schedule the interviews at a time most convenient for the interviewees.
All interviews will take place virtually through Microsoft Teams, ICF’s secure platform for data collection. Prior to the start of each interview, interviewees will be provided an Informed Consent Statement (Attachment 15) and required to provide verbal consent for their participation and for audio-recording. The informed consent statement emphasizes the voluntary nature of participation and lack of any consequences for choosing not to complete any or all of the interview. The consent statement also states that responses will aggregated and presented at the district level and will not be attributable to individual interviewees or their schools, nor linked with any identifying information.
All interviews will be audio-recorded for transcription and thematic analysis. Audio files and transcription files will all be stored on ICF’s secure network, which is password protected and only accessible by approved project team members. The ICF evaluation team will review transcripts to ensure complete and accurate transcriptions. An initial codebook will be developed using the interview guide and evaluation questions. Intercoder reliability will be established and trained ICF team members will code the interview data using the qualitative data management software, MAXQDA. Open and axial coding will be conducted, and salient categories of information representing themes will be identified.
Healthy Schools Questionnaire
The Healthy Schools Questionnaire (Attachments 9) will be sent to all schools in each priority LEA annually to collect information on school health policies, practices, and services related to physical activity, nutrition, and chronic health condition management. ICF will provide step-by-step instructions to recipients and their priority LEAs to notify and invite the schools/school principals in the priority LEAs to respond to the questionnaire each Spring semester of the cooperative agreement (2025, 2026, 2027, 2028). The instructions will include language that can be shared with priority LEAs and their corresponding schools with a unique, password-protected link to the school questionnaire for each school. Screenshots of the programmed questionnaire in Qualtrics are included in Attachment 10. The electronic consent form is included in Attachment 16. School staff must agree to participate in order to access the questionnaire. If they select “I have read the above information and I DO NOT wish to participate.” they will not be able to access the questionnaire.
Completed questionnaires will only be accessible by the ICF team via a password protected account. Data will be exported to a preferred statistical software for further data cleaning and analysis. All data files will be stored on ICF’s secure network servers, and access will be restricted to approved team members identified by user ID and password.
Healthy Student Questionnaire
The Healthy Student Questionnaire (Attachment 11) assesses students’ physical activity and nutrition behaviors, health related support received from school, feelings about school, and grades. The student questionnaire was developed using a combination of validated questions from previous research and national surveillance systems (e.g., Youth Risk Behavior Survey, School Physical Activity and Nutrition Survey) and new items.
ICF will provide step-by-step instructions to recipient organizations and their priority LEAs and corresponding schools to administer the electronic questionnaire (programmed in Qualtrics) within a specific date range during Spring semesters of the cooperative agreement (2025, 2026, 2027, 2028). ICF will provide materials to priority LEAs and schools, who will be responsible for notifying parents of the questionnaire procedures and securing passive or active parental consent as required by the district. Consent and assent forms will be available as paper or electronic versions depending on state or district requirements. Students under the age of 18 will need parent/guardian consent to participate in the questionnaire (Attachment 17). Students ages 18 and older will be given consent forms to complete for themselves (Attachment 18) and will not need parental/guardian consent. Before questionnaire administration, all students will also be asked to assent to completing the questionnaire (Attachment 19). If a student does not assent, they will not be asked to complete the questionnaire.
Each corresponding school will receive a unique and secure password-protected link to the student questionnaire, which will be distributed to students in classes that are randomly selected to participate. Completed questionnaires will only be accessible by the ICF team via a password protected account. Data will be exported to a preferred statistical software for further data cleaning and analysis. All data files will be stored on ICF’s secure network servers and access will be restricted to approved team members identified by user ID and password.
Methods to Maximize Response Rates
Specific methods to increase the response rates and minimize the burden are provided in Table 3 below.
Table 3. Methods to Maximize Response Rates
Information Collection Instrument |
Methods to Maximize Response Rates |
Monthly Progress Reporting Form |
The monthly progress report is a requirement of CDC-RFA-DP-23-002 cooperative agreement recipients. ICF will send monthly invitations to recipients to complete the forms. CDC Project Officers will remind recipients of the form due dates during their monthly Teams calls with recipients and follow up with non-responders as needed. |
Interviews with staff contacts in recipient organizations and priority LEAs |
The CDC will facilitate direct outreach to recipients to initiate contact with the ICF evaluation team for the purpose of scheduling the key informant interviews. Similarly, recipients will facilitate outreach to priority LEA contacts to initiate contact with a staff member to participate in the interview. Numerous date and time options will be offered to participants for scheduling each group interview. Once scheduled, an electronic meeting invitation will be emailed to participants with instructions to join the conference call. Project Officers will be asked to follow up directly with the recipient/priority LEA if the ICF evaluation team does not receive a response within seven business days. |
School questionnaires |
Initial communication to schools will be initiated by the priority LEA to facilitate buy-in for the data collection. Clear and concise communication materials will be provided to the priority LEAs to distribute to schools to convey the purpose and importance of the information collection activities, with clear instructions for school staff to complete the school questionnaire. Schools will have a generous window of time to complete and submit the school questionnaire. |
Student questionnaires |
Initial communication to schools will be initiated by the priority LEA. A district level sampling approach will reduce the burden on schools and students by reducing the number of students asked to complete the questionnaire. Clear and concise communication materials will be provided to the priority LEAs to distribute to schools to convey the purpose and importance of the student questionnaire, with clear instructions for selected schools to administer the student questionnaire in their classrooms. Schools will be given a generous window of time (6-8 weeks) to administer the questionnaire during a time most convenient for their school schedule. Schools will notify parents of the student questionnaire using passive or active consent procedures as required by the district. |
Tests of Procedures
The Healthy Student Questionnaire includes a combination of previously validated and new items designed to address annual performance measures for the 2302 program. The student questionnaire (Attachment 11) was pilot tested with eight students in grades 4-12. Each pilot test participant completed the questionnaire followed by a brief cognitive interview. Students were recruited to participate in the pilot testing through ICF evaluation team members’ professional network via email to parents or guardians. The pilot test protocol was approved by ICF’s IRB. The pilot test was used to make minor adjustments to the student questionnaire and to estimate the completion time of the questionnaire.
The monthly reporting form was tested with funding recipients to ensure it was feasible, accurate, and provided the most valuable information about activities’ progress. This testing helped to validate the monthly form and ensure that the burden is acceptable to funding recipients. The form should take no longer than 30 minutes each month to complete.
We also tested the year 2 and year 4 key informant interviews with funding recipients, and found they took no longer than 60 minutes to complete.
Statistical Consultants
The individuals consulted on statistical aspects and study design:
Division of Adolescent and School Health, National Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control and Prevention
4770 Buford Highway, Atlanta, GA 30341
Yulia Chuvileva, PhD, MA, MSc
Health Scientist, Research Application and Evaluation Branch
Leah Robin, PhD
Team Lead, Research Application and Evaluation Branch
Catherine Rasberry, PhD
Branch Chief, Research Application and Evaluation Branch
Shannon Michael, PhD
Health Scientist, Research Application and Evaluation Branch
ICF
2635 Century Parkway Suite 1000, Atlanta GA 30345
Dana Keener Mast, PhD
Director, Research Science
Sarah Conklin, PhD
Manager, School Health
The individuals responsible for overseeing data collection and analysis are:
ICF
2635 Century Parkway Suite 1000, Atlanta GA 30345
Dana Keener Mast, PhD
Director, Research Science
Sarah Conklin, PhD
Manager, School Health
The individuals responsible for collecting and analyzing the data:
ICF
2635 Century Parkway Suite 1000, Atlanta GA 30345
Sarah Conklin, PhD
Manager, School Health
Rumour Piepenbrink, MPH
Public Health Senior Research Scientist
Alyssa Contreras, MPH
Senior Research Scientist
Keirsten Andersen, MPH
Senior Public Health Research Scientist
Zach Timpe, PhD
Research Scientist V
The following individuals will serve as statistical consultants to this project:
ICF
2635 Century Parkway Suite 1000, Atlanta GA 30345
Zach Timpe, PhD
Research Scientist V
Ronaldo Iachan, PhD
Methodologist/Statistician, Senior Director
Ronaldo.Iachan@icf.com
Robert Stephens, PhD
Senior Research Methodologist
The agency staff person responsible for receiving and approving contract deliverables is:
Division of Adolescent and School Health, National Center for Chronic Disease Prevention and Health Promotion, Centers for Disease Control and Prevention
4770 Buford Highway, Atlanta, GA 30341
Jeffrey Miller
Public Health Analyst
770-488-2651
Attachment 1: Authorizing Legislation—Public Health Service Act [42 U.S.C. 241]
Attachment 2: Authorizing Legislation—Evidence-Based Policymaking Act 2018
Attachment 3: Monthly Reporting Form
Attachment 4: Monthly Reporting Form Email Notification
Attachment 5: Screenshots of Monthly Reporting Form
Attachment 6: Interview Guide Recipient
Attachment 7: Interview Guide Priority LEA
Attachment 8: Interview Invitation Emails
Attachment 9: Healthy Schools Questionnaire (Elementary and Middle/High)
Attachment 10: Screenshots of Healthy Schools Questionnaire
Attachment 11: Healthy Student Questionnaire (Elementary and Middle/High)
Attachment 12: Screenshots of Healthy Student Questionnaire
Attachment 13: Published 60-day Federal Registration Notice Healthy Schools
Attachment 14: Public comments and responses
Attachment 15: Interview Consent Form
Attachment 16: Healthy Schools Questionnaire Consent Form
Attachment 17: Healthy Student Questionnaire Active and Passive Consent Forms (Parent/Guardian)
Attachment 18: Healthy Student Questionnaire Consent Form (18+)
Attachment 19: Healthy Student Questionnaire Assent Form (<18)
Attachment 20: ICF IRB Approval
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Information Collection Request |
| Author | Zanakos, Sophia |
| File Modified | 0000-00-00 |
| File Created | 2024-09-06 |