LSAMP Cognitive Testing Findings

LSAMP COGNITIVE TESTING FINDINGS

August 8, 2024

Table of Contents

SUMMARY: COGNITIVE TESTING METHODS 2

SECTION 1: STAFF & ADMINISTRATOR INTERVIEW PROTOCOL – COGNITIVE FINDINGS 4

SECTION 2: STUDENT FOCUS GROUP PROTOCOL – COGNITIVE FINDINGS 14

SECTION 3: SURVEY – COGNITIVE FINDINGS 19

Appendix 1: LSAMP – STAFF & ADMINISTRATOR INTERVIEW PROTOCOL 20

Appendix 2: LSAMP CASE STUDY PROTOCOL: STUDENT FOCUS GROUP 23

Appendix 3: LSAMP CASE STUDY PROTOCOL: SURVEY QUESTIONS & TOPICS 25

SUMMARY: COGNITIVE TESTING METHODS

Cognitive recruitment and testing occurred from May 2024 to July 2024. NSF provided the recruitment list for administrators and faculty, and students were referred by some of these faculty (upon participation in the cognitive testing interview). Recruitment occurred using three methods: email, phone calls, and NSF outreach. From the initial NSF-provided list of 18 individuals from across alliance types (all of whom are not in a selected case-study site), nine administrators and faculty completed a cognitive interview. An additional seven students¹ ultimately completed a cognitive interview, for a total of sixteen unique participants (Table 1).

Interviews were completed virtually on Teams and recorded for later review if participants consented. Interviews lasted approximately one hour, and no incentive was provided. The interviews were conducted by a member of the NORC team, with no note-taker or observer present. In the interest of covering as much material as possible, each of the three instruments was presented in order of importance, and the survey was considered exploratory generating information for future consideration². For administrators, it was the administrator interview, followed by the student focus group protocol, and as time allowed, the survey. Students only saw and provided feedback on the student focus group³.

Ultimately, the administrator interview was tested with 8 individuals. The student focus group was tested with 14 individuals (7 administrators and 7 students), and the survey was tested with 8 individuals. The preliminary findings from these tests are presented below, by individual instrument.

A note on reviewing findings: each initial set of questions (directly from the approved protocol) are presented, followed by key findings and suggestions following cognitive testing. Suggestions result from the review of cognitive testing feedback focusing on respondent understanding, recall, and key topics shared for us to consider for future inclusion (e.g., respondents sharing important information we hadn’t already included). In reviewing the summaries below, note that for most sections comprehension and recall were adequate and require minimal revisions (e.g., defining a term, removing a redundant topic).

In addition to the findings directly related to the instruments, there were several key lessons learned as it applies to recruitment of participants. These should be considered when developing the final recruitment design in the data collection plan. First, much of the information around the operations of the program can be centrally accessed—meaning, that information may be held by an individual or position—particularly in smaller institutions. However, this would not include experiences of faculty, students, interactions beyond the department, etc. As such, the opportunity to both complete outreach to multiple individuals (e.g., anyone named in proposals or annual reports), and to refer other individuals for participation, or to collect responses prior to the completion of the interview, would improve data quality.

This approach of broader invitations, with referral or collaboration, also helps to alleviate the second main observed challenge during cognitive testing, which was contacting the intended individuals. For many individuals, we were unable to obtain a response via email or phone outreach (and many phone lines were either unavailable or out of date). We are aware that people switch roles within an institution or alliance, or may move to another location entirely, making a contact strategy focused on one individual riskier and more challenging. The final key takeaway we learned as it pertains to recruitment is the frequency and duration of contact necessary to complete interviews with these individuals. While we anticipate the case study approach will alleviate some of these concerns, these individuals are very busy with frequently changing schedules. We will need to account for this with multiple contacts, reminders, and flexibility in order to respectfully garner their participation.

Resulting revisions incorporate specific modifications including NSF requests and suggestions, as well as offering updates that will best facilitate the interviews. Overall, while modifications may look significant, the necessary content changes are relatively minimal. Each section is marked as ‘modify’ if any changes are necessary or ‘retain’ if no changes are required. Resulting recommendations, including recommendations around future recruitment efforts, can be found in the Appendices.

Table 1. Summary of Completed Cognitive Interviews

	Completed Lead Interviews	Completed Partner Interviews	Administrator Interview Tested	Student Focus Group Tested	Survey Tested
Administrators
Arkansas	3		2	3	3
TAMU	2		2	1	1
SW Georgia Pathways	2		2	2	2
University of Michigan	1		1	0	1
STAMP	1		1	1	1
Students	7			7
TOTAL	16		8	14	8

SECTION 1: STAFF & ADMINISTRATOR INTERVIEW PROTOCOL – COGNITIVE FINDINGS

Section A: Introduction

[Goal: orient to person, so we know what questions to ask; how Respondent understands their role in LSAMP.]

1. Can you tell me a bit about your current position? How long have you been in it? What are your primary job responsibilities?

2. [If they did not describe LSAMP] What are your responsibilities directly related to LSAMP [at institution]? What does your involvement with LSAMP look like? Can you provide an example?

3. [If they only describe LSAMP] What are your responsibilities outside of the LSAMP alliance?

Cognitive Interview Findings:

This section challenged interviewees, especially in knowing exactly what information we were seeking. More directed questions would be easier to ask and for respondents to share greater depth of information. Testing participants expressed concern about being able to answer correctly or give us the answer that we’re seeking. Some simple changes would help clarify. For example:

1. How do you refer to the program, if not LSAMP (e.g., BD)?

2. What do you do for LSAMP, officially?

3. How else do you help LSAMP, even if unofficially?

4. Are there dedicated (non-faculty) staff assigned to work with the LSAMP program?

5. How long have you been working with the LSAMP program?

6. Can you tell me a bit about your other work, that is, work not related to LSAMP?

Resulting suggestion: Modify.

Section B: Description of LSAMP Activities

Activities

1. (Must) Please describe your current LSAMP-related activities. What are the activities you do?

   1. Have these changed over time? If so, how and why?

2. Please describe how your current LSAMP-related activities are executed.

   1. Who is in charge? Who maintains? Etc.?

3. How did you pick these particular activities?

4. [If not mentioned] Tell us about the professional development component of your alliance.

Cognitive Interview Findings:

One key challenge shared by interviewees in thinking about these questions is that the types of activities offered and the number of students served are dictated by the grants themselves. If targets are not being met, the lead member/PI needs to resolve the discrepancy to meet the requirements of the grant. This suggests that this section needs to have more questions and framing around the alliance collaborations, which staff/faculty are handling which aspects of programming and help identify the scope of what we are asking (the lists of activities can be very long). Suggestions include:

1. Separate question 1 into two parts.

2. Use the word ‘activity’ as it aligns with reporting.

3. Specify virtual v. in-person activities.

4. Simplify reporting, so lists of activities are offered or tailored, since there are many.

5. Simplify words and presentation (e.g., ‘executed’ in question 2 was confusing).

6. Professional development can mean students, staff alliance level, or staff institution/LSAMP focused, so we need to be clear and specify.

Resulting suggestion: Modify.

How work gets done

5. (Must) What types of meetings or other engagements do you attend as a member of LSAMP?

6. How do you interact with other institutions within your alliance?

   1. Probe. Is communication by email, Teams, f2f? What frequency? Others?

7. How were the other institutions in your alliance selected?

   Cognitive Interview Findings:

   Interviewees noted that they attend many meetings so knowing what types of meeting the interview was interested in would be useful for Q5. Similarly, when asking about components of the programming, interviewees recommended using language similar to that of the annual reports (e.g., use the word activity rather than program or programming). Interviewees also noted that only staff who had been present when some of these decisions were made could answer questions about selection, so rather than altering question, we may need to only ask it of certain positions. Suggestions include:

   1. Specify or separate meeting and obligations (including words like ‘here’) at campus-specific, alliance-level, or NSF-level. Be sure to specify if meetings with students are meant to be included.

   2. Limit asking question 7 to those who were there at the start of the alliance or seek an individual with that information—people who were not there at the beginning would not have access to this information. Finding a way to refer out to other people, and get additional input, would be very informative and helpful for better understanding the history and makeup of institutional involvement and alliances in general.

   Resulting suggestion: Modify.

8. Who is responsible for executing the LSAMP program here?

Institutionalization

9. (Must) How does the LSAMP work get done?

   1. Note: This is not about the activities themselves, but more about the administrative work that supports it. We will want to get at routines, both formal and informal.

   2. Probe: Who is doing the work? What are their positions?

   3. Probe: Are there any on-campus champions for the work? Who are they? How do they champion the work?

   4. Have you received any guidance from NSF? If so, what is that guidance?

10. (Must: ask of PI) Are there particular positions at your institution dedicated to LSAMP-related work?

    1. If so, what is the position and when was it created?

    2. If not, how is the overall work of the LSAMP distributed across your staff?

    3. How are they funded? (e.g., through LSAMP grant)

    4. If the funding went away, would these positions still exist?

Cognitive Interview Findings:

Questions 9 and 10 were viewed as redundant to earlier questions, so they should be considered for modification or removal. Alternatively, a question could ask about whether—if funding for staff went away—LSAMP programming would still be offered. At this point in the guide, many components (e.g., Q10) were organically covered in other sections. Suggestions for modifications include:

1. Ask questions earlier (e.g., Q10) or remove them from this section.

2. Modify current section questions to focus on expanding on detail (e.g., how would the program operate administratively in the absence of funding).

3. Offer context to help understand what level or organizational structure (e.g., alliance or institution level) in the questions.

4. Simplify word use, ‘champions’ is unclear and offer an explanation of what we mean.

Resulting suggestion: Modify.

Student recruitment/retention

11. (Must) What does your student recruitment process look like?

    1. Probe: how do you ensure you are receiving quality candidates?

12. (Must) What do you do to support student retention?

Cognitive Interview Findings: While we conceptualized student retention as a primary goal of this work and working with students, some respondents considered the goal of LSAMP to be the placement of students in higher-level STEM education. The work of staff in the programs is reported to be about advising and student support activities—which may retain students but is not a core goal. That said, interviewees reported that they were interested in knowing the answer to the question about student recruitment, so having access to this data may be considered helpful by institutions.

Resulting suggestion: Retain.

Section C: Contextual Understanding of LSAMP goals

[Goal: identify R’s understanding about LSAMP and the goals of the program.]

1. What was your institution’s motivation for creating an LSAMP program?

   1. Probe: What types of students does LSAMP cater to?

      1. What characteristics/elements do you think position students to succeed at your institution?

      2. What elements do you think are necessary for a student to succeed in STEM at this institution?

   2. Probe: Were you involved in proposal planning for the project?

      1. [If yes] What were your or the institution’s reasons for applying to the LSAMP program? Thinking back, what were your expectations for having an LSAMP alliance at the time?

      2. Note: Get institution specific, alliance specific and national related reasons if they have them.

Cognitive Interview Findings: Interviewees presented shared challenges in this section. First, many components (e.g., the creation of the LSAMP program) predate the individual with whom we are speaking. Having an opportunity to bring in more people to provide this information would be helpful. Additionally, we need to ask more about challenges. For example:

1. Ask what challenges exist when engaging with the business office, financial aid, or human resources. All these offices are needed to help the program move smoothly and are often a challenge. For example, not distributing stipends or understanding how to work within NSF regulations.

2. Ask specifically about race, income, first-generation college students, and transfer students.

3. Frame questions to acknowledge that while institutions or programs may have priorities, they are operating within the context of needing to align with NSF priorities.

Resulting suggestion: Modify.

2. How does your LSAMP program contribute to your institution’s goals of enhancing access to STEM for all students?

   1. Probe: How do you think this institution works for students of color in general? In other words, who does not participate in LSAMP programming?

   2. Note: in other words who does not participate in LSAMP program?

3. What kinds of additional supports do you feel your students need overall?

4. How does your LSAMP program work with other institutional initiatives?

   1. Note: is this isolated effort or connected with other efforts on campus?

Cognitive Interview Findings: Interviewees reported that not all institutions have the same goals, but all will work toward the LSAMP goal of increasing access to STEM fields. Similarly, success may be measured differently, as would ‘student success’. Are we speaking on graduation rates? Careers? Retention in STEM? Suggestions include:

1. Add greater specificity to our questions so we are clear about exactly which outcomes we are seeking.

2. Offer examples to add greater clarity.

Resulting suggestion: Modify.

Section D: Impact of LSAMP Activities

1. What are some achievements you have had in LSAMP?

   1. [If students not discussed, then] what have been some successes with students?

   2. Probe: which activities have been most helpful in supporting students?

   3. Are there students that these activities work better for than others? How do you know?

   4. Has LSAMP increased the number of students completing a STEM degree?

2. (Optional) What successes are you most proud of?

3. What have been some of the challenges? How have you addressed them?

4. (Must) How do you obtain feedback on your LSAMP programming?

   1. Probe: From students? From faculty?

   2. Note: If they mention data/survey, request copies.

5. (Optional) What LSAMP-related activities do you think have been less helpful for supporting students? How do you know?

   1. Note: Probe for differences between recruitment, support, and retention

   2. Are there students that these activities are less helpful for than others? How do you know?

6. (Optional) If you could redo any part of the LSAMP design now, with the benefit of hindsight and experience, what would it be? Why? What difference do you think this change would make and why?

Cognitive Interview Findings: This section presented challenges in terms of how we want interviewees to think about achievements and successes. These could be at the alliance, institution, staff, or student level. They could also extend into whether we seek information about the program, institutional intra-department collaboration, etc. Consider modifications to add clarity and definitions around these concepts, and as possible, integrate or move into earlier sections where many of these topics are organically discussed. Suggestions include:

1. Focusing the questions on specific groups or experiences (e.g., between institutions, departments).

2. Specify specific challenges or populations we’re referring to (students, which type of student, staff).

Resulting suggestion: Modify.

Section E: Influence of the Alliance

1. (Must) How do you engage with LSAMP members at other alliance institutions, if at all?

   • Distinguish between students, staff, administration.

2. (Optional) If LSAMP funding goes away, what pieces of the program do you think would be sustained?

Cognitive Interview Findings: Interviewees shared that the organization and relationship between the institution and the alliance (in terms of requirements, responsibilities, etc.) shaped experiences. Several modifications are suggested here, including moving this section to the introduction to ensure context if offered at the start of the discussion. For example:

1. Add a question about collaboration across the alliance, prior to engagement. “Can you tell me how your institution interacts with other institutions across the alliance? What about with institutions from other alliances?”

2. What incentivizes or disincentivizes institutions to seek large(r) grants?

Resulting suggestion: Modify.

Section F: Wrap-up

1. Based on what we’ve talked about today and the types of questions we’ve been discussing, is there anything else you think I should have asked that I didn’t? 

Cognitive Interview Findings: Funding, and timing of funding, was a theme presented by interviewees. Specifically, interviewees discussed the challenges of limited staff/faculty funding to help support the funded-students (e.g., for conference travel). This inherently limits the students, in the faculty are not supported enough to attend/attend fully. Suggestions include:

1. Ask directly about the resources staff/faculty need for themselves to fully support students. This funding and support is a core theme throughout the interviewees’ contextualization of thinking the successes and limitations to this program.

Resulting suggestion: Modify.

Summary Notes: Administrator Interviews

Beyond the individual questions, several themes emerged from the cognitive testing. To begin, the themes of funding are commonly dictating how everything else operates and is thought about. Second, much of the functioning and establishment of alliances and involvement predates those we will be speaking with. Third, there are societal pressures and factors (e.g., political pushback about race-based efforts in higher education) that are dictating how these programs can operate.

Resulting considerations:

1. Our questions and data collection methods should respond to these concerns by seeking to capture this data in multiple methods from as many involved parties as possible, as very few individuals hold all the answers.

2. This both allows and requires us to be very specific and thoughtful in our wording of questions.

3. Certain institutions (e.g., community colleges) could not be a lead LSAMP institution until 2015, which greatly changes their experiences and institutional recall.

There were also several topics commonly presented by interviewees that they wanted to be asked, and they wanted to know about other institutions. These include:

1. What supports to institutions have or need to host annual alliance meetings (e.g., in DC) so they can collaborate?

2. NSF has asked for demographic data to stop being collected, but how are they supposed to measure outcomes now?

3. How does the LSAMP program differ from the main institution to its satellites? How does this differ by alliance, state, or region?

4. How can local alliances better network and collaborate?

5. The NSF grant doesn’t allow for LSAMP-only staff to focus solely on the students, but does the institution?

6. How does LSAMP communicate and collaborate with other academic departments within an institution?

7. How much do faculty know about the workings of LSAMP, and how can they be exposed to more (beyond supporting students)?

Specific modifications to the interviewer guide are presented in Appendix 1.

SECTION 2: STUDENT FOCUS GROUP PROTOCOL – COGNITIVE FINDINGS

Section A: Opening Discussion 

1. What are your names, your program of study, and your year of school. 

2. (Must) We'd like to start by asking you about how you came to be a part of LSAMP. What was that process like? How did you learn about it? 

1. Probe: Were you approached? Did you do outreach? 

3. (Must) What supports has the LSAMP provided you? 

4. (Can) Can you describe the LSAMP program? What does it look like for you? Is that the way it looks/works for everyone? 

5. (Must) What relationships have you developed with staff members?  

 

Cognitive Interview Findings:

Students were able to communicate clearly about how they became part of the LSAMP program, who referred them, and what the process was like. The supports most discussed were around funding and helping to develop the skills to successfully navigate their program and future plans. One core theme was the time between learning about LSAMP, being recommended to apply, and completing the process. Suggestions include:

1. Asking how long their total process took, and why (e.g., nervous about applying, unfamiliar with personal statements and essays).

2. Understand that academic department/major and schools within an institution will look different, seek to better understand these potential variations.

Resulting suggestion: Modify.

Section B: Successes 

1. (Must) What have you found to be most supportive from the perspective of staff or support from an individual? 

1. How have LSAMP staff contributed to the successes of LSAMP programming? 

2. (Must) Which programs have you participated in that were particularly helpful? 

3. (Can) In what ways would you say LSAMP has supported you in your time here? Probe for specific examples. 

4. (Must) (Level 1) How do you use the financial support you receive through LSAMP? How did you come to decide how to use your financial support? 

1. In what ways might these activities shape your future plans? 

5. (Must) (Level 2) Of the LSAMP-related activities you’ve attended or participated in, what have you found to be the most helpful and why?  

6. (Must) (Level 2) Why did you decide to attend these LSAMP-related activities? What were you hoping to learn or gain?  

 

Cognitive Interview Findings:

Students were able to think about the successes of their participation, but some of the terminology was a challenge to think through uniformly. Many students have direct staff/faculty that they work with, but the program has many moving parts (e.g., the distribution of stipends) that impact them directly but are beyond their direct knowledge. Additionally, experiences differ by type of student and time in the program, so we should be prepared to offer more tailored questions to help improve comprehension and applicability.

Resulting suggestion: Modify.

Section C: Gaps/Opportunities 

1. (Can) What have you found to be least supportive from the perspective of staff or support from an individual?  Please do not name any individuals or roles, but rather actions or approaches. 

2. (Can) Which programs do you wish were being offered? 

3. (Must) (Level 1) Is there anything (e.g., supports, guidance, activities) you feel is missing from the opportunities available to you through your LSAMP funds?

   1. What would change for you if these supports, guidance, activities, etc. were offered? 

4. (Must) (Level 2) Is there anything (e.g., supports, guidance, activities) you feel is missing from the LSAMP-related activities that are available to you?  

   1. What would change for you if these supports, guidance, activities, etc. were offered? 

 

Cognitive Interview Findings:

One core area students articulated as an opportunity was helping students get involved in LSAMP earlier in their academic careers. Additionally, how can students in the program see greater supports or rewards? This is a good opportunity for students to report challenges they’ve faced as well—which interviewees reported being minimal in many cases, but easier to articulate. Suggestions include:

1. Differentiating between official LSAMP staff and student mentors or advisors (who may be different people or may not be affiliated with the LSAMP program) may help students better think through their responses and differentiate between experiences.

2. Most students think LSAMP is great, and doing a great job, so they may struggle to identify areas for improvement. Rephrasing questions to present ‘constructive feedback’ may help solicit deeper responses without students feeling like they are ‘creating complaints’.

Resulting suggestion: Modify.

Section D: Level 3 Students Only 

1. (Must) What are your goals for your time here? 

   1. What are your academic goals, and what was your motivation for pursuing this degree? 

2. (Must) Do you participate in any activities that support you in reaching your goals? 

3. (Must) What, if anything have you heard about LSAMP? (Probe for awareness of available activities or supports.)

4. (Must) Are there reasons why you have not participated or engaged in (available activities or supports)? 

 

Cognitive Interview Findings:

These questions proved a bit more challenging, as students sought to think about their general experiences versus those specific to LSAMP. Interviewees also articulated the differences between future planning and current stage in that progress. Suggestions include:

1. Ask about general future plans, rather than specifics.

2. Contextualize questions to ensure they clearly capture whether a respondent is differentiating between experiences, and allowing interrelated responses.

Resulting suggestion: Modify.

 Section E: Future Outlook 

1. (Can) Where do you see yourself in 1, 5, 10, or 15 years? 

2. (Can) What LSAMP-specific supports are you getting that will help you attain the goals you just described? 

3. (Must) (If non-doctorate holders/students) Do you see yourself enrolling in any educational programs in the future? 

4. (Must) What advice would you give current decision makers developing funding and programming for the next generations of students like yourself? 

Cognitive Interview Findings:

Similar to Section D, interviewees also articulated the differences between future planning and current stage in that progress—asking about general future plans may be something easier for them to process and report. Students are also eager to provide advice to the next group (e.g., apply earlier). Suggestions include:

1. Ask about part of their future outlook as how they would do things differently if they could, and how they would both plan their futures and advise others.

2. Provide the opportunity to be able to provide feedback on how connected they feel to others in the LSMAP community.

Resulting suggestion: Modify.

Summary Notes: Student Focus Groups

Overall, students were very eager to provide feedback on their experiences and reported that those experiences were very positive overall. They were able to articulate the ways in which their involvement in the program benefitted them, and the ways in which they receive support.

Students did not report concerns around sharing such information in group setting (e.g., the focus group), however, they did identify some questions as more personal or challenging. We may be presented with a concern for significant social desirability bias in the focus group setting, specifically in a small campus setting. Future modifications and planning to the research design in this area should take such concerns into account. Specific modifications for the focus group guide are presented in Appendix 2.

SECTION 3: SURVEY – COGNITIVE FINDINGS

Given the time constraints of the testing, and priorities for the interviews and focus groups, the survey was tested but with minimal probing. Core feedback included a desire to have access to this information from across all alliances and institutions, an opportunity to think about different degrees and programs, and differentiate between experiences of staff at different levels.

While annual reporting and other tools help share some information, the opportunity for a survey, which could include feedback from far more people in an institution, would help supplement some of the interview questions that specific individuals cannot answer (e.g., they were not there at the start of an alliance). Additionally, a census (surveying all institutions) approach to a short survey would provide LSAMP faculty, staff, and administrators a unique opportunity to both share their experiences and hear those of others—something we learned in this testing effort was highly desired. Suggested topics for inclusion, presented as modified draft questions, are presented in Appendix 3. Should a survey be fielded in the future, exact questions and response options should be derived from the qualitative findings of the case studies and cognitively tested before employment in data collection efforts.

Appendix 1: LSAMP – STAFF & ADMINISTRATOR INTERVIEW PROTOCOL

Section A: Introduction

1. We are here today to speak about the LSAMP program. Do you know the program by this name? [If no: Before we begin, can you tell me what name you use to refer to your LSAMP alliance [official application title of grant]]?

2. Can you tell me a bit about your involvement with LSAMP?

3. Does any of your other work shape the work you do on LSAMP?

4. What types of students does [Alliance] cater to?

Section B: Description and Implementation of LSAMP Activities

B.1 Activities

1. Given the list of LSAMP activities you just described, what do you think are the two that you do most regularly? Are there any that are on the newer side?

2. For the activities you just mentioned, what do you think is working well and what is challenging?

3. How are these LSAMP-related activities conducted?

4. [If not mentioned] Tell us what professional development for students looks like for at your institution.

B.2 Organizational structure of LSAMP

1. How are students in LSAMP organized? In cohorts? Individually?

2. How are faculty involved in LSAMP?

3. How involved in LSAMP is your administration? For example, what is the president or provost’s involvement if they engage at all?

B.3 How within-alliance works get done

1. What types of LSAMP-related institution-level meetings or institution-based engagements to you take part in?

2. (Ask only of PI) How were the other institutions in your alliance selected?

3. Outside of the pandemic changing everything, thinking about the meetings and engagement we’ve just discussed, have any of these engagements changed in the last few years?

4. What about with NSF? Has your engagement with NSF changed over time?

B.4 Institutionalization

1. (Ask of PI) Are there particular positions at your institution dedicated to LSAMP-related work, and if so, how do they work?

2. Are there any on-campus advocates for the program? Who are they? How do they champion the work?

B.5 Student recruitment/retention

3. What does your student recruitment process look like?

4. What do you do to support student retention?

B.6 Funding

1. How are LSAMP funds distributed and used on your campus?

2. Are you supported financially by the LSAMP grant in any way?

3. Do you know if your institution contributes financial support for LSAMP?

Section C: Contextual Understanding of LSAMP goals

1. Do you know what your institution’s motivation was for creating an LSAMP program? If so, what was the motivation?

2. How does your LSAMP program enhance access to STEM for all students?

3. What kinds of additional supports do you feel the students you work with need overall?

4. How does your LSAMP program work with other institutional initiatives?

Section D: Impact of LSAMP Activities

1. What are some achievements you have had in LSAMP?

2. What have been some of the challenges implementing LSAMP at [institution]? How have you addressed them?

3. In your role, how have you engaged with different departments on campus?

4. How do you obtain feedback on your LSAMP programming?

5. What additional supports do you need to fully support LSAMP students if any?

Section E: Influence of the Alliance

1. How do you engage with [Alliance] members at other alliance institutions, if at all?

Section F: Wrap-up

1. What do you wish NSF knew about your LSAMP program that we have not yet asked about?

Appendix 2: LSAMP CASE STUDY PROTOCOL: STUDENT FOCUS GROUP

Section A: Opening Discussion

1. What is your name, your program of study, and your year in school?

2. Today we’re talking about your participation in the LSAMP alliance [formal name]. [Do you know of it by this name? If no: You may know it is a BD/B2B/ or by some other name. What do you call it?]

3. We'd like to start by asking you about how you came to be a part of LSAMP. What was that process like for you? How did you learn about it?

4. How has your LSAMP program communicated with you about different activities, events, or provided you with information?

5. What types of LSAMP activities or resources have you participated in?

6. Can you describe the LSAMP program at [institution]?

7. How connected do you feel to the LSAMP community?

Section B: Successes

1. What have you found to be most supportive from a perspective of staff or support from an individual?

2. For your journey with LSAMP, can you pinpoint events that encouraged you to keep going? What were they? How did they help you?

3. Which activities have you participated in that were particularly helpful?

4. In what ways would you say LSAMP has supported you in your time at [institution]? Probe for specific examples.

5. How do you use the financial support you receive through LSAMP? How did you decide how to use your financial support?

6. (Level 1) What was the process like for you to obtain your stipend? Can you describe it?

7. (Level 2) Of the LSAMP-related activities you’ve attended or participated in, what have you found to be the most helpful and why?

8. (Level 2) Why did you decide to attend these LSAMP-related activities? What were you hoping to learn or gain?

Section C: Gaps/Opportunities

1. When you think about your LSAMP program, what is not working or what is missing?

2. What activities do you wish were being offered?

3. (Level 1) Is there anything (e.g., supports, guidance, activities) you feel is missing from the opportunities available to you or should be added through your LSAMP funds?

4. (Level 2) Is there anything (e.g., supports, guidance, activities) you feel is missing from the LSAMP-related activities that are available to you? What would change for you if these supports, guidance, activities, etc. were offered?

5. (Freshman/Sophomores) How is the LSAMP program different than what you imagined it would be?

Section D: Level 3 Students Only

1. What are your goals for your time here?

2. What, if anything, did you know about LSAMP before participating?

3. What additional supports would have helped increase or enhance your participation?

4. Are there reasons why you have not participated or engaged in (available activities or supports) offered by LSAMP?

Section E: Future Outlook

1. Where do you see yourself in the future? What you would like to be doing in the next one to ten years?

2. What support from LSAMP would help you attain the goals you just mentioned?

3. Do you plan to continue your education? For example, by attending graduate school (in any field)?

4. What advice would you give your advisors and LSAMP staff to improve the program?

5. What do you think needs to occur to engage students not enrolled in LSAMP?

Section F: Step 3:  Wrap-Up

1. Is there anything that we didn’t ask about that you would like your LSAMP leaders to know about?

Appendix 3: LSAMP CASE STUDY PROTOCOL: SURVEY QUESTIONS & TOPICS

1. When thinking about programming and goals for ensuring all students at your institution can access resources and ensure comparable outcomes, which phrases does your institution use either in writing or colloquially? 
   
   Please select all that apply.

• Equity

• Equal access and outcomes

• Removing barriers

• Opportunity for all

• Something else (please specify) __________________________________________________

2. What are the processes in place at the [institutional, college, department] level for enacting these [equity term] frameworks?

________________________________________________________________

3. What specific programming do you offer to ensure [equity term] goals are met?

________________________________________________________________

4. What are the opportunities for expanding the [equity term] framework within your institution in the next five years?

________________________________________________________________

5. What new degree programs, if any, have you offered in the past five years?

________________________________________________________________

6. What new degree programs, if any, do you anticipate offering in the next five years?

________________________________________________________________

7. Which of the following activities, programs, or actions occurred during the 2023-2024 academic year?

Please select all that apply.

• Encourage students to seek out mentors with shared values and identities who are committed to the mentoring

• Implement policy that fosters the connection to service and communal responsibility afforded by STEM careers as a means of attracting more students from diverse backgrounds

• Foster collaboration that between community colleges and four-year institutions that advocates for greater articulation in state-level transfer policies

• Strengthen formal channels for providing students with information about cross-enrollment policies to decrease disparities

• Include accessible mapping of all programs, exact course alignments and time-to-degree completion for future employment and transfer opportunities

• Provide targeted supports including tutoring, cohort classes, tailored counseling and timely alerts for not meeting requirements

• Gather feedback from students on a regular basis

8. Which of the following activities, programs, or actions occurred during the 2023-2024 academic year?

Please select all that apply.

• Instructors intentionally build community within classrooms by providing culturally relevant curriculum

• Establish more cohesive classroom and instructional practices between two- and four-year institutions

• Support underrepresented students’ STEM identity development

• Support underrepresented students’ STEM identity development in ways designed to offset the adjustment to fitting the “mold” of a traditional STEM student.

• Provide summer research experiences to increase the likelihood of students from underrepresented racial/ethnic minority groups applying to STEM PhD programs

• Provide students with support to manage their coursework

• Provide students with financial support

• Provide advising to support alternative pathways for non-STEM undergraduate majors to matriculate into STEM PhD programs.

• Support development of student’s scientific identities through mentoring relationships

• Encourage students to have multiple mentors as a means of increasing support with navigating STEM pathways

9. Which of the following activities, programs, or actions occurred during the 2023-2024 academic year?
   
   Please select all that apply.
   
   

• Providing hands-on research experience in lab facilities

• Completing math coursework early in community college

• Completing a higher number of overall STEM courses

• Understand pathways using an intersectional lens

• Support pre-enrollment (Align student academic readiness in high school more closely with enrollment in early college course to improve retention on STEM pathways)

• Target academic major planning to occur during second semester of second year or first semester of third year

• Provide 1-1 or small group advising support

• Differentiate advising support based on students’ expressed desire to switch majors or leave STEM major altogether

• Provide opportunities for students to learn about careers and earning potential

• Facilitate a diverse and inclusive curriculum that actively acknowledges and appreciates students’ backgrounds

10. What other activities, programs, or actions occurred during the 2023-2024 academic year, that we have not yet asked about?

________________________________________________________________

11. In your own words, please share anything about this activity that you think is a 'lesson learned', either for your organization or that may be helpful to similar organizations or future programs. 

________________________________________________________________

12. Thinking back to all the activities and programming you've offered for the past five years, how often did you hire or assign a designated full-time staff member to oversee them?

• Never

• Sometimes

• About half the time

• Most of the time

• Always

13. What other staff models did you use to facilitate these activities and programs?

________________________________________________________________

14. Who are the primary contributors to curriculum development?

________________________________________________________________

15. How does your program interact with community colleges?

________________________________________________________________

16. How does this community college engagement compare to your institution as a whole?

________________________________________________________________

17. How does your program interact with other similar four-year programs?

________________________________________________________________

18. How does your institution support LSAMP activities?

________________________________________________________________

19. What supports would you like to see in the future to benefit LSAMP programming and activities?

________________________________________________________________

20. How does your institution view the value of LSAMP programming for [the institution as a whole, LSAMP students, faculty]?

________________________________________________________________

21. Do you feel that services and supports are integrated at your institution, or more department/programming based?

• Institution based

• Department/programming based

22. How does your institution overcome barriers to effective LSAMP offerings?

________________________________________________________________

23. How much collaboration does your department have across the institutions?

• None at all

• A little

• A moderate amount

• A lot

• A great deal

24. How much collaboration does your department have with organizations outside the institution?

• None at all

• A little

• A moderate amount

• A lot

• A great deal

25. How effectively does your institution cultivate a 'STEM identity' in its students?

• Not effective at all

• Slightly effective

• Moderately effective

• Very effective

• Extremely effective

26. What faculty development opportunities are most important in your work?
    
    Please select all that apply.
    
    

• Workshops on teaching

• Professional development

• Research programs

• Diversity training

• Sensitivity training

• Equity training

27. What faculty development opportunities were implemented during the first five years of the program?
    
    Please select all that apply.
    
    

• Workshops on teaching

• Professional development

• Research programs

• Diversity training

• Sensitivity training

• Equity training

28. What faculty development opportunities are currently offered?
    
    Please select all that apply.
    
    

• Workshops on teaching

• Professional development

• Research programs

• Diversity training

• Sensitivity training

• Equity training

29. What faculty development opportunities do you anticipate offering in the next five years?
    
    Please select all that apply.
    
    

• Workshops on teaching

• Professional development

• Research programs

• Diversity training

• Sensitivity training

• Equity training

30. What hiring and staff retention experiences do you expect to have in the next five years?

• Much lower than expected

• Slightly lower than expected

• About the same as expected

• Slightly higher than expected

• Much higher than expected

31. What have your hiring and staff retention experiences been in the past five years?

• Much lower than expected

• Slightly lower than expected

• About the same as expected

• Slightly higher than expected

• Much higher than expected

32. Which attributes of your program have been enacted that were NOT initially conceptualized but emerged during the program?

________________________________________________________________

33. Thinking about your alliance work today, which words would best describe areas of focus (using words and approaches at that time)?

________________________________________________________________

34. Thinking back to your alliance work in 2015, which words would best describe areas of focus (using words and approaches at that time)?

________________________________________________________________

35. Thinking back to your alliance work in 2005, which words would best describe areas of focus (using words and approaches at that time)?

________________________________________________________________

36. How much collaboration is there with other awardees and your department?

• None at all

• A little

• A moderate amount

• A lot

• A great deal

37. Do you engage in any data sharing or information sharing with other awardees?

• Yes

• No

• Unsure

38. How do you engage in community building, leadership opportunities, and lessons learned sharing with other awardees?

________________________________________________________________

39. What opportunities for collaboration across the alliance(s) do you regularly engage in?

________________________________________________________________

40. What are the successes of such collaboration across alliances?

________________________________________________________________

41. What are the obstacles of such collaborations across alliances?

________________________________________________________________

42. How frequently, and for what purpose, do you have intra-institution meetings and collaborations?

________________________________________________________________

1 All students who participated in the interviews received funding through their participation in the program.

2 This is because it was determined prior to testing that a survey would not be included in this current study. As such, any questions and topics developed were included in testing, but with a focus to identify topics and approaches for consideration in a future study rather than producing a formal finalized instrument.

3 With one exception, where the student focus group and survey were the only two items tested, to ensure adequate testing of all instruments.

Page 30 of 30

Findings: Cognitive Testing August 8, 2024