SED Convenings Survey – Interview

OMB CONTROL NUMBER: 1670-XXXX

OMB EXPIRATION DATE: 00/00/20XX

CISA SED Evaluation In-depth Interview Guide

THIS CONSENT FORM MUST BE READ IN ITS ENTIRETY TO THE RESPONDENT BEFORE PROCEEDING WITH INTERVIEW.

Hello. Thank you so much for taking the time today to talk with us. As we mentioned when we emailed or spoke with you previously, the Cybersecurity and Infrastructure Security Agency (CISA) partnered with Team Guidehouse to conduct an evaluation to learn more about stakeholder engagements and satisfaction with CISA’s convening activities, products, and services. By convening activities, we mean CISA’s guidance and expertise, as well as councils, working groups, roundtables, and other types of collaborative activities. When we refer to products and services, we mean things like publications, emails, and webinars. While we will be asking questions about your organization’s experiences with CISA, we are also interested in your individual perspective as a member of that organization.   

 

Just as with our recent web-based survey, your participation in this interview is voluntary. We want to assure you that what you say is confidential and that identified responses will not be released beyond the study team. Also, please know that no specific names, of either individuals or organizations, will be identified in our reports.  If we include quotes in our reports, they will not be attached to a person’s name or organization. Additionally, this is not an assessment of you or your agency; it is intended to help inform CISA’s activities, products and services. You can choose not to answer any questions you do not want to answer, and you can decide not to participate at any time without consequence. 

Today’s call will last up to an hour. We may follow up with you or others afterwards if we have remaining questions after reviewing our notes. [I would like to record today’s discussion just to make sure we accurately record everything you said. No one outside the study team will have access to the recording. 

 

Do I have your permission to record the discussion? Yes/No -- IF NO: No problem, we will just take notes. INTERVIEWER: DO NOT PRESS RECORD AND JUST TAKE NOTES

  

Do I have your consent to continue?  WAIT FOR RESPONSE. IF NO: THANK RESPONDENT AND END INTERVIEW.

IF YES: Do you have any questions before we get started?

Paperwork Reduction Act

The public reporting burden to complete this information collection is estimated at 60 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and the completing and reviewing the collected information.  The collection of information is voluntary. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number and expiration date.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to DHS’s Office of the Chief Procurement Officer, Office of Acquisition Policy and Legislation, 7^th and D Street, Washington, D.C. ATTN: PRA [OMB Control No. 1601-XXXX].

Introductions/ (Who is on the call and) what is/are your title(s) and primary responsibilities at your organization?

• Please tell me about your knowledge and general experience with cyber and physical security threats at (ORGANIZATION NAME)?

• How long have you been at (ORGANIZATION NAME)? How long have you been in this position?

To begin, I’d like to ask some questions about your organization specifically.

• How long has your organization been a member of the (CM, CF, Nuclear) sector council? If you don’t know precisely, your best estimate is fine.

  • Do you know how your organization came to become a member of CISA’s (critical manufacturing/commercial facilities/nuclear) sector?

  • And was your organization ever part of another sector council previously? If so, which ones?

• Are you the point of contact with CISA for your organization?

  • IF YES: How long have you been a point of contact with CISA for your organization?

Now we’re going to talk about CISA’s Convening Activities, Products and Services.

• First, let’s talk about some of the methods of collaboration with CISA, or convening activities, as well as products and services they provide. In the survey, you described (several) convening activities/product(s)/service(s) as “extremely or very helpful”. Specifically, [LIST FROM SURVEY].

  • FOR EACH SECTOR-SPECIFIC CONVENING/PRODUCT/SERVICE/: How have/has [CONVENING/PRODUCT/SERVICE] been beneficial to your organization?

    • Has your organization made any specific decisions or taken any specific action to improve risk reduction as a result of this [convening/product/service]? IF YES: Can you tell me about that?

    • Is there anything you would change or anything you believe would improve the process or structure of [CONVENING/ PRODUCT/SERVICE]?

• Which of these convenings, products or services has been most helpful and why?

• IF APPLICABLE: Okay, Let’s move on to something that you may not have found as helpful. In the survey, you described (CONVENING/PRODUCT/SERVICE) as “Not at all helpful” (if no entries for not at all helpful, ask about the lowest rated service). Why do you feel that this service or product is not helpful?

  • How can it be changed to make it more beneficial to your organization?

  • Have you talked to your CISA contacts about making this change?

    • IF YES: How did they respond?

    • IF NO: Why did you not talk about it with your CISA contacts? Do you think they would be open to this feedback?

IF THERE’S TIME AND MORE THAN ONE PRODUCT/SERVICE IS POORLY RATED, REPEAT QUESTIONS

• IF ANY ITEMS ARE MARKED AS ‘NOT USED’: In the survey, you entered ‘not used’ for (CONVENING(S)/PRODUCT(S)/SERVICE(S)). Why have you not used (this/these) (convenings/products/services)?

  • Were you aware of them?

  • What would help you to use them?

  • If you wanted to use them, would you know how to access them?

• Have you attended a CIPAC meeting? These are defined as only those meetings that will result in and/or are intended to seek consensus advice or recommendations.

  • IF YES: Which meeting(s) did you attend and did you find them to be helpful or valuable? Why or why not?

• Are there any convenings, products, or services not currently provided by CISA that your organization would find helpful?

  • IF YES: Why would that [convening/product/service] be helpful?

  • Have you made this request to your CISA contacts?

    • IF YES: How did CISA responded to the request?

    • IF NO: Why have you not made this request to CISA? How do you think they would respond?

• Were there any other convenings, products, or services provided by CISA that your organization used that we have not yet talked about?

  • How helpful did you find [convening/product/service]?

  • Did having this [convening/product/service] result in your organization making a decision or taking an action to improve risk reduction at you organization? IF YES: How?

  • What material did this [convening/training/product] cover?

• Now let’s talk about how your organization has improved risk reduction and how that relates to each convening, product, or service we’ve discussed.

  • How do you measure or assess risk?

  • What would you say are the most common changes implemented since working with CISA? What specifically prompted that change? [PROBE FOR SPECIFIC DETAIL ABOUT CHANGES, IMPROVEMENTS, ETC. For example, updated policies/plans, updated or new emergency action plan, new positions/functions, new security measures/processes, new training courses or programs]

  • What challenges has your organization faced, if any, in making your organization more secure?

    • On the other hand, what kinds of things, if any, have helped make your organization more secure?

  • Is there more that your organization needs to do to improve risk reduction?

  • Is there anything else CISA can do to help improve risk reduction in your organization?

Stakeholder Relationship

Now let’s talk about the relationship between your organization and CISA. We know there may be others at your organization who engage with CISA, but we would like to know your own perspective of how the organizations relates.

• How would you describe your organization’s current relationship with the CISA (CF/CM/Nuclear) sector team?

• [IF ANY ITEM AT C1 OR C2 IS DISAGREE OR STRONGLY DISAGREE:] In the survey, you [disagreed/strongly disagreed] with the statement(s) [FILL STATEMENT]. Can you tell me more about why you [disagree/strongly disagree] with that/those statement(s)?

• Can you think of anything that CISA could do to improve their relationship with your organization?

  • Is there anything about your organization’s relationship with CISA that you really appreciate?

  • Do you think your individual relationship with the CISA (CF/CM/Nuclear) sector team may be different than from your organization’s perspective? IF YES: How is it different?

• How do you feel about the amount of engagement with the CISA (CF/CM/Nuclear) sector? Probe if necessary Do you feel like it is too much, too little, or just right?

• Do the convenings, products, and/or services affect your relationship with CISA? If so, how? [If needed: For example, do the convenings, products, and/or services help with building and sustaining a trusted relationship with CISA? Or do they not have any effect on the relationship?]

• Do you feel that CISA understands and is responsive to your needs as an organization? How so?

  • Do you ever give feedback to your contacts at CISA?

  • How do you feel your contacts at CISA would receive feedback from you and your organization?

• How has being a member of CISA’s [CM/CF/Nuclear] Sector impacted your organization’s reputation and relationships with your clients?

• Overall, do you consider CISA to be a reliable and trusted source?

  • IF YES: Why?

    • How do you know that you can trust the information and resources that are provided by CISA?

• IF NO: Why not?

  • What can CISA do to become a reliable and trusted source?

Thank you so much for all your feedback so far. It has been very helpful. I just have one final question.

• If you could make one recommendation to CISA [Nuclear, CM, CF], what would it be?

Thank you again!