Supporting Statement for Paperwork Reduction Act Submissions
Title: Program Analysis and Evaluation (PA&E) Office, Stakeholder Engagement Division (SED) Convenings Evaluation
OMB Control Number: 1670-NEW
1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.
The Foundations for Evidence-Based Policymaking Act of 2018 (Public Law No: 115-435), or the Evidence Act, promotes the use of evidence to inform decision-making and requires federal agencies to undertake activities toward this end. Specifically, the Evidence Act requires agencies to develop Learning Agendas and Annual Evaluation Plans.
Learning Agendas are systematic plans for identifying and addressing priority questions relevant to the programs, policies, and regulations of an agency. Learning Agendas document priority research questions that will be answered using different types of evidence (e.g., program evaluation), to help understand how well or how efficiently programs are working. Specifically, OMB defines program evaluation as “an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency.” The term "systematic" indicates that an evaluation requires a structured and consistent method of collecting and analyzing information. Evaluations may also include assessment of projects, practices, or interventions within a program.
Annual Evaluation Plans describe the evaluation activities that an agency plans to undertake, including “significant” evaluations related to the agency learning agenda and any other “significant” evaluations, including those required by statute. DHS considers several criteria when designating evaluations as “significant,” including the following: supports the DHS learning agenda; aligns with leadership priorities; responds to a mandate; has potential for agency-wide impact or engagement; has potential for high financial impact; and has potential for high stakeholder impact.
The Cybersecurity and Infrastructure Security Agency’s (CISA) Learning Agenda questions are documented in the Department of Homeland Security FY 2022-2026 Learning Agenda. In addition, its evaluations are included in the Department’s Annual Evaluation Plans, indicating that the Department has recognized those evaluations as “significant.” The Stakeholder Engagement Division (SED) Convenings Evaluation is one such significant evaluation and was included in the Department of Homeland Security FY 2023 Annual Evaluation Plan. CISA’s Program Analysis & Evaluation (PA&E) Division (and any contractors, as applicable) is conducting this study.
SED Convenings Evaluation
CISA’s SED leads CISA’s national and international voluntary partnerships and engagements with critical infrastructure stakeholders while serving as the agency’s hub for the shared stakeholder information that unifies CISA’s approach to whole-of-nation operational collaboration and information sharing. CISA’s voluntary partnership model relies on constant feedback and collaboration with critical infrastructure partners. One mechanism to seek this input is through the various convening activities, including Councils, Boards, and Committees, that CISA manages through SED. These convening mechanisms provide structure and an iterative process for bringing government, industry, and academic partners together to drive whole-of-nation operational collaboration. Other products and services offered to partners include analysis, reports, guidance, trainings, and scenario-based drills developed to help the entire community do their part to raise the security baseline of critical infrastructure’s assets, systems, and networks.
Of the 16 critical infrastructure sectors, SED identified the Critical Manufacturing; Commercial Facilities; and Nuclear Reactors, Materials, and Waste (herein referred to as “Nuclear”) sectors for inclusion in this study for the following reasons:
CISA is the Sector Risk Management Agency (SRMA) for these three sectors and is therefore able to engage directly with them.
These sectors span a range of infrastructure types (high-security hardened facilities, open public facilities, product development, service providers) that face a wide range of distinct threats and issues, providing an opportunity to explore the widest possible range of evaluation approaches and factors influencing evaluation within them.
The Critical Manufacturing and Commercial Facilities sectors, specifically, were the subject of recent DHS audits from the Office of Inspector General and Government Accountability Office, who are especially focused on developing approaches to evaluate the effectiveness of the sectors in executing their protective responsibilities and CISA’s role as the SRMA.
The SED program teams supporting these sectors have productive working relationships with sector leaders and partners and the bandwidth to support this evaluation.
This evaluation will assess the extent to which CISA’s convening activities, products, and services 1) provide timely, accurate, and useful information about security and risk resilience, including opportunities for meaningful information exchange between CISA and sector stakeholders; and 2) are accessed and used by stakeholders to enhance their abilities to respond to critical threats and improve strategic decision-making and risk reduction. This study also aims to increase understanding of the best practices for getting stakeholders engaged and building trusted relationships.
2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.
This is a new information collection. Information will be collected by CISA’s PA&E (and any contractors, as applicable). The potential respondent universe for this evaluation includes individual representatives (approximately 1,000 cyber and physical security, emergency, and business continuity managers) of approximately 300 member organizations from three critical infrastructure sectors [Critical Manufacturing, Commercial Facilities, and Nuclear Reactors, Materials, and Waste (herein referred to as “Nuclear”)]. Those who have served as a representative for less than 3 months will be excluded.
Within each sector, member organizations belong to either the Sector Coordinating Councils (SCC) or Government Coordinating Councils (GCC). SCCs are self-organized and self-governed entities that represent critical infrastructure owners and operators and their respective trade or equivalent associations. GCCs are formed as the government counterpart for each SCC to enable interagency and cross-jurisdictional coordination. GCCs are comprised of federal, state, local, tribal, and territorial government entities.
Information will be collected via 10-minute online surveys with approximately 1,000 individual representatives and in-depth virtual interviews (up to 1 hour) with a subsample of up to 75 individual representatives. CISA’s SED sector chiefs will provide the study team with lists of the member organizations and the names, emails, and phone numbers (when available) of the member organization representatives.
The survey will ask questions about the representatives’ member organization (size and type); their satisfaction with CISA’s convening activities, products, and services; the types of organizational changes made as a result of CISA’s convening activities, products, and services; representatives’ suggestions for improvement of CISA’s convening activities, products, and services; and perceived quality of relationships and engagements with CISA. The interviews will ask more in-depth information about representatives’ reasons for satisfaction or dissatisfaction with CISA’s convening activities, products, and services; types of organizational changes made as a result of CISA’s convening activities, products, and services; and the quality of relationships with CISA. The survey and interview by the participants will be on a voluntary basis.
The findings from this study will inform SED, other CISA stakeholder engagement programs, and CISA at-large about how best to deliver valuable products and services in ways that are most useful for stakeholders. Specifically, findings will be used to improve the guidance CISA issues to partners that make it more meaningful, relevant, timely, and actionable; and to expand and enhance guidance distribution channels to meet partners where they are, while also directing them towards CISA-sponsored channels (e.g., the Homeland Security Information Network Critical Infrastructure).
In addition, understanding the extent to which SED convening activities offer meaningful information exchange and contribute to enhanced trust (between CISA and its stakeholders) will allow CISA to improve the planning and execution of its convening activities, as well as the adoption, use, and impact of CISA products and services through enhancements, refinements, and the development of new products and services. Findings will identify areas for improvement in how CISA collaborates and interacts with stakeholders through these convening activities, how CISA shares information through the activities, and how CISA uses the outcomes of the convening activities to prioritize the delivery of products and services that best support information exchange within and across sectors.
3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden. Please include a statement that usability testing was conducted and what the results from testing did for the collection.
The proposed data collection makes use of email to contact potential participants to recruit their participation in an online survey. The survey will be programmed in Qualtrics in order to minimize burden. These data will be supplemented by virtual interviews.
Surveys. A 10-minute online survey will be administered to representatives (e.g., cyber and physical security, emergency, and business continuity managers) from the member organizations across the Critical Manufacturing, Commercial Facilities, and Nuclear sectors. The survey will be created and sent using Qualtrics, a professional-grade survey software.
Using the email addresses of the representatives provided by the SED sector chiefs, the study team will send a link that participants can use to access and complete the survey using a tablet, smartphone, or laptop. Electronic submission will ensure the maximum response rate while also permitting respondents to complete the survey at a time of their own choosing.
The survey is designed so that each sector has a customized link with specific questions for that particular sector to account for some minor differences in the convening activities, products, and services that each sector provides. This will help ensure that the representatives of each sector are asked questions that are most relevant to them.
Interviews. The study team will also conduct a series of virtual interviews (up to 1 hour) with up to 75 participants who complete the online survey and agree to participate in the interview. The study team plans to conduct the in-depth interviews by telephone or via a web-based conference call platform, such as Microsoft Teams. This format should be less burdensome to study participants than in-person interviews since they do not have to host study team members.
Usability Testing. The study team tested the survey instruments through a pilot study conducted with a convenience sample of nine representatives from member organizations across the three sectors. The purpose of the pilot was to estimate survey length, assess respondents’ understanding of the survey questions, test usability, and identify improvements in the flow and structure of the instruments. Results indicated that the survey took 10 minutes or less to complete. Some modifications, mainly wording changes that made questions clearer and more relevant, were made to the survey based on results from the pilot test.
While the interview guide was not tested, it was reviewed by subject matter experts (SMEs). Since it is an open-ended, semi-structured guide, the trained interviewers on the study team can rephrase and/or elaborate upon questions until the respondent understands them. Interviewers are also trained to stop at the one-hour mark, so the time burden is already known.
4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.
The evaluation team is collecting information that is not available elsewhere. None of the instruments ask for information that can be reliably obtained through other sources.
5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I), describe any methods used to minimize.
Some member organizations may be small businesses. The evaluation team will only request information required for the purposes of the evaluation. The burden for respondents will be minimized by restricting the survey and interview length, by conducting interviews at times convenient for respondents, and by not requiring record-keeping or written responses on the part of the participants.
6. Describe the consequence to Federal/DHS program or policy activities if the collection of information is not conducted, or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.
Without collecting this information, CISA will not meet the requirements of the Evidence Act to conduct program evaluations—particularly, this SED evaluation, which was included in the Department of Homeland Security FY 2023 Annual Evaluation Plan as a “significant” evaluation. In addition, without collecting this information, SED, other CISA stakeholder engagement programs, and CISA-at-large will not be able to understand whether and how CISA’s convening activities, products, and services provide value and utility for stakeholders to enhance their decision-making and risk reduction. Thus, we will not have the information needed to learn how to improve the planning, execution, and delivery of the convenings, products, and services so that they are more meaningful, relevant, timely, and actionable for stakeholders. Without collecting this information, we will also not be able to assess how to best engage and build trusted relationships with stakeholders, which is needed to identify areas for improvement in how CISA collaborates and interacts with stakeholders to support information exchange within and across sectors.
7. Explain any special circumstances that would cause an information collection to be conducted in a manner:
Requiring respondents to report information to the agency more often than quarterly.
The special circumstances contained in item 7 of the Supporting Statement are not applicable to this information collection.
Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it.
The special circumstances contained in item 7 of the Supporting Statement are not applicable to this information collection.
Requiring respondents to submit more than an original and two copies of any document.
The special circumstances contained in item 7 of the Supporting Statement are not applicable to this information collection.
Requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years.
The special circumstances contained in item 7 of the Supporting Statement are not applicable to this information collection.
In connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study.
The special circumstances contained in item 7 of the Supporting Statement are not applicable to this information collection.
Requiring the use of a statistical data classification that has not been reviewed and approved by OMB.
The special circumstances contained in item 7 of the Supporting Statement are not applicable to this information collection.
That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use.
The special circumstances contained in item 7 of the Supporting Statement are not applicable to this information collection.
Requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.
The special circumstances contained in item 7 of the Supporting Statement are not applicable to this information collection.
8. Federal Register Notice:
a. Provide a copy and identify the date and page number of the publication in the Federal Register of the agency’s notice soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.
b. Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.
c. Describe consultations with representatives of those from whom information is to be obtained or those who must compile records. Consultation should occur at least once every three years, even if the collection of information activities is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.
|
Date of Publication |
Volume # |
Number # |
Page # |
Comments Addressed |
60-Day Federal Register Notice: |
9/5/2024 |
89 |
172 |
72414-72415 |
0 |
30-Day Federal Register Notice |
12/22/2025 |
90 |
243 |
59851-59852 |
0 |
A 60-day notice for comments was published in the Federal Register on 9/5/2024. No comments were received related to the 60-day notice.
A 30-day notice for comments was published in the Federal Register on 12/22/2025. 0 comments were received related to the 30-day notice.
9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.
There is no offer of monetary or material value for this information collection.
10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.
The following privacy notice, obtained from CISA’s Office of Privacy, will be displayed on the survey landing page on each survey:
Authority: Homeland Security Act § 871 (6 U.S.C. § 451), 71 FR 14930 (Mar. 24, 2006), and Executive Order 12862 “Setting Customer Service Standards” 58 FR 48255 (Sept. 11, 1993) authorize the collection of this information.
Purpose: The purpose of this survey is to assess the extent to which CISA’s convening activities, products, and services 1) provide timely, accurate, and useful information about security and risk resilience, including opportunities for meaningful information exchange between CISA and sector stakeholders; and 2) are accessed and used by stakeholders to enhance their abilities to respond to critical threats and improve strategic decision-making and risk reduction. This survey also aims to increase understanding of the best practices for getting stakeholders engaged and building trusted relationships. This survey is intended to help inform and improve CISA’s activities, products, and services.
Routine Use: This information may be disclosed as generally permitted under 5 U.S.C. §552a(b) of the Privacy Act of 1974, as amended. This includes using the information, as necessary and authorized by the routine uses published in DHS/ALL-002 - Department of Homeland Security (DHS) Mailing and Other Lists System of Records (November 25, 2008, 73 FR 71659).
Disclosure: Your answers are confidential. Your name and/or your organization’s name will not be identified in our reports. Findings will be presented in aggregate form to CISA. All data from the survey will be destroyed after the report is complete. All CISA policies on handling data will be properly followed, and only a small number of members of the Stakeholder Engagement Team will see individual responses. Furnishing this information is voluntary; however, failure to provide any of the information requested may prevent CISA from obtaining the information needed to evaluate the effectiveness of CISA’s convening activities with Stakeholder engagements.
For those who elect to participate in an interview, at the start of the interview, interviewers will read a statement that informs them that participation in the interview is voluntary; that what they say will remain confidential and identified responses will not be released beyond the study team; and no specific names, of either individuals or organizations, will be identified in reports (and if quotes from the interviews are included in reports, they will not be attached to any person’s name or organization).
11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.
There are no questions of a sensitive nature.
12. Provide estimates of the hour burden of the collection of information. The statement should:
Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desired. If the hour burden on respondents is expected to vary widely because of differences in activity, size, or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices.
CISA’s SED sector chiefs (from the Critical Manufacturing, Commercial Facilities, and Nuclear sectors) will provide the study team with lists of the member organizations and names, emails, and phone numbers (when available) of the member organization representatives. The surveys will be shared with all representatives of each member organization from the three sectors (approximately 1,000 individuals representing about 300 member organizations). Participants will be notified in the survey that they may be contacted for an in-depth interview. The sample for the in-depth interviews will be a convenience sample; the study team will follow up with respondents by email soon after their surveys are completed and attempt to schedule a time for the in-depth interview, until a total sample size of 75 interviewees is reached.
Survey and interview responses will be collected only once. The survey and interview are voluntary, and the estimated total time needed to complete the data collection activities ranges from 0.17 hours (10 minutes) for those completing just the online survey, to 1.17 hours (1 hour and 10 minutes) for those completing both the online survey and in-depth interview.
To estimate the cost of this collection, CISA uses the average wage for General and Operations Managers of $62.181 times the wage rate benefit multiplier of 1.45462 (to account for fringe benefits) equaling $90.45. At a loaded average hourly wage rate of $90.45 multiplied by the total annual burden of 65.8 hours, the annual respondent cost is $5,954.44 for the Critical Manufacturing online survey only. Combined with the Critical Manufacturing online survey and in-depth interview, as well as both instruments for Commercial Facilities and Nuclear, the total annual respondent cost is $21,858.07.
Instrument |
Number of Respondents |
Number of Responses per Respondent |
Average Burden per Response (in hours) |
Total Annual Burden (in hours) |
Loaded Average Hourly Wage Rate |
Total Annual Respondent Cost |
CMa Online survey only
CM Online survey & in-depth interview |
395
30
|
1
1 |
0.17
1.17 |
65.8
35.0
|
$90.45 |
$5,954.44
$3,165.65
|
CFa Online survey only
CF Online survey & in-depth interview |
395
30
|
1
1 |
0.17
1.17 |
65.8
35.0
|
$90.45 |
$5,954.44
$3,165.65
|
Nuclear Online survey only
Nuclear Online survey & in-depth interview |
135
15
|
1
1 |
0.17
1.17 |
22.5
17.5 |
$90.45 |
$2,035.06
$1,582.83 |
Total |
1,000 |
- |
- |
242 |
- |
$21,858.07 |
a CM= Critical Manufacturing; CF= Commercial Facilities
13. Provide an estimate of the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14.)
There are no recordkeeping, capital, start-up, or maintenance costs associated with this information collection.
14. Provide estimates of annualized cost to the Federal Government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing and support staff), and any other expense that would have been incurred without this collection of information. You may also aggregate cost estimates for Items 12, 13, and 14 in a single table.
The estimated total cost to the Federal government is a one-time cost of $327,510.00 on an annualized basis, based on the expected cost of the government contract used to design the study, create and conduct the survey and interviews, and analyze the results.
15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-I. Changes in hour burden, i.e., program changes or adjustments made to annual reporting and recordkeeping hour and cost burden. A program change is the result of deliberate Federal government action. All new collections and any subsequent revisions of existing collections (e.g., the addition or deletion of questions) are recorded as program changes. An adjustment is a change that is not the result of a deliberate Federal government action. These changes that result from new estimates or actions not controllable by the Federal government are recorded as adjustments.
This is a new information collection.
16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.
Analyses will consist of basic descriptive statistics and correlations for the quantitative data (i.e., from the online survey). Content analyses will be used to assess the findings from the qualitative data (i.e., in-depth interviews). Given that the study aims to inform and improve CISA’s convening activities, products, and services, as well as relationships and engagements with stakeholders, findings may be incorporated into documents that are made public. In these instances, individual and/or organization names will not be identified, and findings will be presented in aggregate form.
The table below shows the planned schedule for the SED Convenings Evaluation:
Activity |
Schedule |
Prepare PRA materials Internal PRA review and submission |
April 2024 April 2024 – March 2025 |
OMB approval |
April 2025 |
Data collection |
May 2025 – September 2025 |
Analyze data and prepare final evaluation report(s) |
September 2025 – October 2025 |
Internal briefings on study results |
October 2025 – November 2025 |
17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain reasons that display would be inappropriate.
CISA will display the expiration date for OMB approval of this information collection.
18. Explain each exception to the certification statement identified in Item 19 “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.
CISA does not request an exception to the certification of this information collection.
1https://www.bls.gov/oes/2023/may/oes_nat.htm#00-0000
2 The load factor is estimated by dividing Total compensation ($46.14) by Salaries and wages ($31.72) = 1.4546, based on the 2024 Q1 Bureau of Labor Statistics Employer Costs for Employee Compensation News Release, Table 2: Employer Costs for Employee Compensation for civilian workers by occupational and industry group. https://www.bls.gov/news.release/archives/ecec_06182024.htm
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |