Form 0920-1050 CDC USABILITY AND DIGITAL CONTENT TESTING Content Feedba

Attachment E

OMB#: 0920-1050

Exp. Date: 6/30/2025

CDC Usability and Digital Content Testing

Content Feedback, A/B Comparison, Design Feedback, and Data Visualization Usability Tests

Introduction, Questions, & Screenshots

The following pages include the usability test questions and some example screenshots from the UserTesting web-based tools that will be used to conduct the unmoderated usability tests.

Please note: CDC will conduct just one of the following usability tests at a time. Up to 18 questions would be selected per test to stay within the 20-minute time limit.

Introduction for all Tests

For each usability test, participants will see an introduction screen with instructions, OMB Fast Track number, expiration date, and burden statement, along with a “Next” button to start.

Welcome!

Thank you for agreeing to participate today. Your opinion is valuable to us!

During the session, you will be asked to give your honest opinions. Your feedback will help us make improvements to information and materials provided on CDC.gov website or CDC social media platforms.

The session will take about 20 minutes. Please remember to think out loud as you complete the tasks.

To get started, please click “Continue”. Thank you for your time.

Burden Statement

Public reporting burden of this collection of information is estimated to average 20 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to CDC/ATSDR Reports Clearance Officer; 1600 Clifton Road NE, MS H21-8, Atlanta, Georgia 30333; ATTN: PRA (0920-1050).

Test Type 1: Content Feedback Usability Test

Background Questions

1. Have you visited the CDC website in the past year for information on [disease, condition]? [Multiple choice: Yes, No, Not sure or Can't remember]

Example Screenshot from tool taken on computer desktop. Asking for background information before showing content.

2. Please rate your level of agreement with the following statement: “I trust the CDC or Centers for Disease Control and Prevention as a source of health information.” Please explain your rating out loud. Rating scale [Strongly disagree – Strongly agree; 5-point scale].

3. What information on [topic, condition] is most important to you? Why?

4. Briefly tell us in 30 seconds or less what you do in [public health; healthcare]?

First Impressions/Attitudes Questions

5. How would you sum up in just a few words your first impression of this message? Do you like it? Not like it? What makes you say that?

Example Screenshot from tool taken on mobile phone

6. If you saw or heard this [message, tagline, webpage], would it get your attention? Why or why not?

Example screenshot from tool taken on computer desktop

PLEASE NOTE: The remaining questions will use the same structure as those above – question shown with content.

7. What would you read or click on first? Why?

8. What are your reactions to how this [webpage, content, design element] page is laid out or organized?

9. Does this message appeal to you? Why or why not?

10. What feelings do you have in reaction to this message? Anything positive? Anything negative?

11. Do you strongly agree or disagree with any part of this message? If so, what?

Content Clarity/Understanding Questions

12. What is the main idea that this message is trying to get across, in your own words?

13. Please rate your level of agreement with the following statement: "The content is easy to understand." Please explain your rating. [Rating scale Strongly disagree – Strongly agree; 5-point scale].

14. What, if anything, is confusing, unclear, or hard to understand when you see this [webpage, message, content]? Please explain. What suggestions do you have to improve this?

15. How could we improve this overall [webpage, menu, content, etc.]?

Content Actionable/Use/Behavioral Intent Questions

16. How useful would this information be for you? [Rating scale: Not useful –Very useful; 5-point scale].

17. Thinking about the next 6 months, how likely are you to look for more information on how you do these recommended actions? Please explain the reason for your rating. [Rating scale Not at all likely—Very likely; 5-point scale]

18. Does this message motivate you to take action? Why or why not?

19. Is there anything that could be changed to make it more likely you would be motivated to take action?

20. Please tell me how much you agree or disagree with this statement: "By doing the behaviors suggested by the message, I can [reduce the risk of/effects of] this [disease/condition]." Please explain the reason for your rating. [Rating scale Strongly disagree – Strongly agree; 5-point scale]

21. What additional information would you need in order to more strongly believe this message?

22. On a scale of 1-5, how likely are you to take at least one of these recommended actions to protect yourself and others? Please explain your rating [Rating scale Not at all likely—Very likely; 5-point scale]

23. What if anything would be difficult for you to do? Why or why not?

24. Is there any more information or resources that you would need that would help you do some of these strategies?

25. After reading everything, what questions do you still have?

26. What additional feedback do you have?

Test Type 2: Design Feedback Usability Test

First Impressions Questions

1. What item on this [webpage, graphic, image] would you read first and why? Please explain your reason out loud.

Example screenshot from tool taken on computer desktop

2. Are there any links on this page you would click on to learn more information? Why or why not?

Example screenshot from tool taken on computer desktop

PLEASE NOTE: The remaining questions will use the same structure as those above – question shown with content.

3. What are your initial reactions to this [page]?

4. Are there any particular [elements] that stood out to you? What about those [elements] stood out?

5. What are your impressions of the [webpage, video, table, image, picture, infographic]?

6. What are your reactions to how this page is laid out or organized?

7. What is the main message or insight you get from this [webpage, graphic, image, video]?

8. What, if anything, is confusing, unclear, or hard to understand when you see this [webpage, graphic, image, video]? Please explain.

Design Elements Questions

9. What did you think about the design of this [component – e.g., webpage, graphic, image, video, etc.]?

10. What do you like best about this [component]?

11. Is there anything you think is missing from this [component]?

12. If you had to give this [component] a grade, like in school, where "A" is the best and "F" is a failing grade, what grade would you give this? Please choose your grade and explain your reason for the grade.

13. On a scale of 1 to 10, how satisfied are you with the product’s design? [10-point Rating scale: Not Very Satisfied to Very Satisfied]

14. What could improve the overall look and feel of this [webpage, graphic, image, content, etc.]?

15. How could we improve this [webpage, graphic, image, content, etc.] to make it more effective, clear, or engaging?

Navigation/Findability Questions

16. You would like to learn more about [health topic] and landed on this page. Where on the [page/site] would you expect to find [health information]? Please explain.

17. On this [page/site] where would you look for [health information/elements]? (Click/tap on the page where you would go.)

18. Why did you choose that [option, section, label, grouping]?

19. Was there anything you struggled with [today when looking for health topic]? Please explain.

20. How difficult or easy was this [question/task] for you? [5-point scale: Very difficult to Very easy]

21. Thinking about what [you are most likely to use/is most important to you/is most helpful to you], how would you rank these [areas/sections] of the page (where 1 is most important)?

22. Some [items/elements] are highlighted. Did you notice the highlighted [items/elements] previously? Please explain. Which areas are most useful? Please explain.

23. Which of these headings makes sense to you based on what you just read? (Choose all that apply.) If there is 1 or more you did not select, please explain why it doesn't make sense.

24. In this task, you’ll review a list of headings and move them into the center of your screen. Place your preferred heading on the top with your least preferred heading on the bottom. Please talk aloud to explain your answer choices.

Quantitative Assessment of Design Satisfaction Questions

25. How would you rate the [visual appeal/layout/trustworthiness/content/organization/usefulness of images and/or infographics/your satisfaction] of this [page/site]? [5-point Rating scale: Very unappealing to Very appealing/Very dissatisfied to Very satisfied]

26. How easy or difficult is it to [navigate the site/understand the categories or labels/scan the pages for information]? [5-point Rating scale: Very easy to Very difficult]

27. Which of the following words describes this [page/site]? (Choose up to 3.) [Multiple choice: Attractive, Approachable, etc.]

28. How likely is it that you would recommend this [site] to a friend or colleague? [11-point Rating scale: Not at all Likely to Extremely Likely]

29. What did you like [most/least] about this [page/site]?

30. If you could change one thing about this [page/site], what would you change?

31. What additional feedback, if any, do you have?

Test Type 3: A/B Comparison Usability Test

Showing each webpage, design element, or content item separately. Participants will be shown both option A and B, but the order of which option they will be shown first will be randomized to control for response and recency bias and ordering effects.

A/B Testing Comparing Digital Content

Show Option A [Questions 1-9 specific for comparing two different types of digital content; Questions 10-21 specific for comparing two different website designs]

1. What is the main idea that this message is trying to get across, in your own words?

PLEASE NOTE: The remaining questions will use the same structure as those above – question shown with content.

2. What is your first impression of this [graphic, content, message, etc.]? What do you like about it? What don’t you like?

3. What feelings do you have in reaction to this [graphic, content, message]? Anything positive? Anything negative?

4. Is there anything confusing, unclear, or hard to understand when you read the message and look at the image(s)? What suggestions, if any, do you have to improve this?

5. Please rate your level of agreement with the following statement: "The message compels me to take action." Rating scale [Strongly disagree - Strongly agree; 5-point scale].

6. Please rate your level of agreement with the following statement: "The message is persuasive.” Rating scale [Strongly disagree - Strongly agree; 5-point scale]

7. Is there anything you think is missing from this [graphic, content, message, etc.]?

8. What could be done to improve your satisfaction with this [graphic, content, message, etc.]?

Show Option B (Questions same as Option A specific for digital content)

9. What is the main idea that this message is trying to get across, in your own words?

Example screenshot from tool taken on computer desktop

PLEASE NOTE: The remaining questions will use the same structure as those above – question shown with content.

10. What is your first impression of this [graphic, content, message, etc.]? What do you like about it? What don’t you like?

11. What feelings do you have in reaction to this [graphic, content, message]? Anything positive? Anything negative?

12. Is there anything confusing, unclear, or hard to understand when you read the message and look at the image(s)? What suggestions, if any, do you have to improve this?

13. Please rate your level of agreement with the following statement: "The message compels me to take action." Rating scale [Strongly disagree - Strongly agree; 5-point scale].

14. Please rate your level of agreement with the following statement: "The message is persuasive.” Rating scale [Strongly disagree - Strongly agree; 5-point scale]

15. Is there anything you think is missing from this [graphic, content, message, etc.]?

16. What could be done to improve your satisfaction with this [graphic, content, message, etc.]?

Comparison/Preference Between Option A and Option B

17. Which [graphic, content, message] do you prefer 1 or 2? Why do you prefer it over the other one? Please explain your reason for the preference.

Example Screenshot from tool taken on mobile phone

18. Which of these gets your attention the most? Why or why not?

19. Which is the easiest to understand, and why?

20. Which message do you think is the most [useful, effective, trustworthy] and why?

21. Which message do you think is the least [useful, effective, trustworthy] and why?

22. What is your overall reaction to the [graphics, content, messages] you saw today?

A/B Testing Comparing Website Designs

1. You would like to learn more about [health topic] and landed on this page. Where on the [page/site] would you expect to find [health information]? Please explain.

2. On this [page/site] where would you look for [health information/elements]? (Click/tap on the page where you would go.)

3. Was there anything you struggled with [today when looking for health topic]? Please explain.

4. How difficult or easy was this task for you? [5-point scale: Very difficult to Very easy]

5. What are your [general] impressions of this [page/video/table/image/picture/infographic]?

6. Are there any particular [elements] that stood out to you? What about those [elements] stood out?

7. Looking at the [page/site], which of the following words best describe the [page/site]? (Choose up to 3.) Annoying; Attractive; Approachable, etc.)

8. How would you rate the [visual appeal/layout/trustworthiness/content/organization/usefulness of images and/or infographics/your satisfaction] of this [page/site]? [5-point Rating scale: Very unappealing to Very appealing/Very dissatisfied to Very satisfied]

Example screenshot from tool taken on computer desktop

9. How easy or difficult is it to [navigate the site/understand the categories or labels/scan the pages for information]? [5-point Rating scale: Very easy to Very difficult]

10. What do you like [most/least] about the [page/site]?

11. If you could change one thing about the [page/site], what would you change?

12. Is there anything confusing about this [page/site]?

Show Option B (Questions same as Option A)

13. You would like to learn more about [health topic] and landed on this page. Where on the [page/site] would you expect to find [health information]? Please explain.

14. On this [page/site] where would you look for [health information/elements]? (Click/tap on the page where you would go.)

15. Was there anything you struggled with [today when looking for health topic]? Please explain.

16. How difficult or easy was this task for you? [5-point scale: Very difficult to Very easy]

17. What are your [general] impressions of this [page/ video/table/image/picture/infographic]?

18. Are there any particular [elements] that stood out to you? What about those [elements] stood out?

19. Looking at the [page/site], which of the following words best describe the [page/site]? (Choose up to 3.) Annoying; Attractive; Approachable, etc.)

20. How would you rate the [visual appeal/layout/trustworthiness/content/organization/usefulness of images and/or infographics/your satisfaction] of this [page/site]? [5-point Rating scale: Very unappealing to Very appealing/Very dissatisfied to Very satisfied]

21. How easy or difficult is it to [navigate the site/understand the categories or labels/scan the pages for information]? [5-point Rating scale: Very easy to Very difficult]

22. What do you like [most/least] about the [page/site]?

23. If you could change something about the [page/site], what would you change?

24. Is there any confusing about this [page/site]?

Comparison/Preference Between Option A and Option B

25. Which [website/page] do you prefer? Why do you prefer this [design/page/element/site]?

26. Which [site/page] was easier to [use/navigate]? What made this [page/site] easier to [use/navigate]?

Summary Questions

27. What is your overall reaction to the webpages you saw today?

28. What additional feedback, if any, do you have?

Test Type 4: Data Visualizations Usability Test

Used to assess graphics, maps, charts, and infographics on webpages

First Impression Questions

1. What information on [topic, condition] is most important to you? Why?

2. Take 30 seconds to review the [graphic, map, chart, webpage]. What is the first thing that catches your eye or stands out to you most? Please answer out loud.

3. What is the main message or insight you get from this [graphic, map, chart, webpage]?

4. What, if anything, do you like most about the design?

5. What aspects of the design need the most improvement?

6. Data Interpretation QuestionsLooking at the graphic, what would you say is the level of risk in [county, state, other geographic area] for [you/your family] being impacted by [public health topic]?

Example Screenshot from tool taken on mobile phone

7. If you were a [public health decision maker; healthcare decision maker] in [county, state, other geographic area], what is this graphic trying to say about the likelihood [of public health topic] risk is [increasing, decreasing, or holding steady]?

Example Screenshot from tool taken on desktop computer

8. If you were a person living in [county, state, or other geographic region], what is this graphic trying to say about [public health topic] in your community?

Functionality of Data Visualizations for User

9. On a scale from 1 to 5, was it hard or easy for you to find information on the [county, state, other geographic area’s] [public health topic]? Please explain out loud the reason for your rating. [1-Very hard 5-Very easy]

10. On this [graphic, map, chart, web component], where would you look for [health information/elements]? (Click/tap on the page where you would go.)

11. On a scale from 1 to 5, was it hard or easy for you to complete [the specific task]. Please explain out loud the reason for your rating. [1-Very hard 5-Very easy]

12. Were there any aspects of the design that you found difficult to use?

13. Did the product perform as expected in terms of functionality? Were there any technical issues that you encountered?

Summary Questions

14. Overall, what if anything, was confusing, unclear, or hard to understand you when you reviewed this [graphic, map, chart, webpage] and completed the tasks? Please explain. What suggestions do you have to improve this?

15. Rating question [5-point Rating scale: Not at all useful to Very useful]: How useful is [graphic, map, chart, webpage] to you to make decisions about [public health topic]? Please explain your rating.

16. What additional feedback, if any, do you have about how we can improve this [graphic, map, chart, webpage]?

Audience Screener Questions

Industry and profession screener questions for public health and healthcare professionals audience as not included as option for UserTesting audience filter.

• Which of the following best describes the industry you work in?

  • Agriculture (reject)

  • Construction/manufacturing (reject)

  • Education (reject)

  • Finance/economics (reject)

  • Healthcare (accept)

  • Law (reject)

  • Public health (reject)

  • Sports/athletics (reject)

  • None of the above (reject)

  • Prefer not to answer (reject)

Public health professional screener questions

• Please identify the classification that best represents your role in the public health department.

• Administrative

• Agency leadership

• Clinical and lab

• Communication and/or community outreach

• Preparedness

• Environmental health

• Epidemiology

• Violence prevention

• None of the above

• Prefer not to say

Healthcare Professional Screener questions

• What is your current profession?

  • Care coordinator

  • Dentist

  • Emergency Medical Technician

  • Healthcare system administrator (e.g., hospitals, nursing homes, outpatient clinics like dialysis, rehab facilities, in-home care providers, lab companies, etc.)

  • Healthcare system department head or leadership (e.g., hospitals, nursing homes, outpatient clinics like dialysis, rehab facilities, in-home care providers, lab companies, etc.)

  • Infection Prevention Specialist

  • Medical assistant

  • Medical student

  • Nurse practitioner

  • Registered nurse

  • Pharmacist

  • Physician assistant

  • Primary care doctor

  • Specialist doctor

  • Receptionist

  • Social worker

  • Prefer not to say

  • None of the above

• What is the primary specialty of the practice in which you work? 

• Family Medicine

• Internal Medicine

• Obstetrics/Gynecology

• Oncology

• Pathology 

• Pediatrics

• Psychiatry 

• Clinical Genetics

• Prefer not to say

• None of the above 

Race/Ethnicity (in compliance with new OMB guidance for race/ethnicity)

• What is your race and/or ethnicity? (Select all that apply)

  • American Indian or Alaska Native

For example, Navajo Nation, Blackfeet Tribe of the Blackfeet Indian Reservation of Montana, Native Village of Barrow Inupiat Traditional Government, Nome Eskimo Community, Aztec, Maya, etc.

• Asian

For example, Chinese, Asian Indian, Filipino, Vietnamese, Korean, Japanese, etc.

• Black or African American

For example, African American, Jamaican, Haitian, Nigerian, Ethiopian, Somali, etc.

• Hispanic or Latino

For example, Mexican, Puerto Rican, Salvadoran, Cuban, Dominican, Guatemalan, etc.

• Middle Eastern or North African

For example, Lebanese, Iranian, Egyptian, Syrian, Iraqi, Israeli, etc.

• Native Hawaiian or Pacific Islander

For example, Native Hawaiian, Samoan, Chamorro, Tongan, Fijian, Marshallese, etc.

• White

For example, English, German, Irish, Italian, Polish, Scottish, etc.

Gender Identity (OMB approved language)

• Are you (select all that apply):

  • Female

  • Male

  • Transgender, non-binary, or another gender

  • Prefer not to answer/Decline

Page 15 of 15