MFHP Study Overview

APPENDIX A

Test Plan Overview for MFHP Usability Study

Table of Contents

Overview ………………………………………………………………………………………………………………………. 3

Introduction ………………………………………………………………………………….…………...……… 3

Study Objectives ………………………………………………………………………………………………... 3

Study Design …………………………………………………………………………………………………………………. 3

Methodology ……………………………………………………………………………………………..………. 4

Test Environment and Equipment ……………………………………………………………………… 4

Evaluation Measures ………………………………………………………………………………………..... 5

Overview

Introduction

As part of the overall evaluation plan for the Division of Cancer Prevention and Control (DCPC) My Family Health Portrait: Cancer (MFHP: Cancer) mobile application, a diary study will be conducted, as part of an overall usability study, to assess application effectiveness in assessing and relaying personal cancer risk to users, and overall usability of the application.

The purpose of this study is to gather feedback on the current DCPC MFHP: Cancer mobile application and gather information about the utilization of currently offered features and functionalities, user experience with the process and results, and identify any areas for improvement.

Study Objectives

• Collect qualitative and quantitative feedback on the current application

• Identify if the application is fulfilling its purpose and goals and answer the following:

  • Are users easily able to collect and input personal and family health history into the application?

  • Is the information shared on the application regarding personal risk of cancer clear and understandable to users? Do users understand the factors affecting their result?

  • Once a user uses the application to assess personal risk of cancer, is the information shared on the application regarding next steps clear and understandable to users?

• Determine how well the application design and functionality matches user mental models and user expectations

• Identify areas of improvement where applicable based on user feedback

Test design

The study will be comprised of a structured, primarily unmoderated diary study, meaning participants will be provided with prompts and specific survey questions throughout the course of the study.

A detailed screener will be provided for recruitment of participants within the target user group for the MFHP: Cancer mobile app. The screener will be comprised of a series of demographic questions, as well as technology-related questions to ensure participants have access to necessary devices in order to participate.

Throughout the study, participants will be assigned a series of tasks to complete as well as short surveys reflecting on the task-completion and usability experience. Both qualitative and quantitative feedback will be collected via survey, as well as during the concluding interview at the end of the study.

Methodology

Below is a high-level outline process summary of the diary study:

• Planning and Preparation:

  • The Peraton and CDC teams will collaborate to write and approve test plan and study materials such as: screener questions, script, and surveys.

  • OMB submission and approval of all related study materials

• Recruitment:

  • Following collaboration with the CDC team on participant requirements and screener questions, the Peraton team will work with the recruitment firm to recruit 20 participants for the diary study.

  • Recruitment firm collects participant consent during recruitment, with a collaborative effort from both the Peraton and CDC teams to ensure a clear and appropriate consent and disclaimer.

• Pre-study Meeting:

  • The study moderator will meet with each participant to review details of the study. This includes:

    • Overview of the study and application

    • Setting expectations

    • Explaining materials and instructions (tasks, surveys etc.)

    • Answering any questions participants may have

• Logging Period:

  • Participants will be sent email reminders weekly detailing the weekly/biweekly task assignments as well as the survey link. Moderator(s) will review and analyze user survey feedback on a weekly basis.

• Post-study Interview:

  • Participants answer the post-study questions and debrief with the moderators at the end of the logging period.

• Data Analysis:

  • Following completion of all sessions, moderator(s) will compile and analyze results.

Test Environment and Equipment

The pre-study brief and post-study interview sessions will be conducted via Zoom for Government. Communications will be sent to participants via email. Surveys will be created using Optimal Workshop. To participate in this study, participants will need a mobile device on which they can download and use the MFHP: Cancer application, as well as internet access, and a device through which they can connect to the two required Zoom check-ins.

Evaluation Measures

Data will be recorded by moderator(s) during pre-study and post-study in-person sessions in the form of real-time notes, as well as via surveys throughout the duration of the study.

This study will utilize qualitative and quantitative feedback from participants which the moderator will review and analyze to discover patterns and insights. Data collection will address:

• Clarity of application instructions and functionality

• Ease of personal and family history data collection and input

• Clarity of information conveyed regarding personal risk of cancer and risk factors

• Usefulness and effectiveness of current features and functions regarding next steps (after personal risk of cancer is assessed)

• Areas of improvement for the application moving forward

Qualitative data related to ease of use, clarity of instruction, and effectiveness of application functionality will also be collected throughout the study. Collection of all data will be analyzed and compiled in a summary report, detailing implications and recommendations based on study results.