Encourage continuous learning throughout implementation

The field of ePRO use continues to mature, and like most health IT tools, intersect with many sociotechnical layers of healthcare delivery. The launch of an ePRO tool should be considered the start of a continuous effort to observe, evaluate, adapt, and improve all aspects of ePRO tool design and implementation.

We recommend engaging with the QI (quality improvement) resources in your healthcare organization or utilizing the QI methodology referenced in this section’s Overview throughout the entire integration process of implementation planning, launch, and ongoing monitoring. We also recommend establishing structures that facilitate and disseminate continuous learning across the organization around ePRO technology advancements, implementation best practices, and continued alignment with care transformation.

Strategy A
Engage in routine implementation monitoring to improve ePRO workflows and tools over time

As part of the implementation process, it is important to monitor and iteratively evaluate implementation progress. This may occur via activities such as real-time reporting and stakeholder feedback. The initial launch phase of an ePRO tool should maintain engagement of the full multidisciplinary project team, including dedicated time from IT and/or informatics team members, as well as time for iterative cycles of reflection and refinement of functionality or workflows (e.g., Plan-Do-Study-Act cycles).

Sample implementation monitoring metrics

  • % of eligible patients who receive ePRO notification
  • % of eligible patients who submit ePRO responses
  • % of missing data (i.e. incomplete ePRO responses)
  • % of ePRO responses appropriately documented by clinical team

Project teams should identify key implementation process and outcome metrics that can support ongoing assessment of implementation progress, including feasibility, fidelity, acceptability, sustainability, adoption, and cost. The selection of implementation process and outcome metrics should align with your health system’s internal goals or initiatives (see Governance Activities guideline, Strategy C), and can be guided by formal evaluation frameworks (see Integration section Overview). Project teams should also ensure that clinical champions and key staff roles have the ability to easily access and view data reports in real time throughout implementation.

While quantitative data will provide critical insight into implementation progress, feedback from users (patients, healthcare teams) through formal or informal routes (e.g., interviews, conversations, observing practice) can better contextualize issues with workflow or tool alignment and inform adaptations that improve ePRO use.

The Tools and Resources section provides a sample ePRO implementation monitoring plan for reference.

Strategy B
Identify site-level leadership to monitor changes to PRO technology over time

In alignment with implementation and change management process, identifying a champion who can monitor changes can help sustain ePROs tool use over time. ePRO tools are often part of a larger technology ecosystem within a healthcare setting, and changes to other technologies can affect how well ePRO tools function over time. For example, changes to patient portal functionality could impact how patients access ePRO questionnaires. Changes to clinical order sets might affect how ePRO data is cataloged and stored within the electronic data warehouse. The maintenance of ePRO tools beyond their initial launch should reside with the clinical teams that use them most; that way, the status of ePRO tools are taken into account as broader changes are made to tools that impact care delivery. In parallel, though, there should be strong communication pathways between the clinical teams managing ePRO implementations and the PROs governance teams that can enable broader support for communication, standardization, and resource allocation as ePRO use scales beyond single settings.

Beyond maintenance of established ePRO tools, ePRO technology itself is advancing. As project teams design ePRO tools, they may encounter functionality needs that are out of scope or not fully developed. ePRO project teams, in partnership with PROs governance teams, as well as with teams that govern other aspects of health system technologies (patient portal, data and analytics, etc.), may develop a roadmap that documents future ePRO functionality needs and that ensures project teams are alerted to advances in ePRO technologies (e.g., the release of new functionality by the ePRO vendor).

Strategy C
Ensure learnings from ePRO implementation and use are communicated throughout the organization

Throughout implementation monitoring, project teams should report learnings back to PROs governance teams, healthcare leadership (administrative and clinical), and IT teams to support continuous learnings for future implementations, particularly around best practices for standardization of workflows and tools that can support health system efficiencies for ePRO use.

ePRO implementation efforts require support across different levels of the organization, from clinical and operational leadership to frontline staff to patient and family advocacy teams. Communicating lessons learned across these different levels serves to educate about work accomplished, challenges faced, and opportunities for future improvements. Broad communication also ensures that change does not occur in silos and that other teams across the organization learn from and build on prior experiences.