Evaluation of the Strategy for Patient-Oriented Research

Final Report

Table of Contents

This report has been prepared by KPMG LLP (“KPMG”) for the Canadian Institutes of Health Research ("Client") pursuant to the terms of our engagement agreement with Client dated August 10, 2015 (the “Engagement Agreement”). KPMG neither warrants nor represents that the information contained in this report is accurate, complete, sufficient or appropriate for use by any person or entity other than Client or for any purpose other than set out in the Engagement Agreement. This report may not be relied upon by any person or entity other than Client, and KPMG hereby expressly disclaims any and all responsibility or liability to any person or entity other than Client in connection with their use of this report.

© 2016 KPMG and the KPMG logo are registered trademarks of KPMG International, a Swiss entity

List of Acronyms

ACAHO
Association of Canadian Academic Healthcare Organizations
ACCESS
ACCESS Open Minds
CCTCC
Canadian Clinical Trials Coordinating Centre
CD
Capacity Development
CIHR
Canadian Institutes of Health Research
CLAHRC
Collaboration for Leadership in Applied Health Research and Care, UK
CT
Clinical Trials
EAC
SPOR's External Advisory Committee on Capacity Development
ED
Executive Director (typically for SUPPORT Units)
FP7
Seventh Framework Programme of the European Union
GBF
Graham Boeckh Foundation
HC
Health Canada
KI
Key Informants
MRC
Medical Research Council, UK
NICE
National Institute for Health & Clinical Excellence, UK
NSC
SPOR National Steering Committee
PAA
Program Alignment Architecture
PCORI
Patient-Centered Outcomes Research Institute
PE
Patient Engagement
PHAC
Public Health Agency of Canada
PIHCI
Pan-Canadian SPOR Network in Primary and Integrated Health Care Innovations
PMS
Performance Measurement Strategy
POR
Patient-Oriented Research
REB
Research Ethics Board
SPOR
Strategy for Patient Oriented Research
TPMI
Newfoundland Translational and Personalized Medicine Initiative

Executive Summary

Overview

This report presents the findings from the evaluation of the Canadian Institutes for Health Research (CIHR) contributions to Canada’s Strategy for Patient-Oriented Research (SPOR). Although many of the SPOR partners were engaged in the evaluation, the evaluation was of CIHR’s contribution to SPOR only.

The SPOR evaluation covers the period from 2010-11 to 2015-16 and was scoped to address the evaluation elements outlined by Treasury Board related to program relevance, design and delivery, and performance to meet accountability requirements and additionally to inform program decision-making. Due to SPOR’s early stage of evolution, the evaluation focused on the deployment of SPOR core elements and short-term progress towards objectives.

SPOR Context and Profile

Patient-oriented research (POR) refers to a continuum of research that engages patientsFootnote 1,Footnote 2 as partners, focuses on patient priorities and improves patient outcomes individually and in communities such as vulnerable populations. This research, conducted by multidisciplinary teams in partnership with relevant stakeholders, aims to apply the knowledge generated to improve healthcare systems and practices. It involves ensuring that the right patient receives the right clinical intervention at the right time, ultimately leading to better health outcomes.Footnote 1

Announced in 2011, SPOR is a ten-year national Strategy. It encompasses five core elements:

  1. SUPPORT Units: specialized, multidisciplinary research service centres located in provinces and territories across Canada;
  2. Networks: pan-Canadian research networks that represent a collaboration of patients, health service providers, policy/decision makers, and health researchers;
  3. Capacity development: targets activities for training and support for POR;
  4. Patient engagement: supports efforts to engage patients in a meaningful way through active collaboration in governance, priority setting, and the conduct of research; and,
  5. Clinical trials: an element targeted to improve the clinical trials environment in Canada, primarily through the Canadian Clinical Trials Coordinating Centre.

While CIHR plays a leadership role within SPOR, it engages a broad coalition of stakeholders from across Canada.

Each SPOR element has been developed and implemented within varying timeframes. Although SPOR planning, design, partnership development, and foundational elements have been in place since its creation in 2010-11, several SPOR core elements have been in place for less than two years, with most only being active for one year at the time of the evaluation. Overall, SPOR is still in early stages of being implemented to scale and the results of this evaluation, particularly results related to intended outcomes, need to be interpreted with this condition in mind.

Findings

The evaluation found that SPOR is relevant to addressing Canadian health systems needs for integrating research into care. Key findings include:

Some elements of the design and delivery processes for SPOR were found to be effective, but areas for further improvement were also identified.

The design and delivery elements found to be effective include:

There are six areas in which effectiveness can be improved, including:

Although SPOR is in its initial phases of implementation, it has advanced toward achievement of its stated immediate outcome areas. The main areas of achievement include:

In terms of progress toward intermediate outcomes, SPOR is early in its influence on the health research landscape and shifting the culture towards POR. The alignment and integration of the activities of SPOR’s core elements, enhanced engagement of patients, stronger communication of key concepts, future plans and priorities, and early wins will help to increase buy-in for POR and support the further implementation and achievement of intended results.

Recommendations

Overall, SPOR is relevant, addresses an ongoing need for POR, and is demonstrating expected performance given the stage of implementation, supporting the need for continued involvement and investment in SPOR by CIHR; however, some improvements are required to further strengthen its design and delivery.

The following recommendations were identified to improve program performance:

  1. CIHR should increase efforts to strengthen SPOR’s role in a common agenda for change to patient-oriented research.
    • CIHR needs to continue to focus on increasing buy-in of POR and changing the culture, identify and communicate best practices in POR and enhance communication to clarify definitions of many POR terms, including POR itself.
    • SUPPORT Units need to increase communications and outreach to their broad stakeholder community in relation to SUPPORT Unit services available and initiatives undertaken.
  2. CIHR should provide strategic guidance regarding how SPOR elements are to work together toward achieving SPOR’s intermediate and long-term outcomes.
    • CIHR, in collaboration with its established SPOR governance structures, should enhance guidance on operationalizing SPOR elements, in particular, clarifying how elements are expected to work and coordinate together.
  3. CIHR should communicate plans for moving beyond the initial five year funding period to manage sustainability expectations for CIHR investments in SPOR.
    • CIHR needs to provide clear communications regarding SPOR funding, and options beyond the current five-year funding commitment to some elements.
  4. CIHR should strengthen approaches to enable cross-learning, sharing of best practices, and collaboration; this should occur within and across SPOR elements and between CIHR and Canadian and international organizations.
    • CIHR should re-examine the structure, operations and effectiveness of working groups, and encourage cross-provincial initiatives, particularly among SUPPORT Units.
    • CIHR and all SPOR elements should encourage ongoing interaction/connection and relationship building with other POR initiatives.
  5. CIHR should continue to support effective management and administrative functions within funded SPOR SUPPORT Units and Networks and across these elements.
    • CIHR should require SPOR SUPPORT Units and Networks to be supported by CEO/COO-type management positions, if not already present, to help manage operational obligations, administrative requirements and the high corresponding workloads in these areas.
    • CIHR should review the funding model in place, and adjust funding flow based on the stage of development/need of the element.
  6. CIHR should revise the existing SPOR performance measurement strategy to balance administrative/operational outputs with outcomes/impacts.
    • Indicators should be re-oriented from tracking primarily activity-based or output indicators toward outcomes and impacts; consider applying a “collective impact” lens.
    • CIHR should improve its financial monitoring and coding for SPOR grants and awards expenditures (including partner contributions) and for operating and maintenance expenditures.

1. Introduction

This report presents the findings from the evaluation of the Strategy for Patient-Oriented Research (SPOR). This report includes a summary description of SPOR, the approach and methodology used for completing and analysing the data sources supporting the evaluation, and the presentation of concluding findings from all lines of enquiry. Key findings are drawn against the applicable evaluation questions relating to SPOR’s relevance, design and delivery, and performance.

1.1 SPOR Profile

Patient-oriented research (POR) refers to a continuum of research that engages patientsFootnote 1,Footnote 3 as partners, focusses on patient-centric priorities and improves patient outcomes individually and in communities. This research, conducted by multidisciplinary teams in partnership with relevant stakeholders, aims to apply the knowledge generated to improve healthcare systems and practices.Footnote 4

POR can range from initial studies in humans to comparative effectiveness and outcomes research, and the integration of this research into the health care system and clinical practice. The goal of POR is to better ensure the translation of innovative diagnostic and therapeutic approaches to the point of care (i.e., to patient diagnoses and treatments), so as to help ensure greater quality, accountability, and accessibility of care. It aims to ensure that “the right patient receives the right clinical intervention at the right time, ultimately leading to better health outcomes.” While clinical research has provided the basis for the development and application of health interventions, comparative evaluations of these interventions have lagged in their ability to provide guidance as to when and for which patients to apply them. Such results in clinical research have led to a push internationally towards POR, which has become a high priority for international peer organizations including those in the United States, United Kingdom, and Australia.

Various factors in the Canadian environment throughout the turn of the century have added impetus to the need to move forward on the development and implementation of a comprehensive national strategy for POR. First, while investments in health research have led to the development of a vast array of preventive, diagnostic and treatment interventions for health, there has been increasing impatience among clinicians, policy makers and patients with the slow pace at which scientific discovery has resulted in new products or interventions. Second, significant gaps in high quality evidence on comparative effectiveness have shown that it can be difficult to establish guidelines for appropriate care. Third, in tighter economic times, funders of basic biomedical research, including federal and provincial governments and health charities, have been anxious to see and to explain to taxpayers and donors the public benefit of the billions of dollars invested in scientific research.

In response to this identified need, Canada’s Strategy for Patient-Oriented Research was developed to set out a vision about how Canada’s POR model will be designed. SPOR’s Strategy document depicts the Canadian landscape as, in effect, facing a dual challenge in the research-to-practice continuum. The strategy states this often being referred to as the two "Death Valleys". Valley 1 refers to the decreased capacity to translate the results of discoveries generated by basic biomedical research in the laboratory to the bedside or careside as well as to successfully commercialize health discoveries. This negatively impacts Canada's clinical research and knowledge base and its international competitiveness. Valley 2 refers to the limited capacity to synthesize, disseminate and integrate research results more broadly into health care decision-making and clinical practice.Footnote 1

Exhibit 1 – The Two Valleys of the Research-to-Practice Continuum

Low capacity to translate the results of discoveries generated by basic biomedical research in the lab to the bedside as well as to successfully commercialize health discoveries. Bridging Valley 1 by fostering public-private partnerships and by providing clinical validation of research results.

Limited capacity to synthesize, disseminate and integrate research results more broadly into health care decision-making and clinical practice. Bridging Valley 2 by ensuring that research results are integrated into the healthcare system.

Adapted from Steven Reis, University of Pittsburgh and Harold Pincus, Columbia University.

Exhibit 1 long description

Exhibit 1 describes how Canada faces a dual challenge in the research-to-practice continuum, often referred to as the two Valleys. Valley 1 is represented as a dip in a line graph that happens in between basic biomedical research and Clinical Science & Knowledge. This negatively impacts Canada’s clinical science and knowledge base and its international competitiveness. Valley 2 is represented as another dip in a line graph that happens between clinical science and knowledge and clinical practice and health-care decision-making. These two valleys must be bridged if Canada is to bring evidence to bear to enhance health outcomes and ensure a sustainable health care system.

In response to these pressures, SPOR was developed with the vision to demonstrably improve human health outcomes and enhance patients' health care experience through integration of evidence into the health care system and clinical practice. It is expected that through the integration of research evidence into clinical practices, the act of putting patients first would support Canada in ensuring that research will have a greater impact on treatments and services provided in clinics, hospitals and doctors' offices throughout the country.

Chevrons at the bottom of the exhibit represent the research cycle starting with discovery and moving to Translation, Evaluation and Implementation. SPOR begins at the second chevron, Translation. SPOR does not focus on basic biomedical research but on 1st in human studies and beyond.

In response to these pressures, Canada’s Strategy for Patient-Oriented Research was developed with the vision to demonstrably improve health outcomes and enhance patients' health care experience through integration of evidence into the health care system and clinical practice. SPOR was first announced in August 2011. It is expected that through the integration of research evidence into clinical practices, the act of putting patients first would support Canada in ensuring that research will have a greater impact on treatments and services provided in clinics, hospitals and doctors' offices throughout the country. SPOR represents a shared agenda among federal, provincial and territorial partners dedicated to the integration of research into care.

The immediate and intermediate outcomes defined for SPOR include:

Immediate Outcomes Intermediate Outcomes
  • Key stakeholders, including patients, are supported to engage in research & decision-making processes
  • Multi-disciplinary, integrated and accessible research platforms are established
  • Pan-Canadian networks are mobilized to address health challenges
  • SPOR elements provide capacity development opportunities that align with the SPOR CD framework
  • CCTCC is established
  • Methods in patient-oriented research are advanced
  • Patients, health care providers, and decision-makers are active partners in both research and implementation of evidence-based improvements
  • Infrastructure and support services are responsive to stakeholder and health systems needs
  • Research evidence is generated, published, disseminated and integrated into health care practice and policy
  • Strengthened Canadian capacity in patient-oriented research
  • Improved clinical trial environment in Canada through CCTCC, specialized services and methodological expertise

An important principle of SPOR is the leveraging of non-federal partner funds to match, at a 1:1 ratio, the investments made from the federal government toward the core elements of SPOR. To date, $357 millionFootnote 5 has been invested or committed by CIHR.

Several additional principles are in place for the federal investments in the core elements of SPORFootnote 6.The principles include:

1.2 Core Elements of SPOR

SPOR consists of five core elements, with each element having been structured to address specific challenges identified as having delayed or prevented the translation of high quality research to improvements in patient outcomes within Canada. These elements include:

SPOR was initially launched in 2011, with the release of the formal Strategy document. In the five years since that time, each SPOR element has been developed and implementedFootnote 7 within varying timeframes. For the most part, SPOR core elements have been in place for a maximum of two years, with most only being active for one year.

Official program documents indicate there was a planned delay through the first two years of SPOR with activities identified for 2011-12 to include the establishment and engagement of SPOR’s National Steering Committee, the determination of priorities, the creation of the funding opportunities for the Networks and SUPPORT Units, as well as some specified work in the Clinical Trials area.

Additionally, there are a number of relevant and important programs preceding SPOR that helped to set the stage/prime the Canadian clinical and patient-oriented research community for the launch and implementation of SPOR. SPOR has a number of Foundational Investments associated with it (e.g., operating grants, catalyst grants, knowledge synthesis grants. These Foundational Investments are aligned with SPOR; however, may have begun prior to SPOR being announced, maintained during the design and implementation of SPOR, or were sunsetted post-SPOR implementation.

For example, there are a number of Foundational Investments that were, and continue to be, important, as most play a role in the development of SPOR-relevant research capacity and are directly aligned with SPOR. For example:

These Foundational Investments are not included in the evaluation’s scope.

Exhibit 2 depicts the progressive implementation of the core SPOR elements.

Exhibit 2 – Timeline of SPOR Elements and Key Documents since 2011

Exhibit 2 long description

This figure is a timeline of SPOR elements and lay documents since 2011.

SPOR was initially launched in 2011, with the release of the formal Strategy document: Canada’s Strategy for Patient-Oriented Research. The National Steering Committee (NSC) was established also in 2011.

In 2012 the SPOR Working Group was established.

In 2013 the SPOR Performance Measurement Strategy was developed and the SUPPORT Unit Council (SSUC) was established. The first four SUPPORT Units were implemented in Alberta, Manitoba, Maritimes and Ontario.

In 2014 two more SUPPORT Units were implemented in Newfoundland & Labrador and Quebec and two SPOR Networks were launched: ACCESS Open Minds and Primary and Integrated Health Care Innovations (PIHCI). Also the Canadian Clinical Trials Coordinating Centre (CCTCC) was established and the SPOR Patient Engagement Framework was developed.

In 2015 the SPOR Capacity Development Framework was developed and Patient Engagement Collaboration Grants were initially launched.

In 2016 three more SUPPORT Units were implemented in British Columbia, Saskatchewan and Northwest Territories, the SPOR Network in Chronic Diseases is launched and Patient Engagement Collaboration Grants continue to be offered.

The description and evolution of each core SPOR element is described in the following sections.

1.2.1 SUPPORT Units

Support for People and Patient-Oriented Research and Trials (SUPPORT) Units are locally accessible, multidisciplinary groups of specialized research resources, policy knowledge, and patient perspectives. SUPPORT Units have been created to provide support and expertise to those pursuing POR. Additionally, they are tasked with facilitating decision-making within the health services setting, fostering the implementation of best practices, and promoting collaboration among researchers engaged in POR.

SUPPORT Units have been or are being established in collaboration with the provinces and territories, who have a significant role in directing the work they carry out.

The concept of SUPPORT Units was mapped out at a one-day meeting in March 2011 that included researchers, clinicians, policy makers, and CIHR representatives. In 2012, CIHR made available to all provinces/territories, a funding opportunity that outlined a set of instructions to assist with the development of jurisdictional SUPPORT Unit business plans. The first SUPPORT Units were approved for funding in 2013 and spent much of that year finalizing strategies and work plans.

Each SUPPORT Unit has committed to the following six key components (these are CIHR requirements):

Each SUPPORT Unit must also undertake a demonstration project or projects. These are meant to show the benefits of POR, and demonstrate quickly that the POR model can work.

To date, CIHR has invested $68MFootnote 8 in SUPPORT Units (Alberta, Manitoba, Maritimes, Ontario, Newfoundland & Labrador, and Quebec) and will invest $127MFootnote 9 over the next five years to roll out the remaining SUPPORT Units (British Columbia, Saskatchewan, and three territories). The implementation stage of the SUPPORT Units as of February 2016 is:

A SUPPORT Unit Council (SSUC) was established in 2013. The mandate of the SSUC is to provide an opportunity for information sharing and collaboration among SUPPORT Units throughout the development and implementation of this element of SPOR. It does not have powers to compel actions.

The SSUC is comprised of two co-chairs, one from CIHR and one from the SUPPORT Units, one lead person from each SUPPORT Unit and two other members from CIHR. The SSUC Terms of Reference identifies they have been meeting four to five times per year.

The SSUC focuses its attention on the following:

SSUC Working Groups (WGs) were formed in 2014 and 2015 to address issues of common interest and are designed to focus on a particular issue. The WGs report to the SSUC and have been established as follows:

1.2.2 SPOR Networks

SPOR Networks are national collaborative research networks involving the full range of SPOR stakeholders (patients, health professionals, decision makers, health researchers and other stakeholders). They focus on specific health challenges identified as priorities in multiple provinces and territories. They are intended to pursue research and generate evidence and innovations designed to improve patient health and health care systems.

In January 2011, CIHR hosted a round-table meeting to discuss the vision for the pan-Canadian Networks. The meeting’s goal was to determine how Canadian clinical research networks should be best organized, governed, and resourced. The specific goals of the roundtable were:

  1. To understand what constituted a Canadian Patient-Oriented Research Network and how it might best function.
  2. To share best practices and realities in developing and running national research networks.
  3. To provide CIHR with guidance on the development and implementation of future Canadian Patient-Oriented Research Networks and information needed to develop funding opportunities related to them.

Through the SPOR development process, CIHR solicited stakeholder input on research areas of greatest need. Based on the input received, the SPOR National Steering Committee (NSC) then helped to refine the areas in which SPOR funding should be targeted for the creation of Networks.

The application process has been different for each of the Networks. The process may include a combination of the following: expressions of interest, letters of intent, strengthening workshops and full applications. The purposes of the strengthening workshops are to strengthen applications and facilitate dialogue between applicants, potentially identifying synergies among applicants. In some instances, funding is allocated to applicants through the letter of intent stage to facilitate development of the full application.

To date, seven networksFootnote 11 in three different areas, described below, are being implemented to deliver on the SPOR objectives. These networks are pan-Canadian initiatives.

From 2010-11 to 2020-21, CIHR has committed $83.2 for Networks, of which $12.6M had been spent during the timeframe of the evaluation (2010-11 to 2015-16).

SPOR Network in Youth and Adolescent Mental Health – ACCESS Open Minds

The SPOR Network in Youth and Adolescent Mental Health, with delivery through ACCESS Open MindsFootnote 12, aims to bring about transformational change in addressing adolescent and youth mental health and well-being. The Network seeks to improve the care provided to young Canadians with mental illness through assisting in connecting patients and youth with researchers, health care professionals, and decision-makers in order to foster the translation of research findings into practice and policy.

The funding opportunity for the Youth and Adolescent Mental Health Network was launched in January 2013. In June 2014, the ACCESS Open Minds Canada research network for youth and adolescent mental health was announced. The Network is a collaborative effort between CIHR and the Graham Boeckh Foundation.

Pan-Canadian SPOR Network in Primary and Integrated Health Care Innovations (PIHCI)

In November 2013, the funding opportunity for the Primary Integrated Health Care Innovations (PIHCI) Network was launched. PIHCI entered Phase I implementation in 2014. Phase 2 of PIHCI proceeded through 2015 with Quick Strike research projectsFootnote 13 and applications for full membership in the pan-Canadian Network.

This pan-Canadian Network (a network of networks) is intended to support an alliance between research, policy and practice to create responsive learning networks. It is intended to deal with delivery of care both within and across sectors of health care (e.g., public health, home and community care, primary, secondary, and tertiary care) and outside the health sector (e.g., education, social services).

PIHCI is a key CIHR initiative under SPOR and also the Community-Based Primary Health Care Signature Initiative (CBPHC). A core requirement for PIHCI Network membership necessitates members to link with CBPHC Innovation Teams in a member’s jurisdiction.

SPOR Networks in Chronic Disease

In October 2014, the funding opportunity for the Networks in Chronic Diseases was launched. The focus of these networks is on the translation of existing and new knowledge generated by basic biomedical, clinical, and population health research into testing of innovations that can improve clinical science and practice and foster policy changes, leading to transformative and measureable improvements in patient health outcomes, and in efficient and effective healthcare delivery within five years.

Applications were submitted in early October 2015, with results released in late February 2016. Five networks were funded in March 2016. As these networks were in the applications phase at the time of this evaluation, they were purposely excluded from the evaluation scope.

1.2.3 Capacity Development

Capacity development is intended to grow, support and sustain the capacity for a collaborative, interdisciplinary and innovative patient-oriented research environment capable of addressing evolving health care questions, contributing to enhancing patients' health care experience and improving health outcomes.

In August 2012, an External Advisory Committee (EAC) was mandated to develop a report identifying the deficiencies in patient-oriented research within Canada. In 2013, a national workshop was held to collect feedback on barriers to patient-oriented research. The EAC drafted guiding principles and suggestions for implementation and released its report “Training and Career Development in Patient-Oriented Research” in June of 2013. Further consultation with SPOR stakeholders including those that are involved directly in building capacity in a variety of sectors took place through a workshop on capacity development in March 2014, with a final CD Framework being released in August 2015.

The SPOR CD FrameworkFootnote 14 was designed to encourage a shared vision, key principles, and considerations for capacity development in POR. In alignment with this framework, training, mentoring, and career support is to be integrated into the SPOR Networks and SUPPORT Units, with each Network and SUPPORT Unit being required to articulate a training and capacity development strategy.

1.2.4 Clinical Trials

An important goal of SPOR is to strengthen organizational, regulatory, and financial support for clinical trials in Canada and enhance patient and clinician engagement in these studies. Implementing and funding multi-centre clinical trials is difficult in the current Canadian context, and Canada is perceived to be losing its competitive edge. To improve Canada's competitiveness in conducting clinical trials, SPOR has developed this element, which is designed to help overcome a number of identified barriers. The first ever Canadian Clinical Trials Summit was held on September 15, 2011 in partnership by CIHR, HealthCareCAN (formerly the Association of Canadian Academic Healthcare Organizations), and Innovative Medicines Canada (formerly Canada’s Research-Based Pharmaceutical Companies (Rx&D)). The purpose of the Summit was to include various public, private and academic participants in the development of an action plan to “further guide academia and clinical sites, government, and industry on a common path” in order to “help Canada regain its capacity to attract clinical trials.” The first recommendation of the Action Plan called for the development of “a national headquarters for clinical trials improvement activities”. The Canadian Clinical Trials Coordinating Centre (CCTCC) was subsequently established through a joint public, private, and academic partnership between CIHR, Innovative Medicines Canada, and HealthCareCAN in 2014. The CCTCC is working to implement the recommendations from the 2011 Clinical Trials Summit Action PlanFootnote 15, which were to:

  1. Establish implementation and coordination headquarters and resources;
  2. Measure, monitor, manage, and market clinical trial performance improvements;
  3. Integrate health system and research infrastructure to ensure quality and sustainability;
  4. Improve efficiencies of ethics reviews and advance strategic issues;
  5. Develop a database of registries and consider a national patient recruitment strategy;
  6. Adopt common Standard Operating Procedures, training and certification;
  7. Improve and use the common clinical trials contract;
  8. Optimize intellectual property protection policy, scientific research and experimental development tax credits; and
  9. Signal Canada’s interest globally.

Additionally, the clinical trials element is incorporated as a core function of each provincial/territorial SUPPORT Unit, as well as through the clinical trials activities conducted across the SPOR Networks, and through the engagement of clinicians and clinical researchers across the SPOR governance and reporting structures.

1.2.5 Patient Engagement (PE)

By encouraging a diversity of patients to tell their stories, new themes may emerge to guide research. Patients are expected to gain many benefits through their involvement in research, including increased confidence and mastery of new skills, access to information they can understand and use, and a feeling of accomplishment from contributing to research relevant to their needs.

A key objective of Canada's SPOR is for patients, researchers, health care providers, and decision-makers to actively collaborate to build a sustainable, accessible and equitable health care system. Experience has shown that the priorities of clinical researchers and healthcare systems may not always perfectly match the priorities, concerns, and needs of patients.

Engaging patients is therefore a vital element to be integrated with the development and implementation of all elements of SPOR, such as SUPPORT UnitsFootnote 16 and Networks. There is also activity in sharing knowledge on PE across SPOR elements. For example, the SUPPORT Units are noted to be actively sharing and involved in CIHR-sponsored workshops on the topic.

In response to feedback regarding the need to clarify SPOR’s approach to PE from a variety of sources, including the research community, SPOR's National Steering Committee requested the development of a PE Framework. The PE Framework has been designed to establish key concepts, principles, and opportunities for PE in identifying health research priorities and in designing and conducting research projects.

In January 2014 and the spring of 2014, CIHR hosted a workshop and consultations to develop the SPOR PE Framework. This Framework was published and disseminated to stakeholders in June 2014.

The SPOR PE FrameworkFootnote 17 elaborates on what patients contribute to the research process and why it is needed:

“Patients bring the perspective as "experts" from their unique experience and knowledge gained through living with a condition or illness, as well as their experiences with treatments and the health care system. Involvement of patients in research increases its quality and, as health care providers utilize research evidence in their practice, increases the quality of care.

Engaging patients in health care research makes (investments in) research more accountable and transparent, provides new insights that could lead to innovative discoveries, and ensures that research is relevant to patients concerns. The international experience with engaging citizens and patients in research has shown that involving them early in the design of studies, ideally as early as at the planning stage, leads to better results."Footnote 18

As a partner in SPOR and seeking to align itself with the SPOR PE Framework, CIHR developed a citizen and patient engagement (CPE) implementation strategy with a number of cross-cutting components, some with direct implications for SPOR. These particular components are also at various stages of implementation:

Examples of further work by CIHR include:

1.3 Target Populations

All Canadians are expected to benefit from SPOR, as it is hoped to lead to:

The patient should benefit from SPOR through receiving the right care in the right place at the right time. Patient-oriented researchers should benefit from training, research support services and an improved environment for clinical research. Health care professionals and policy makers should benefit from the timely and efficient translation of research innovations from the research setting to patient care settings, as well as the evaluation and synthesis of existing knowledge and its proper transfer to the clinical setting. Finally, the provincial governments and health care administrators should benefit from a more cost-effective, efficient, and affordable health care system.

1.4 Stakeholders

SPOR maintains a broad range of stakeholders, involved and interested in the development and implementation of its objectives including:

Federal Departments Provincial and Territorial Governments and Funding Agencies National Partners and Stakeholders International Stakeholders

Other federal departments with an interest in issues relevant to SPOR may be consulted by SPOR where appropriate. These include:

  • Health Canada
  • Innovation, Science and Economic Development
  • Global Affairs
  • Public Health Agency of Canada
  • Statistics Canada
  • Treasury Board Secretariat
  • Ministry of Finance
  • Privy Council Office

SPOR partners maintain linkages with provincial and territorial governments, primarily through the following mechanisms which have responsibilities to address health issues:

  • Federal/Provincial/Territorial Ministers Responsible for Health
  • Federal/Provincial/Territorial governments and agencies
  • National Alliance of Provincial Health Research Organizations

SPOR maintains linkages with many players concerned with POR, including patients and caregivers, health charities and other not-for-profit organizations, researchers, academic institutions, health practitioners, health organizations, and the pharmaceutical sector. Some examples include the following:

  • Health Charities Coalition of Canada
  • Graham Boeckh Foundation
  • Canadian Institute for Health Information
  • Universities Canada
  • Canadian Association of Research Administrators
  • Canadian Medical Association
  • Canadian College of Health Leaders
  • Innovative Medicines Canada
  • HealthCareCAN

SPOR maintains linkages with several international organizations, which include:

  • National Institutes of Health, US
  • Patient-Centered Outcomes Research Institute, US
  • Medical Research Council, UK
  • National Institute for Health & Clinical Excellence, UK
  • National Coordinating Centre for Public Engagement, UK
  • James Lind Alliance, UK
  • National Institute for Health Research, UK
  • INVOLVE, UK
  • National Health and Medical Research Council, Australia

1.5 Governance

The governance structure of SPOR consists of a National Steering Committee and a CIHR SPOR Working Group. The governance structure is supported by the Priority-Driven Research Branch within the Research, Knowledge Translation and Ethics (RKTE) Portfolio of CIHR. In addition, each SUPPORT Unit and SPOR Network is required to have a governance structure, which includes appropriate mechanisms for PE.

SPOR National Steering Committee

The National Steering Committee (NSC) was established in 2011 and oversees SPOR development and implementation.

The committee is co-chaired by the Deputy Minister of the Ontario Ministry of Health and Long Term Care and the President of CIHR. Members include patients, federal/provincial/territorial governments, provincial health research funders, academic institutions, health care organizations, researchers, health charities and industry. The National Steering Committee’s composition also respects a nationwide geographical representation. CIHR provides secretariat services for the Committee.

SPOR Working Group

The SPOR Working Group was established in 2012 and provides scientific leadership within CIHR for the development, implementation and co-ordination of CIHR’s activities and initiatives related to SPOR. The Working Group also provides on-going monitoring and identifies refinement of activities and initiatives, as needed. Ad-hoc external advisory committees are engaged for advice on various specific issues as necessary.

The Working Group is chaired by the Chief Scientific Officer of CIHR, and consists of five Institute Scientific Directors, each being a champion of one of the five SPOR elements. It also includes the Associate VP Research, Knowledge Translation and Ethics, the Director General of Priority-Driven Research Branch, and Manager of Major Initiatives.

1.6 CIHR Institute Involvement in SPOR

CIHR integrates research through a unique interdisciplinary structure made up of 13 "virtual" institutes. Each Institute is dedicated to a specific area of focus, linking and supporting researchers pursuing common goals. All CIHR Institutes are engaged in SPOR, though to varying degrees. Institute engagement with SPOR can be characterized in the following ways, which are not mutually exclusive (see Appendix B for details):

1.7 Resources

An overview of CIHR grants and awards expenditures on SPOR over the evaluation timeframe is provided in the table below:

2010-11 2011-12 2012-13 2013-14 2014-15 2015-16 Total
$63,561,791 $54,927,762 $49,018,043 $57,581,279 $61,022,134 $70,729,241 $356,840,250

*Summary of figures provided by CIHR current as of March 21, 2016.

1.8 Evaluation Purpose and Scope

SPOR is a complex and complicated program and reflects a large system of stakeholders, delivery partners, activities, and linkages to other programming efforts both within CIHR and in the provinces and territories. Although many of the SPOR partners were engaged in the evaluation through interviews, the evaluation was of CIHR’s contribution to SPOR core elements only.

The SPOR evaluation covers the five-year period 2010-11 to 2015-16 and was scoped to cover issues of relevance, design and delivery, and performance. The purpose of the evaluation is twofold:

In order to prioritize the focus on the implementation of the core elements and assess progress toward immediate outcomes as outlined in the SPOR performance measurement strategy, the following calibration considerations were made:

Given these scoping considerations, the evaluation focused on the deployment of SPOR core elements and short-term progress towards objectives, and used the program profile contained in the SPOR Performance Measurement Strategy as the baseline implementation plan.

The focus on Foundational Investments was limited to outlining the investments made through the evolution of SPOR from 2010-11 to 2015-16 as well as considering these investments in the assessment of efficiency under the performance issue. In terms of assessing the outcomes of the Foundation Investments, these were scoped out given they were not deliberately designed at the outset to achieve SPOR outcomes as stated in the logic model, resources to assess the investments were not available, and some of the investments were in the scope of previous, ongoing (e.g., scholarships) and planned (e.g., operating support) evaluative activities.

The scope of the evaluation is summarized in the following table:

The evaluation covers: The evaluation does not cover:
  • The time period 2010-11 to 2015-16
  • CIHR investments in SPOR only
  • Five core elements of SPOR to assess overarching design, delivery, and progress towards outcomes, limited to:
    • SUPPORT Units (Alberta, Manitoba, Maritimes, Ontario, Newfoundland and Labrador, Quebec).
    • Research Networks (ACCESS Open Minds, PIHCI (limited).
    • Capacity Development
    • Improving Clinical Trials Environment (CCTCC)
    • Patient Engagement
  • SPOR Networks in Chronic Diseases (still in competition at time of evaluation).
  • Overall added-value of the Strategy or the coherence across the SPOR core elements due to its early stage evolution.
  • A comparison with other initiatives due to the uniqueness of SPOR. However, interviews with international stakeholders explored how other countries are approaching initiatives with similar objectives.
  • Input from the general researcher population (i.e., those not directly involved in SPOR in some way).
  • “Foundational Investments” (e.g., operating grants, catalyst grants, knowledge synthesis grants).
  • Other investments (e.g., Studentships, Fellowships, Clinical Trials funding).
  • CIHR’s Citizen and Patient Engagement Strategy (Patient Engagement Collaboration grants and initiatives under development).
  • CIHR Institute involvement in SPOR.

Further elaboration of some of the components excluded from the evaluation scope can be found in section 2.3 on study limitations.

1.9 Evaluation Questions

A set of evaluation questions were prepared and vetted by CIHR. The following seven questions were addressed in the evaluation:

SPOR Evaluation Issues and Questions

Relevance
  1. To what extent does the research funded under SPOR address the need for evidence-informed health care?
  2. To what extent is SPOR aligned with federal roles and responsibilities?
  3. To what extent is SPOR aligned with federal government and CIHR priorities?
Design and Delivery
  1. To what extent has SPOR been implemented as planned?
Performance
  1. To what extent has SPOR made progress toward the achievement of expected immediate outcomes?
  2. To what extent has SPOR made progress toward the achievement of expected intermediate outcomes?
  3. To what extent is SPOR being delivered in a cost-efficient manner?

1.10 Report Structure

The findings and analysis for each of the seven evaluation questions are provided in the following sections of this report. Each section defines the specific evaluation question(s), summarizes the key findings against each of the issue areas, provides details on the analysis and evidence and provides supporting conclusions. There is a final chapter on recommendations for improvement.

2. Methodology

To provide cross-cutting representation and feedback from the various stakeholders involved in SPOR, the evaluation was designed to use multiple lines of evidence. With a very complex program design and implementation that is early in its lifecycle, along with variations by jurisdiction and degrees of progress, the evaluation design relied heavily on interview and case study techniques. Overall, the methodology included the following data sources:

The data sources have been mapped to each of the evaluation issues in the following table:

Data Sources
  Document Review Performance Data Review Key Informant Interviews Case Studies Governance and Admin Network Analysis International Comparative Review
Relevance    
Design and Delivery  
Performance  

Each of the data sources are described in the sections following and the supporting instruments are provided in the appendices to this report. The number of people interviewed for the key informant interviews and the number of people interviewed for the case studies are presented collectively in the following Exhibits 3 (by method) and 4 (geographically).

Exhibit 3 – Evaluation Response Rates by Method

Method Total
Case Studies  
Capacity Development 2
Clinical Trials 4
Networks 6
Patient Engagement 10
SUPPORT Units 68
International KI Interviews 3
Key Informants 15
Grand Total 108

Exhibit 4 – Evaluation Response Rate by Geography

Jurisdiction Total
Alberta 11
British Columbia 6
Manitoba 14
Maritimes 19
NL & Labrador 11
Nunavut 1
NWT 1
Ontario 25
Quebec 14
Saskatchewan 2
Yukon 1
International 3
Grand Total 108

2.1 Data Sources

2.1.1 Document Review

Approximately 40 documents were reviewed, mainly in support of the evaluation questions related to relevance. These included CIHR strategic documents (such as its annual reports and the SPOR overarching strategy document), federal and provincial level strategic documents and reports, industry evaluations, SPOR element specific reporting and other grey literature. The document review was also used to form the contextual description of SPOR utilizing early planning documents to report on the background and vision of the strategy.

2.1.2 Performance Data Review

The evaluation compiled a summary of activity data presented in the 2013-2014 and 2014-2015 SUPPORT Unit Annual Performance Reports as well as the 2014-2015 Annual Performance Report pertaining to the ACCESS Open Minds Network. This was the entire scope of performance reporting available during the evaluation timeframe. The performance data selected for presentation in this report aligns to the performance indicators as identified in SPOR’s performance measurement strategy. The data is reported at an aggregate level representative of SUPPORT Unit activities collectively. The data presented in this report has not been validated by KPMG and is presented as submitted to CIHR by the provincial SUPPORT Units and the ACCESS Open Minds network.

2.1.3 Key Informant Interviews

A key informant interview sample of 19 individuals was developed that included representatives from CIHR management as well as a sample of SPOR partners, stakeholders and users as part of the evaluation work. The sample was purposefully selected to include individuals with broad knowledge of SPOR. Considerations for interviewee selection included knowledge across all SPOR elements as well as (to the extent possible) other factors such as the academic/institutional/healthcare/patient involvement dimensions, and provincial/territorial representation. The sample of 19 individuals was identified by SPOR program personnel. Fifteen interviewees participated out of the sample (15/19=80% response rate).

2.1.4 Case Studies

Five case studies were developed that focused on the implementation of the five core elements of SPOR in a cross-cutting way against all of the SPOR programming. Two sets of questions were addressed with case study interviewees. The first set of questions related to the specific core element of SPOR being studied and the second set of questions related to the overall strategy of SPOR. The five cases conducted, ordered alphabetically, include:

2.1.5 Governance and Administrative Network Analysis

The governance and administrative network analysis was employed as a minor supporting method in the evaluation. Data were collected for each SPOR component having information available through a public internet site. Data were collected for each SUPPORT Unit and the SPOR National Steering Committee in this manner. Contact lists provided from the SPOR program in support of the evaluation were also used to compile data for the PIHCI network, SPOR SUPPORT Unit Council and CIHR’s internal SPOR Working Group.

Web sites were mined for all information related to each component’s governance structure. This included governance, management, and advisory committees. A total of 296 data points were used in the network analysis.

The data elements collected to form the initial edge listFootnote 19 included the person’s name, the organization with which they were associated, the SPOR element with which they were associated, their province/territory, role, and the part of the governance or corporate/administrative structure to which they belonged.

This data were then cleaned and standardized before importing to two separate social network analysis software packages. Two packages were used to obtain the graphical functionality and the mathematical processing functionality required for analysis and presentation. Data were imported at the organizational level, not at the level of each individual. For example:

If the following ties were identified:

The following data were imported:

A tie represents a connection from one organization to another. The ties in this example are between the University of Alberta and the SPOR NSC and the University of Alberta and the Alberta SUPPORT Unit.

NodeXLFootnote 20 was used to create the network graphs and assemble groups based on the network attributes of “province/territory” and “role”. UCiNETFootnote 21 was used for the mathematical calculations and analysis of degree centralityFootnote 22 and betweennessFootnote 23.

Typical terminology found in social network analysis has purposely not been used (to the extent possible) in this report for ease of portraying the information to readers unfamiliar with the methodology or network theory.

2.1.6 International Comparative Review

The international comparative review was undertaken to explore how other countries are approaching POR initiatives with similar objectives to SPOR. The objective of the review was to gather information on the relative merits of other approaches and models and to obtain perspectives on the pros and cons of the SPOR model versus others. A direct comparison with other initiatives was not undertaken due to the uniqueness of SPOR (large and nationally based).

Through scoping interviews, three initiatives from two organizations were mentioned most frequently as being of most interest to SPOR. These organizations were selected for their experience in POR. The two main international comparison organizations which were reviewed are: The Patient-Centered Outcomes Research Institute (PCORI) in the US and the UK’s National Institute for Health Research with two initiatives: INVOLVE and the Collaboration for Leadership in Applied Health Research and Care (CLAHRC).

The comparative review was conducted in two stages. First the websites for each of the three initiatives were mined for descriptive information on the models/approaches employed. This was followed by interviews with one representative from each of the initiatives. The contacts for PCORI and INVOLVE were identified through the SPOR program. The SPOR program could not identify contacts for the CLAHRC so respondents were identified through the interviews conducted with key informants and case studies. The main purpose of the interviews was to obtain opinions on SPOR’s key strengths and weaknesses as compared to a respondent’s patient-oriented initiative and to identify lessons learned through their experience in implementation and delivery.

2.1.7 Cost efficiency analysis

By assessing not only a program's relevance and performance, but also the resources the program uses, value for money can be determined.

For the SPOR evaluation, both allocative and operational efficiency were examined in order to determine: (a) whether resources requested from TBS were spent on the activities for which they were requested; and (b) whether this spending occurred at the level and in the timeframe for which it was requested. Resources are considered in relation to the outputs or outcomes achieved, respectively.

The SPOR evaluation was designed to focus only on the SPOR core elements – SUPPORT Units, Networks, CD, PE and Clinical Trials. “Foundational Investments” as well as other investments that are part of SPOR (e.g., Studentships; Fellowships, Clinical Trials funding) were excluded from the evaluation of effectiveness and relevance due to feasibility (available budget, timeframe of evaluation vis a vis funding flow) and direct and deliberate relevance of the timing of these investments relative to SPOR. In many cases, the Foundational investments were well-alignedFootnote 24 with SPOR, however may have begun prior to SPOR being announced, maintained during the implementation of SPOR, or had sunsetted post-SPOR implementation (e.g., operating grants, catalyst grants, knowledge synthesis grants). However these investments are included in the cost-efficiency analysis to help ensure a complete accounting of resources used to deliver on SPOR.

Allocative and Operational Efficiency

Data for the cost-efficiency analyses were provided from CIHR’s Finance and Administration Branch, and Performance Measurement, Reporting, and Data Unit and confirmed with the SPOR Program Manager and staff. These data were examined in relation to the Treasury Board submissions for SPOR over the same period of time. Data for the cost-efficiency analysis included:

A fidelity assessment (O’Connor, Small & Cooney, 2007) was conducted to determine the allocative efficiency and assess the degree to which SPOR was implemented as planned and to determine the extent to which these implementation variances impacted the outputs, outcomes, and costs. In the case of the SPOR cost-efficiency analysis, the fidelity assessment is defined generally as a qualitative approach to assessing operational efficiency. This approach focuses on assessing the degree to which a program was implemented according to its initial plans, identifying variances in implementation, and examining these in order to determine the rationale for the variances and the effect they had on costs or on the achievement of outputs or outcomes. These approaches also allow for an assessment of the changes in delivery made by the program in its implementation approach and the impact that these had on costs or on the production of outputs or achievement of outcomes.

A proportion for operational efficiency was calculated annually for SPOR by dividing the total administration costs (direct and indirect) by the total SPOR expenditures (administration costs and grants and awards expenditures) (Exhibit 23).

As SPOR core elements have stated requirements for contributions for partnership funding (see Section 1.1), the extent of external partner contributions to SPOR were examined by tabulating applicant level partner contributions from 2010-11 to 2020-21, as provided in the applications. Partner contributions included cash and cash equivalent in-kind contributions for the full timeframe of the respective investments. A ratio of CIHR to partner contributions was calculated by SPOR element.

2.2 Data Analysis

Technical reports were prepared for each data source. For the data sources that included interviews, interview notes were prepared for individual interviews conducted and were then summarized and consolidated into an Excel database. Summarized notes were organized by question as presented in the interview guide and then mapped to the corresponding evaluation question. Questions were laid out in rows and the interviewee names formed the columns. A multi-stepped approach was used for the analysis. The first step was to complete a review of all interview summaries. This review provided the analyst with a preliminary understanding of the issues arising from the interviews. The next step in the process was a detailed review of the interview summaries looking at each interview question, or logical groupings of questions, noting keywords and statements and identifying common themes through the content analysis. Finally, findings were aligned by summarizing findings for each evaluation question.

In the development of the findings associated with the interviews, the analyst took into consideration the number of people or percentage of people who provided a specific response, comment or discussion along the same theme to understand shared or similar opinions across the group. Additionally, where there were rating questions associated with open ended questions, the analyst compared the qualitative findings with the quantitative findings to qualify coherence in the results and assess internal consistency.

For the purpose of this report, the following terminology is used to specify if there were single ideas or ideas shared by more than one individual:

Caution should be applied in over interpreting the magnitude of specific perspectives; a lack of response by a participant may not mean they do not have an opinion or similarly, an opinion by a few participants may not be shared by many more participants. Additionally, not all interviewees answered all questions. The number of respondents differs for each question and the analysis was performed based on the responses received for each question separately. As a result, the magnitude of responses presented in this report is based upon the sample of interviewees providing feedback for each relevant question. The average response ratings have been calculated as a standard weighted average based on the number of respondents.

A similar process was used to prepare the overall evaluation report, where each technical report was reviewed to identify logical groupings of findings across data sources noting keywords and statements and identifying common themes through the content analysis. Finally, findings were aligned by summarizing findings for each evaluation question and then rolled up to reflect the three evaluation issue areas.

2.3 Strengths and Limitations of the Evaluation

2.3.1 Strengths

Early stage of implementation. The evaluation was conducted at a time early enough to collect real-time feedback illuminating potential areas of opportunity to influence ongoing delivery. The evaluation was designed to focus on learning, while still measuring progress towards outcomes (measurable change).

Coverage of stakeholders. All SPOR stakeholder groups were reached through implementation of the study methodology; this included researchers, policy makers, health system decision makers, clinicians, patients, and industry. Additionally, a large cross section of respondents was reached based on geographical location, with interviews being conducted with stakeholders situated in every province and territory across Canada.

Depth of enquiry to evaluation questions. The evaluation methodology relied heavily on interview and case study techniques, which are much finer instruments for delineating the “how, what, why, and extent” of effects taking place. Data collection instruments were designed specifically to support inquiry into SPOR as an overarching instrument in addition to specific probing at the SPOR element level. Extensive personal interviewing to discuss progress, achievements, and impacts in depth, allowed for follow-up questioning. The open-ended questions were supported by quantitative rating questions that assisted the study team in understanding the general tenor of responses and aided in the roll up of data for analysis.

Rigorous challenge and review process of evaluation findings and conclusions. All evaluation reporting, technical line of evidence reports and draft findings reports were presented to and reviewed with CIHR’s Evaluation Unit, the SPOR program, and in some instances CIHR’s senior management team and the SPOR Working Group. These exercises provided the opportunity to challenge the evaluation findings, provide additional evidence and confirm contextual data being presented. 

2.3.2 Limitations

Potential bias in opinion data. In developing the key informant and case study interview samples, the number of stakeholders who had broad enough exposure to SPOR and its core elements was limited. During the design phase of the evaluation, it was determined that the individuals holding the broadest knowledge of SPOR and all its core elements included only those individuals involved in SPOR’s governance structures and those involved in operationalizing SPOR’s core elements. The opinions and perspectives of these sample groups may be somewhat biased due to their closeness to the development and implementation of SPOR. However, the possibility of this limitation is believed to be minor as the study team acknowledges the open, direct, and candid nature of the interviewee’s responses.

Absence of comparison groups. The evaluation design did not include a comparison group – either of those not engaged in SPOR or of other initiatives. The evaluation focused at the strategy level and in most instances CIHR is one step removed from direct users. A survey (or other far reaching activity) was not possible due to limited ability to identify all individual connections to all SPOR elements, in addition to the budget and timing allocated to the evaluation to enable the identification of all such connections (particularly the research community). However, through the key informant interviews and case studies all stakeholder groups were represented, including researchers and clinicians in the SUPPORT Unit case studies and patients in the PE case. Note also that many individuals involved in designing and implementing SPOR are also either end-users themselves, or representatives of end-user groups.

The evaluation does not include a comparison with another initiative due to the uniqueness of SPOR (large and nationally based). However interviews with international stakeholders were conducted to explore how other countries are approaching initiatives with similar objectives were undertaken.

Validity of self-reported data. The evaluation reports the observations and opinions of the individuals contacted. Within study resources, and because SPOR is so new, the accuracy of opinions cannot be confirmed at this time (e.g., it is too early to really determine if all needs are being met). However, most of the individuals interviewed are very experienced in their respective domains.

Early stage. The analysis is mainly limited by the very early stage of the operations of SPOR. The evaluation does not cover the overall added-value of SPOR or the coherence across the SPOR elements (SUPPORT Units and Networks in particular) due to early and varying stages of implementation (funding) and operationalization. Thus, the interview findings should be taken as early considerations that may affect the future design and delivery of the SPOR elements and how they interact with other SPOR elements.

Internal validity. The analysis of performance reporting is limited by challenges in interpreting the information in performance reports and confirming the internal validityFootnote 25 of the data, particularly in separating out existing networking, collaboration, and cooperation around patient oriented research topics and/or evidence-based healthcare that pre-existed the creation and implementation of SPOR. The evaluation scope, timeframe and allocated resources for the study did not allow for mitigation of this effect in its design.

Ratings. The results of the rating questions used within the interview guides are likely not fully replicable as presented in the report, mainly because the early stage of development often meant that respondents were uncertain how to answer. In some cases, there are relatively few respondents who provided ratings. Thus, small changes in the number of respondents in a given rating category would, in many cases, significantly change the apparent distribution of responses and the average ratings, easily leading to misinterpretation and/or over-interpretation of findings based on these quantitative data.

Quotes. Some anonymized quotes are provided to illustrate the perspective of respondents. These have been selected to be representative of commonly-held opinions, or in a few cases, less common opinions that may represent unique and important insights. The report has indicated which line of enquiry the quote is aligned with and in some cases the position of the individuals quoted. However, because of the group nature of some interviews with SUPPORT Unit EDs and component leads, it was not always possible to determine this – in these cases the quotes are attributed to “ED or component lead”.

Basic form of network analysis. The data compiled for the network analysis were limited to gathering publicly available information and data available through documentation provided by the SPOR program that reported on key SPOR component governance and administrative structures. The data collected do not include all participant information at the operational level (e.g., researchers accessing SUPPORT Unit services, patients involved in projects). Therefore, the data do not represent the SPOR network in its entirety.

The analysis applied to the SPOR governance structure network is basic in its form. This is a limitation of the scope of the evaluation, the data available, as well as the software used for analysis. As the data used for analysis are not complete (i.e., not all relationships were captured), more extensive mathematical measures (underpinned by graph theory) would not produce valid results and would skew the interpretation of findings and therefore were not completed. As a result, the data and analysis presented in this report should not be interpreted to alone explain any of SPOR’s outcomes.

Cost-efficiency analysis limitations. Given the nature of SPOR, which is a large-scale strategy composed of numerous elements rolled out at different points in time and supported through unique design features described above, there is not a suitable comparator within CIHR against which to benchmark the operational efficiency proportion so no conclusion as to the adequacy of the cost-efficiency ratio has been provided.

The grants and awards expenditures and the operating and maintenance costs were collected at different points in time, though within the same Fiscal Year. Operating and maintenance costs for the core elements were challenging to capture due to re-organization of human resources within the portfolio where SPOR is managed.

For the applicant level partnership contributions, as noted above, the data were pulled from applications and, as such, reflect commitments confirmed at that time as opposed to actual commitments made through the full course of the investment, however, the program monitors and validates these commitments on an annual basis. As a result, there is the potential that applicant level partner contributions could be an underestimate or an overestimate of those actually made. Further, for those programs where partnership matching was not a requirement, the partnership data were not consistently entered into CIHR’s funding database across programs and/or over time. As a result, partner contributions to these programs could be an underestimate due to data entry inconsistencies.

High heterogeneity. This limitation is at the level of the individual SUPPORT Unit element. Each SUPPORT Unit represents a different response to the healthcare needs and pressures within its own provincial/territorial environment, including: different levels of existing POR initiatives; different levels of baseline capacity in the research, clinical, and policy communities; and varying levels of existing interactions among researchers, service providers, provincial/territorial healthcare authorities, policy makers, patients, and other stakeholders. Where findings appear to be consistent across most or all SUPPORT Units, they are discussed jointly; where they appear to differ significantly across different SUPPORT Units, some discussion of differences is provided – most of the differences relate to the size and existing research and clinical capacity of each province/territory. Note that while there are a number of common aspects, for many aspects of design and implementation, these SUPPORT Units have faced quite different challenges (and opportunities) to date as a result of the variability across provincial/territorial contexts. As a result, SUPPORT Units yet to be implemented will need to interpret the relevance of best practices and lessons learned in relation to their own context. This limitation also applies to the ACCESS Open Minds and PIHCI networks, being very different initiatives, where lessons learned within these two instances may not translate well to future Networks in development.

3. Evaluation Findings

3.1 Relevance

  1. To what extent does the research funded under SPOR address the need for evidence-informed health care?
  2. To what extent is SPOR aligned with federal roles and responsibilities?
  3. To what extent is SPOR aligned with federal government and CIHR priorities?

3.1.1 Key Findings:

3.1.2 Analysis

3.1.2.1 Addressing the need for evidence-informed health care (Evaluation Question #1)

The nature of the ongoing need of evidence-informed health care has been long-established in Canada. As early as 1986, the federal government has highlighted evidence-based, patient-oriented, and pan-Canadian approaches to research.

In 1984, the federal government replaced two previous acts with the Canada Health Act, a law that set pan-Canadian standards for hospital, diagnostic and medical care services.Footnote 26 In 1986, the Canadian government played an important role in establishing the globally influential Ottawa Charter for Health Promotion. This declaration focused on the process of enabling people to increase control over, and to improve, their health.Footnote 27 Additionally, through a review of the 2004 Health Accord, the Standing Senate Committee on Social Affairs, Science and Technology was notified that the development of benchmarks was not sufficiently evidence-based or patient-centered. The Committee made various recommendations, inclusive of the development of strategies to address such evidence-based and patient-centered pan-Canadian benchmarks.Footnote 28 Further, patient-centered research has remained a topic of discussion for various federal agencies, inclusive of Health Canada, the Public Health Agency of Canada, as well as CIHR.

Trends in public opinion polls indicate a view for a need for Canadian health care reform, as well as demonstrate support for patient-oriented approaches to research. These results indicate that, at a minimum, there is a perception that reform should be undertaken, research is perceived to be an integral component of health care, and that patients would like to be engaged in the research. In a survey conducted in 2010, 61 percent of the patients sampled identified that the Canadian health system was in need of fundamental reforms or to be rebuilt completely, which was lower than in Australia (75%) and the United States (68%) but higher than France (58%), Sweden (53%) and the United Kingdom (37%).Footnote 29 In another report, 90 percent of Canadians have said that health and medical research makes an important contribution to health care and 70 percent of Canadians have said they are interested in participating in health and medical research, though only 24 percent reported being aware of opportunities to do so.Footnote 30

There remains an ongoing need for Canada to stay current in terms of health care and health research approaches in order to address national needs and to be competitive within the global market. Also, an international trend towards evidence-informed health care and patient-oriented research provides Canada with the ongoing requirement to remain relevant in this field.

In order to gain insight regarding the perceptions surrounding the evolution of patient-oriented research as a model on an international scale, international interviewees contacted as part of this evaluation were asked for their views on the context for patient-oriented research evolving internationally. Generally, it was felt by all international interviewees that there is a strong trend internationally towards patient-oriented research models. A few countries are perceived as being “quite vibrant” in moving towards this trend, including the UK, Canada, the US, Denmark and Australia. All interviewees identified CIHR SPOR as playing an integral role in Canada’s position in support of implementing a patient-oriented approach to research.

The PE case study also identified how successes internationally in patient-oriented research have only served to strengthen the need for such a national strategy in Canada. Indeed, various studies have supported the theory that patients who are active as participants in the health care decision making process results in enhanced health care outcomes at lower costs than less active patientsFootnote 31. The PE case study additionally identified that engagement of patients within the research process contributes towards an enhanced ability to translate research into practice, as it helps to ensure uptake by the end user – the patient. Experiences internationally would serve to support this theory as well, with considerable evidence suggesting that PE can improve patient experience and satisfaction and also can be effective clinically and economically.Footnote 32

Documentation reviewed from international sources supports the findings in the PE case study. A few examples of international support towards patient-oriented research and evidence-informed decision making include, but are not limited to, the following:

All key informant interviewees acknowledged the need for change in the manner in which health research is conducted in Canada and the valuable opportunities available for continued development. The common areas of need being addressed by SPOR were expressed by most interviewees as:

The extent to which SPOR elements are addressing key needs varies by area. All key informant and case study interviewees responded to this question within the context of the current stage of each SPOR element, identifying that it is “still early days for SPOR.” Exhibit 5 depicts the results of all interviewee responses to the rating question.

Exhibit 5 – Extent to which SPOR’s core elements meet key needs

Exhibit 5 long description
  Not at all
(1)
To some extent
(2)
To a moderate extent
(3)
To a great extent
(4)
To a very great extent
(5)
(η) Average Rating
SUPPORT Units
avg rating = 3.3
n = 43
1 5 22 11 4 43 3.3
Networks
avg rating = 2.9
n = 20
1 6 9 3 1 20 2.9
Capacity Development
avg rating = 2.4
n = 16
3 7 2 4 0 16 2.4
Patient Engagement
avg rating = 3.3
n = 24
0 5 10 7 2 24 3.3
Clinical Trials
avg rating = 2.7
n = 18
0 10 4 3 1 18 2.7
Not at all (1) To some extent (2) To a moderate extent (3) To a great extent (4) To a very great extent (5)
SUPPORT Units
avg rating = 3.3
n = 43
2% 12% 51% 26% 9%
Networks
avg rating = 2.9
n = 20
5% 30% 45% 15% 5%
Capacity Development
avg rating = 2.4
n = 16
19% 44% 12% 25% 0%
Patient Engagement
avg rating = 3.3
n = 24
0% 21% 42% 29% 8%
Clinical Trials
avg rating = 2.7
n = 18
0% 55% 22% 17% 6%

*Rounding has been forced to add to 100%.

*Average rating has been calculated as a standard weighted average based on the number of respondents.

The SUPPORT Units and PE elements have the highest average rating, with the majority of key informant and case study interviewees rating these elements as meeting needs between a moderate to great extent (average rating of 3.3).

Key informant interviewees identified SUPPORT Units as being in various stages of implementation across the country and most interviewees were unsure of the extent to which needs were being met. However, all SUPPORT Unit respondents believed their SUPPORT Unit was designed and operating to meet needs very well due to their explicit mapping of SUPPORT Unit initiatives against provincial/territorial priorities and needs. Many SUPPORT Unit case study participants noted that, although the concepts of PE and POR aren’t new, the SUPPORT Unit provides a formal place to go where stakeholders can talk about it and ask questions. Many respondents noted that this ability is very new and is in large part due to SPOR.

The PE case study identified the PE element as integral to assisting SPOR in engaging patients at all levels and in all phases of research to improve outcomes of health care systems across the country. PE case study interviewees recognized the PE element of SPOR as setting the tone for defining PE within research, both within SPOR and across Canada, as researchers become increasingly familiar with PE through SPOR grant applications.

The SPOR Networks and the Clinical Trials environment were rated as meeting needs to just around a moderate extent, with aggregate average ratings of 2.9 and 2.7 respectively. Key informants were mostly unsure of the extent to which the Networks were meeting needs, while the case study identified ACCESS Open Minds topics (i.e., mental health) to be an area in significant need for large, multi-faceted, initiatives to demonstrate how youth mental health care could be transformed. The Network case study respondents noted that necessary key activities are being undertaken at present, however implementation has been slower in some areas than expected, and there is considerable site-by-site variation that must be overcome. Respondents to the PIHCI survey noted that PIHCI is intended to address both national and in-province/territory needs, e.g., cross-jurisdictional needs identification and learning; integration with clinicians and policy makers; regional linkages across academia, clinicians, healthcare, and policy makers; and developing common languages among sectors. PIHCI addresses the diversity across the country and scale of primary care, with the opportunity for secure and shared data. Meeting needs in the Clinical Trials environment was identified through the case study to be mainly centred on the progress being made in the CCTCC, although it was noted that needs would not be fully met through this entity alone.

CD is the SPOR element identified as meeting needs to some extent (aggregate average rating of 2.4); lower than any of the other elements, and the area where the greatest number of interviewees identified this element to be not meeting needs at all. The CD case study identified three core areas of need including: enhancing patient participation in research, increasing the engagement of researchers conducting PE, and, supporting the development of careers regarding patient-oriented research. These core areas were seen to be lagging behind in implementation and it was identified through the key informant interviews that some key stakeholders were missing from the table to be able to move it forward. Key informant interviewees did acknowledge the release of the SPOR CD Framework but did not identify outcomes in this area to be sufficient beyond this. It was perceived that there is no plan in place to move CD forward and that there has not been sufficient work with partners (e.g.., the universities). Key informant interviewees identified a further need to undertake targeted activities for capacity development to increase understanding and drive action in this area. It appears it is not well known what the SUPPORT Units are doing in this area and there is an expectation that SPOR needs to be doing more.

3.1.2.2 Roles and responsibilities (Evaluation Question #2)

Though the federal government enacted the Canada Health Act in 1984 to set pan-Canadian standards for hospital, diagnostic and medical care services, there continue to be emerging health challenges across the provinces and territories. In Canada’s Strategy for Patient-Oriented Research, published in 2011, noted that such challenges speak to the need to develop a fulsome, collective strategy at the federal level in order to move forward.

SPOR is clearly aligned with the mandated role and responsibilities of CIHR in its founding instrument, the Canadian Institutes of Health Research Act, assented 16 years ago in April 2000. The Act states CIHR’s main objective is to “excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system.”Footnote 36

The federal Minister of Health is responsible to Parliament for CIHR and the government has a stated objective of making Canada one of the five leading health research nations in the world.Footnote 37 Health Canada provides evidence-based, authoritative information to Canadians and key stakeholders, including health professionals such as physicians, pharmacists and natural health practitioners, to enable them to make informed decisions. Health Canada’s emphasis on evidence based information echoes CIHR SPOR’s focus on evidence based research. SPOR has been referenced directly within federal level priority statements, such as those delivered in Canada’s Economic Action Plan, and CIHR Departmental Performance Reports (DPRs), Reports on Plans and Priorities (RPPs), and Strategic Plans. Additionally, patient-oriented initiatives have been reflected throughout various federal priority statements, such as within Health Canada and Public Health Agency of Canada (PHAC) objectives, thus demonstrating the federal commitment and role in supporting patient-oriented and evidence-based approaches.

“[CIHR is the] only national group trying to put solutions to this [patient-oriented research].”

KI Interviewee

All key informant interviewees and case study interviewees stated that it was critical CIHR play a role in advancing patient-oriented research. Many suggested existing efforts would not be present if CIHR were not supporting SPOR, not only financially, but also from a structural and coordination perspective. Pockets of POR activities may have been active, but not to the national extent triggered by SPOR. A number of reasons were cited for the criticality of CIHR’s involvement, including:

In recognizing the role of CIHR, a few key informant interviewees also identified the importance of provincial/territorial involvement and buy-in as being necessary as well, as they are the ones managing the respective health care systems.

3.1.2.3 SPOR’s alignment with government priorities (Evaluation Question #3)

In October 2013, the President of the Treasury Board approved a new Program Alignment Architecture (PAA) for CIHR that took effect April 1, 2014. The PAA consists of one Strategic Outcome and three Programs (including Internal Services) that support the Strategic Outcome. Within the updated PAA structure, SPOR objectives have been identified predominately within “Priority 2: Priority Driven Health Research.” The goal of this program is to advance health knowledge and its application, in specific areas of research identified by CIHR in consultation with other government departments, partners and stakeholders, in order to improve health systems and/or improve health outcomes in these priority areas.

Additionally, the over-arching strategic objective for CIHR did not experience significant changes through the update to the PAA structure, with the objective being for Canada to be a world leader in the creation, dissemination and application of health research knowledge.

The mandate of CIHR as reflected in its Strategic Plan 2014-15 to 2018-19 – Roadmap IIFootnote 38, particularly pertaining to knowledge translation, is “improved health for Canadians, more effective health services and products and a strengthened Canadian health care system.”Footnote 39 This is aligned to, and complementary of, SPOR’s objective to “foster evidence-informed health care by bringing innovative diagnostic and therapeutic approaches to the point of care, so as to ensure greater quality, accountability, and accessibility of care” which was first identified in 2010.

The CIHR Strategic Roadmaps have undergone various iterations, but have remained highly aligned throughout the years. The Roadmaps consistently set out a high-level vision with relatively consistent strategic directions and supporting objectives. Key areas notable for SPOR are depicted in the Exhibit 6.

Exhibit 6 – CIHR’s Strategic Roadmap 1 and 2

Most recently the Report of the Advisory Panel on Healthcare Innovation (the Naylor Report) identified three priorities for innovation including PE and empowerment, health systems integration with workforce modernization (research to practice), and technological transformation via digital health and precision medicine (right treatment, right person, right time). The Report specifically identifies SPOR as synergistic with some of the objectives delineated in its report.

“SPOR is complementing discussions in the provinces on how do we do things differently. For example, efforts in the clinical trials area. How do we really change the environment to help entice more activity back to Canada? What SPOR brings to the table complements what jurisdictions are already embarking on."

KI Interviewee

Additionally, and key to SPOR’s approach, is the alignment to provincial/territorial objectives and priorities in health care. The concepts of PE, POR, and evidence-based healthcare are not new, and there are other initiatives within Canada that address them to some degree. Most key informant interviewees and interviewees across all case studies noted that SPOR is complementary to provincial and territorial initiatives in patient-oriented research and there is joint interest between federal and provincial/territorial priority areas. The creation of synergies and focal points to address priorities and improve the conduct of health care delivery were key reasons. In part through leveraging and connecting with existing provincial/territorial organizations already working on delivery of programs and policies that will improve the conduct of health care delivery. It was also cited that SUPPORT Units, in particular are aligned with provincial/territorial efforts, through their extensive initial needs assessment exercises and ability to build services in a flexible manner aligned to provincial/territorial needs.

3.1.3 Conclusions

SPOR is relevant to the Canadian health care system and is responding to key needs intended to drive research into practice with the main objective to impact health outcomes of Canadians and enhance patient experiences in health care systems across the country. SPOR is “in play,” being part of the international trend towards employing patient-oriented research models, and is perceived to be one of the “quite vibrant” initiatives being implemented. SPOR is aligned with Canadian public policy, in the government’s long standing drive for patient-centered research, and measures up to Canadian public opinions (i.e., as measured by Research Canada) that research, supported by government efforts, is an important contributor to improving health outcomes in which the public has a desire to be involved.

The definition of the core elements of SPOR is complete and relevant at this time; meeting needs as requirements have evolvedFootnote 40. Each element of the strategy has been designed to meet key needs in areas of increasing collaborations among stakeholders, enhancing access to common resources, providing support for research that can be scaled up and translated across sites and regions, facilitating patient involvement in research, enhancing the clinical trials environment and building capacity around patient-oriented approaches with both researchers and patients. The extent to which SPOR is meeting needs is highly influenced by the current early state of implementation of each element.

There is clearly a role for CIHR to play in SPOR. First and foremost, SPOR is a direct extension of the Canadian Institutes of Health Research Act. The Act states CIHR’s main objective is to “excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system.” Full support for CIHR’s involvement was furnished by all categories of stakeholders, from researchers to policy makers, acknowledging its logical place in delivery as the national health research funding agency in Canada and its international and national credibility. Successful coordination with the provinces and territories was believed to be due to CIHR’s national structure and ability to bring commonality in design and delivery. In addition, respondents noted SPOR’s complementarity to provincial and territorial initiatives in patient-oriented research as there is joint interest between federal and provincial/territorial priority areas. The creation of synergies and focal points to address priorities and improve the conduct of health care delivery were key strengths for the successful coordination of SPOR with pan-Canadian priorities CIHR’s integral role is also acknowledged by the international community.

3.2 Design and Delivery

  1. To what extent has SPOR been implemented as planned?

3.2.1 Key Findings

3.2.2 Analysis

3.2.2.1 Design and delivery at the strategy level (Evaluation Question #4)

SPOR was initially launched in 2011, with release of the formal Strategy document. In the five years since that time, each SPOR element has been developed and implemented within varying timeframes. There were significant efforts undertaken by CIHR to plan and operationalize the SPOR vision and activities that would support its objectives (i.e., bringing the theory of SPOR into practice), as follows:

According to the data collected on the CIHR grants and awards expenditures on SPOR, it took three years after launching the strategy in 2010-11 to begin expending funds to the SUPPORT Units and Networks, with a gradual ramp up of funding over the 2013-14 through 2015-16 period (see Exhibit 21 in the cost-efficiency section). Official program documents indicate there was a planned delay through the first two years of SPOR with activities identified for 2011-12 to include the establishment and engagement of SPOR’s National Steering Committee, the determination of priorities, the creation of the funding opportunities for the Networks and SUPPORT Units, as well as some specified work in the Clinical Trials area. The unplanned delays are further articulated in section 3.3.3.

Key informant interviewees specifically mentioned the federal-provincial/territorial engagement, cooperation and collaborations, including the buy-in of provinces and territories to the overarching objectives of SPOR, as key to the current successes of SPOR. Key informant and case study interviewees identified a number of elements designed well and delivered in an effective manner with the implementation of SPOR. For the most part, process and structural areas were rated to be effective to a moderate extent or higher with the exception of one area – performance measurement.

The results of the effectiveness ratings, by area, are presented in Exhibit 7 and further elaborated with discussion of each area following the chart.

The areas rated highest, on average, for effectiveness include SPOR governance structures, SPOR engagement of stakeholders, and delivery of SPOR funding. These three areas were seen to be effective to a great extent by respondents.

Performance measurement was rated lowest, viewed to be effective to almost a moderate extent on average, and was the one category with the greatest number of respondents identifying that the performance measurement area was not effective at all.

Exhibit 7 – Extent of effectiveness in delivery of processes and structures – Strategy level

Exhibit 7 long description
Not at all (1) To some extent (2) To a moderate extent (3) To a great extent (4) To a very great extent (5) (η) Average Rating
Governance structures
avg rating = 3.6
n = 41
0 8 11 12 10 41 3.6
Engagement of stakeholders
avg rating = 3.6
n = 48
0 9 11 16 12 48 3.6
Delivery of funding
avg rating = 3.5
n = 39
0 10 9 11 9 39 3.5
Coordination of communications
avg rating = 3.1
n = 43
0 13 16 10 4 43 3.1
Performance measurement
avg rating = 2.7
n = 28
3 9 10 5 1 28 2.7
Risk Management
avg rating = 2.7
n = 19
1 8 6 3 1 19 2.7
Implementation of peer review
avg rating = 3.4
n = 24
0 6 5 10 3 24 3.4
Not at all (1) To some extent (2) To a moderate extent (3) To a great extent (4) To a very great extent (5)
Governance structures
avg rating = 3.6
n = 41
0% 20% 27% 29% 24%
Engagement of stakeholders
avg rating = 3.6
n = 48
0% 19% 23% 33% 25%
Delivery of funding
avg rating = 3.5
n = 39
0% 26% 23% 28% 23%
Coordination of communications
avg rating = 3.1
n = 43
0% 30% 37% 23% 10%
Performance measurement
avg rating = 2.7
n = 28
11% 32% 36% 18% 3%
Risk Management
avg rating = 2.7
n = 19
5% 42% 32% 16% 5%
Implementation of peer review
avg rating = 3.4
n = 24
0% 24% 21% 42% 13%

*Rounding has been forced to add to 100%.

* Average rating has been calculated as a standard weighted average based on the number of respondents.

SPOR Governance

The SPOR governance structures received one of the highest average ratings. The main characteristics of governance effectiveness were noted by key informant interviewees to be the national coordinating body with lateral structures across provinces and territories for inter-jurisdictional learning and the corresponding structure that includes representatives external to CIHR (i.e., a “lets discuss together” approach). The buy-in of the provinces and territories was noted to be critical in moving SPOR to the position it is now. Most key informant and case study interviewees noted that SPOR did have the right people in the right places for effective governance.

Through the CT and SUPPORT Unit cases, CIHR staff were noted to be supportive and open, and viewed CIHR as striking a balance between being hands-off in the involvement of the work conducted by each of the SPOR elements while maintaining support for the elements.

“SPOR has lots of hierarchy and needs more coordination. There are 5 separate ships going in separate directions, then seven or eight working groups, we don’t know what they all do."

SUPPORT Unit Executive Director

However, many of the case studies (CD, PE and CT) found there was a perception for the need to strengthen a more national approach to governance in support of all elements to enhance alignment, integration and limit duplication, in particular for the SUPPORT Units and Networks. Recognizing there are mechanisms in place that are intended to enable cross-jurisdictional exchange and collaboration (e.g., the SSUC and its working groups), the case studies identified that direction at the national strategy level could be enhanced with clear central leadership. One specific example provided in the PE case was that the SPOR SUPPORT Unit Council PE Working Group is still in the process of defining itself, and would require increased clarity in definition moving forward. A lack of consistency in approaches to PE was noted across the SUPPORT Units, and it was suggested a stronger national approach would be beneficial to accompany the SPOR SUPPORT Unit Council PE Working Group. It was also identified through the SUPPORT Unit case study that the roles and actions of the SUPPORT Units versus the SSUC Working Groups were not well defined. Interestingly, the international key informants also identified that with SPOR having SUPPORT Unit operations governed at the provincial level, there may not be enough interaction at the federal level. Each of the international comparative initiatives are more targeted towards a national POR approach.

Another area identified as having potential for enhancement is the engagement of patients across the various levels of governance. This was identified in the PE and CT case studies and is consistent with findings in the performance area (see section 3.3.1.1), where it is noted there is a need for further efforts to enhance PE across the various levels of governance. This includes support for patients to engage as equal stakeholders at these levels, inclusive of governance training.

This perception, whether accurate or not, is supported by the results of the governance and administrative network analysis. The patient component of the Network, albeit connected to many of the governance bodies, is one of the smallest pieces of the structure. The network analysis was based on publicly available information and even though does not paint a complete picture of SPOR, does imply that the communicated patient involvement may be misunderstood.

Engagement of stakeholders

All key informant and case study interviewees noted very extensive efforts in SPOR’s engagement of stakeholders, identifying SPOR as being effective in engaging a broad group, from the public, private, and academic sectors. Particularly in the CT and PE cases, respondents noted that “all kinds of efforts” have been made at both the national and provincial/territorial levels. Additionally, the ability to bring various stakeholders together at the same table is an asset of the CCTCC that was noted to be unique within Canada.

Further elaboration on the engagement of stakeholders is documented in the section on Performance (see section 3.3.1.1).

Delivery of funding

Key informant interviewees identified that the requirement to secure matching funding contributed to the variation in timing of establishing some of the SPOR elements, particularly the SUPPORT Units.

SPOR funding was seen to be timely and flexible where required. Both the SUPPORT Unit and Networks case studies identified initial negotiations were required to determine categories of eligible costs, as the SPOR model is outside the bounds of traditional CIHR eligibility with an increased need for operating and service delivery funds. However, there is a concern in the CT area where it is not seen that SPOR will cover all expenses associated with running clinical trials. Additionally, the SUPPORT Unit case study identified that funds were said to be intended to support infrastructure and there are no operating funds, so the SUPPORT Units are sometimes searching for funding for specific projectsFootnote 41. Given the perceived high competition rates for CIHR Open Grants if researchers have no connections with large industry (e.g., pharma) to find matching funding, the SUPPORT Unit case study identified funding as a significant challenge.

Through the Networks case, it was identified that consideration should be given to the timing for distribution of funds. CIHR’s funding model assumes that at the moment funding begins, participants already have the required infrastructure and delivery models in place, neither of which was true in the case of ACCESS Open Minds – some ramp-up period is required. Key informant interviewees also identified the ramp-up activities required to establish infrastructure and new corporate entities.

The biggest issue raised around funding is the uncertainty surrounding the ability of CIHR to continue to provide financial support for SPOR, and hence the sustainability of partner funding was also raised, despite there being a requirement for the SUPPORT Units, in particular, to develop sustainability plansFootnote 42. The five year funding term was a concern in the CT, PE, and SUPPORT Unit cases. Key informant interviewees also shared this viewpoint. Key informant interviewees suggested the short timeframe to build sustainability around the concepts and results associated with SPOR was considered to be the biggest challenge. For example, if CIHR funding was to disappear at the end of the funding term, would the focus on patient-oriented research and conducting research in this different way still be embraced? This was acknowledged to be a challenge not only in regards to sustaining a major paradigm shift in how research is done, but being able to do this within a very short period of time. Concerns were also expressed around the risk of escalating costs (i.e., as POR takes more time and resources) or decreasing funds and the ability of funding partners to maintain the levels of financial commitment required. None of the provincial government respondents interviewed suggested that their ministries would provide 100% funding for SPOR initiatives, rather than the 50% now being provided. Some SUPPORT Units are already trying to build various kinds of fee-for-service models into their programming, e.g., for training, but this is considered unlikely to be able to maintain SPOR momentum. The issue around the short funding term was also noted to potentially impact the adoption of POR approaches due to uncertainty of the sustainability and longevity of support.

Coordination of communications

Issues around the coordination of communications was identified by the KI interviewees, the PE, Networks, CT and SUPPORT Unit cases.

“Communications is a big pet peeve – the initiative is not well communicated or well understood – people have a “blank stare” when you talk about SPOR.”

KI Interviewee

Communication of concepts and results were perceived to be highly important and noted to take dedicated resources and efforts to accomplish. Knowledge translation, best practice dissemination, and sharing of successes were noted to be required to keep momentum moving and for SPOR to achieve its greatest potential. Without an enhanced communication structure effectively outlining SPOR objectives and deliverables, and success stories of POR or PE achievements there is a risk of misunderstanding “what SPOR is, and what SPOR is not.” Further, it was noted there is a need to clarify definitions, such as that of PE and patient-oriented research, to reduce confusion and contribute towards enhancing adoption within all stakeholder communities, including the public. Lessons learned included the tailoring of communications towards target audiences (also noted by the US international interviewee), and potentially enhancing website and social media communications by CIHR and the SUPPORT Units and NetworksFootnote 43. Respondents identified the need for consistency of communications and messages from the many possible national and provincial/territorial sources was critical.

“[We’d love to see an announcement for a webinar for each platform, each platform lead could talk about what they could do for [researchers] . . . more information, how/ when/ where – and costs would be useful.”

Researcher

Some key informant interviewees and the SUPPORT Unit interviewees also noted the increased need for communication with stakeholders by SUPPORT Units. It was identified that SPOR could help to make SUPPORT Units better known through increased engagement with the research community, including through the engagement of foundations, research institutes, and charities.

The international interviewees noted that convening the community and a need to maintain visibility of the POR agenda is one of the most important elements. Bi-annual conferences are held in the UK and were noted to be getting bigger and more diverse in the mix of attendees participating. As the UK is also structured with a mix of many entities focused on patient-oriented research (or parts of it), it was identified that they attempt to inform participants/stakeholders as to what one another are doing to ensure some consistency – particularly in relation to training – but did not elaborate on how they do this.

The US initiative noted that it spends a lot of time on communications and particularly dissemination of results. The US interviewee identified that researchers are typically not necessarily the best disseminators, hence the importance of integrating stakeholders into the research, so ownership is established and more enthusiasm for dissemination of results is generated. Having partnerships with organizations that reach patients, clinicians, and health organizations is important as it is these that reach the decision-makers.

Performance measurement

A performance measurement strategy does exist for SPOR as a whole. Linked to this is a common performance framework for the SUPPORT Units with annual performance reporting requirements as well as performance reporting requirements for ACCESS Open Minds.

Although the program stated that progress reporting requirements were developed and refined collaboratively through consultations with partners and funding recipients, issues with current performance measurement processes, tools and targets were consistently identified across the key informant interviews and the case studies, including the SUPPORT Unit, Network, CT (specific to the CCTCC) and PE cases. The issues around performance measurement were identified by individuals who receive the reporting (advisory members and funders), as well as those required to submit the reporting (SPOR element leads). There is a need to continue to develop relevant performance measures.

The key informant interviewees perceived a lack of attention to specific outcome performance measures and the uncertainty surrounding what is required and expected in the longer term. In addition the varying requirements of all partner funders was an issue. It was noted there is no consistency in requirements or expected outcomes across partners which was identified to be quite burdensome for the SUPPORT Units in particular. The SUPPORT Unit and Network case study support this viewpoint with the SPOR performance metrics being identified as extensive and onerous to complete. Few SUPPORT Unit case study respondents believed it was effective in measuring outcomes and impacts, and several respondents commented it was very difficult to understand exactly what the SUPPORT Units and SPOR were accomplishing, especially given other non-SPOR PE and POR initiatives within the provinces/territories. In addition to the workload, a few respondents commented that the static “activity” metrics will not contribute to SPOR’s sustainability as they will not prove the value of POR/PE approaches, as opposed to a scientific approach to measuring what works best and when.

One specific area identified through the PE case study is a need for detailed evidence of the use of the PE framework by SUPPORT Units supported by proof of meaningful engagement of patients to better inform the measurement of the progress and impacts of the initiative. As an example, it was suggested a measure of patient opinions on levels of engagement in preparing applications for research grants could be considered.

The importance of robust performance measurement framework was highlighted by the international key informant interviewees as well. All international interviewees highlighted the importance of ensuring outcome driven performance measures, set in a way to encourage alignment of the initiatives to delivery on outcomes. Each international interviewee felt it was integral to report on the impacts and the overall progress of the initiative. As an example, performance measures under CLAHRC are viewed as being directed towards delivery of their research results. The metrics serve as quality drivers for the translation of research through application to customer uptake (based on public/patient opinion). One key area of improvement for SPOR, as identified by international interviewees, was to ensure data being captured includes applicants, reviewer comments, milestones, and progress to date against plan. All international interviewees perceived SPOR as not clearly defining the impacts it intended to make and how these should be measured from the start of the initiative. International interviewees believed SPOR requires enhanced clarity on how it would assess its overall progress to date, as stakeholders need visibility of the agenda in order to maintain the momentum of the initiative and support for it. Further SPOR was encouraged to define measures focused on whether the initiative was “driving better outcomes.”

Consideration for robust performance measurement was a lesson learned in the UK and is still something to be resolved. One international respondent noted that their initiative began without defining what clear impacts it should be measuring. This is an ongoing challenge for the future to be able to answer this question more appropriately, however, the UK has set three measures:

Risk Management

There were very few key informant interviewees or case study findings that further elaborate on the effectiveness ratings provided by respondents. Basically, there was not enough exposure to this area for respondents to comment knowledgeably. For the key informant interviews, it was identified that for the most part, the risk management area was impacted by other processes, such as slowness of implementation, communication and concerns related to the robustness of the performance measurement structure, and generally the uncertainty of how risks are managed.

Implementation of peer review

The engagement of patient stakeholders in the peer review process was seen as positive and a novel approach within the CD, Networks and PE case studies identifying the effectiveness of this process. However, it was suggested in both the CT and PE cases that there is still increased support required for patient representatives to be fully engaged. The need for increased training for patient representatives to assist in developing skills to contribute towards the conduct of peer reviews, including enhancing communications in terms of patient roles, responsibilities, and time commitments required, and engaging peer review personnel through increased in-person meetings would assist with this process.

Through the Networks case study the application and proposal review process were identified to have been transformative in terms of generating interest in starting up parallel projects, and in continuing collaborations among some of the teams that did not receive Network funding. One respondent commented that although this process was time and resource intensive, it provided significant dividends for both successful and unsuccessful applicant teams.

3.2.2.2 Design and delivery of core elements (Evaluation Question #4)

Two SPOR core elements have independent implementation structures with facets of design and delivery separate from the overarching strategy. This section covers the design and delivery elements of the SUPPORT Units and Networks. The design and delivery of the PE, CD and CT elements are included in the discussion above on the implementation of the strategy as a whole as PE and CD crosscut all elements and the CCTCC is a shared resource.

Each of the case studies has identified individual elements of design that are working well in addition to areas of challenge that require ongoing attention. Collectively the case study respondents rated their respective delivery structures as being effective to some extent. It is important to note that all SPOR elements are very young, with the earliest implementation being achieved by four SUPPORT Units established in 2013. All other pieces have been launched subsequent to this, so for the most part have only been operational for one year. As these structures are still evolving, it was difficult for respondents to know how well these governance and administrative structures will function once fully developed.

Exhibit 8 – Extent of effectiveness in delivery of governance and administrative structures – by SPOR element

Exhibit 8 long description
Not at all (1) To some extent (2) To a moderate extent (3) To a great extent (4) To a very great extent (5) (η) Average Rating
SUPPORT Units
avg rating = 3.3
n = 32
1 6 9 13 3 32 3.3
Networks
avg rating = 2.7
n = 6
0 2 4 0 0 6 2.7
Capacity Development
avg rating = 2.0
n = 1
0 1 0 0 0 1 2.0
Patient Engagement
avg rating = 3.0
n = 9
0 4 1 4 0 9 3.0
Clinical Trials
avg rating = 2.5
n = 4
0 3 0 1 0 4 2.5
Not at all (1) To some extent (2) To a moderate extent (3) To a great extent (4) To a very great extent (5)
SUPPORT Units
avg rating = 3.3
n = 32
3% 19% 28% 41% 9%
Networks
avg rating = 2.7
n = 6
0% 33% 67% 0% 0%
Capacity Development
avg rating = 2.0
n = 1
0% 100% 0% 0% 0%
Patient Engagement
avg rating = 3.0
n = 9
0% 44% 12% 44% 0%
Clinical Trials
avg rating = 2.5
n = 4
0% 75% 0% 25% 0%

*Rounding has been forced to add to 100%.

*Average rating has been calculated as a standard weighted average based on the number of respondents.

Each SUPPORT Unit is organized and governed somewhat differently, and in these mostly early days it was difficult for respondents to know how well these governance structures would function once fully developed. However, the SUPPORT Unit components were developed through extensive consultation with stakeholders from all sectors to address key research priorities and/or fill critical gaps in capabilities. Thus, these consultations are considered key to implementing the SUPPORT Units effectively. The consultations have helped to ensure that matching funding has been available, and that local stakeholders find the SUPPORT Unit services and priorities to be in line with their own. Of all aspects of governance and administration, this was by far the most frequently cited as important and effective. This is also true of the PIHCI network, where the phased approach to developing and rolling out the network, which allowed for sufficient time to consult on design and ensure it would meet the needs of the community was noted to be a key strength.

But clearly the SUPPORT Units have taken longer – in some cases, far longer – to be implemented than was planned. No single and consistent reason was offered by SUPPORT Unit case study respondents for this situation, but the attempt to be “all things to all people,” coupled with the complexity of the many national, provincial/territorial, and SUPPORT Unit components, along with the complexities of the university research and provincial/territorial healthcare environments, have all certainly played a role. Simply getting appropriate stakeholders together and agreeing on how to proceed has been very time consuming. Once operational, implementing the budget (both cash and in-kind) among the many funding partners was said to be very complex.

This is not unexpected and confirmed by the experience of the international comparative initiatives where it was noted to be a challenge within the CLAHRC’s to to bring together various stakeholder types, inclusive of patients, researchers, and policy makers, together to work towards a shared vision. Similar to SPOR, it was stated that different types of stakeholders tend to work and think in different ways, leading to some struggles in initially learning how to work together.

Similarly, the governance structures for ACCESS Open Minds are seen by respondents as moderately successful to date, but they are still evolving significantly (and frequently) from what was originally proposed, and all parties are working to make them effective. The ACCESS Open Minds case study respondents found it very difficult to predict how long it will take to get ACCESS Open Minds fully operational. Participants originally thought it would take eight months to a year to “get there.” The Network grant came in the middle of 2014, from which it took 3-4 months to set up the Network, with respondents noting ACCESS Open Minds has been about 11 months in development now and it is still not operating at its intended scale. This is considerably slower than the timeline originally projected.

The main PIHCI governance mechanism (at this point in time) is the tri-partite Interim Leadership Council which has one clinical, one research and one policy representative from each province/territory. Although the PIHCI Network is not operating at its intended scale and PIHCI respondents were not asked explicitly about this topic, some participants made comments that indirectly suggest they believe governance and administration will be effective. For example, they commented that PIHCI builds upon existing community-based primary care research infrastructure in several provinces (e.g., Alberta), and PIHCI can leverage that as an interprovincial collaboration unit. The provincial network integrated into a national network (i.e., network of networks model) is considered by some respondents to be a strong design feature that is necessary to encompass the diversity of primary care and to share and learn from each other’s success and failures.

Understanding the early stage of these elements, learnings from implementation experiences to date can be identified, and in some cases, are shared across the SUPPORT Units and Networks. The biggest implementation challenges noted through the SUPPORT Unit and Networks case include:

“One problem is the interaction among SUPPORT Units and Networks, and between SUPPORT Units – I don’t feel terribly engaged yet. We’re not yet developing collaborative, cohesive programs.”

SUPPORT Unit ED

3.2.3 Conclusions

Some of SPOR’s areas of implementation are on track to becoming effective, but not all. The design of the strategy overall is sound, with national governance mechanisms seen by stakeholders as generally strong and appropriate. Key success factors to effective SPOR governance were noted to be its national composition accompanied by lateral structures across provinces and territories to facilitate inter-jurisdictional learning and the inclusion of representatives external to CIHR, promoting a “let’s discuss together” approach. Stakeholder engagement is seen to be “exceptional” and SPOR has been quite successful at developing relationships and reaching a broad group of stakeholders across sectors and participant types.

Peer review processes are also strong with the inclusion of patient representatives seen to be a positive and novel approach. The extensive and iterative process for Network development and selection was noted to have continuing effects on even those projects that were not selected for funding (i.e., groups not funded are still working together).

The most important early outcome of SPOR implementation is that provincial/territorial priorities and needs were heavily considered in the SUPPORT Unit development process, and this is considered a great strength of the flexible approach CIHR has taken. However, this has impacted the length of time for implementation for the SUPPORT Units and most are still working towards integrating and coordinating all required SPOR components. A dedicated operational leadership position (for the funded Networks and SUPPORT Units, tasked with program/project and administrative management was identified as a best practice and one that is necessary for efficient operations.

However, there are opportunities for improvements in delivery to support the effectiveness of ongoing implementation. Some of the most prevalent opportunities are the requirement for further enhanced engagement of patients within SPOR’s governance structures as well as those of SPOR funded entities and a need to increase a national approach to governance and delivery in support of all elements, but particularly to enhance alignment, integration and limit duplication among the SUPPORT Units.

Communications overall need improvement where further clarification of the mandates for each of the SPOR elements, available support by the SUPPORT Units, consistent and common definitions of POR and PE, and communications targeted to different stakeholders is required. This also includes sorting and communicating how the individual SPOR elements are intended to work together, particularly the SUPPORT Units and Networks, but also applies to how Networks will work with other Networks. It is also not clear to what extent this is intended or expected to occur.

Performance measurement was also consistently identified as needing attention. It is not clear that the right things are being measured (i.e., considerable measurement of activities and limited measurement of outcomes) and the collection and reporting of performance data is perceived to be quite onerous for funded recipients.

There are major concerns surrounding the sustainability of activities and momentum for POR within the initial five year funding timeframe. The 1:1 matching funding requirement also raised concerns with respect to sourcing funds, as many SPOR elements may potentially be approaching the same organizations for funding multiple times, for initiatives which could be seen to be overlapping. As well, there are anticipated conflicts that may be raised by the possible mismatch between provincial/territorial and national priorities and concern regarding how these types of issues may be resolved, let alone funded.

3.3 Performance

  1. To what extent has SPOR made progress toward the achievement of expected immediate outcomes?
  2. To what extent has SPOR made progress toward the achievement of expected intermediate outcomes?
  3. To what extent is SPOR being delivered in a cost-efficient manner?

3.3.1 Key Findings:

SPOR is in the initial phases of implementation with all core elements at various stages of development and operation, but SPOR appears to be on track to achieving most of its expected immediate outcomes:

Although SPOR is believed to be too new by most respondents to have significantly changed the research environment to date through concrete, measurable outcomes, SPOR is making some impact in influencing a change in the POR landscape and changing the research culture. This has been achieved through the alignment with provincial/territorial priorities, increased involvement of all stakeholders in health research, promoting the importance of PE and the establishment of an environment where the focus is on outcomes of importance to patients.

SPOR is now committing funding at full scale to its core components and has leveraged partner dollars, meeting is own requirement for 1:1 matching and in most cases exceeding this ratio. SPOR is refocusing its efforts to increase support to the core elements as planned, evidenced by a significant decrease in funding directed towards Foundational Investments.

3.3.2 Analysis

3.3.2.1 Progress towards Immediate Outcomes (Evaluation Question #5)

SPOR’s six immediate outcomes have been defined in the SPOR Performance Measurement Strategy (PMS) (October 2014) and identifies SPOR will undertake programming that results in:

All key informant and case study interviewees predicated their responses on rating the extent of SPOR’s progress on expected immediate outcomes to the early stage of implementation of all core elements. As identified in Exhibit 9, all of the immediate outcomes have been rated, on average, to being achieved “to a moderate extent,” with the implementation of the SUPPORT Units receiving the highest number of interviewees (31%) perceiving this outcome to be achieved “to a great extent.”

Exhibit 9 – Extent of progress towards immediate outcomes

Exhibit 9 long description
  Not at all (1) To some extent (2) To a moderate extent (3) To a great extent (4) To a very great extent (5) (η) Average Rating
Research platforms
avg rating = 3.2
n = 48
2 9 18 15 4 48 3.2
Stakeholder engagement
avg rating = 3.0
n = 46
0 12 24 9 1 46 3.0
Advancement in methods
avg rating = 2.8
n = 45
1 18 16 8 2 45 2.8
Capacity development opportunities
avg rating = 2.8
n = 39
4 11 13 9 2 39 2.8
Pan-Canadian networks mobilized
avg rating = 2.9
n = 38
1 13 14 7 3 38 2.9
Not at all (1) To some extent (2) To a moderate extent (3) To a great extent (4) To a very great extent (5)
Research platforms
avg rating = 3.2
n = 48
4% 19% 38% 31% 8%
Stakeholder engagement
avg rating = 3.0
n = 46
0% 26% 52% 20% 2%
Advancement in methods
avg rating = 2.8
n = 45
2% 40% 36% 18% 4%
Capacity development opportunities
avg rating = 2.8
n = 39
10% 28% 33% 23% 5%
Pan-Canadian networks mobilized
avg rating = 2.9
n = 38
3% 34% 37% 18% 8%

*Rounding has been forced to add to 100%.

*Average rating has been calculated as a standard weighted average based on the number of respondents.

All key informant and SUPPORT Unit case study interviewees noted the varying stages of implementation of all the SUPPORT Units. Some SUPPORT Units are ahead in implementation while others are still working on establishing administrative and governance structures. This is similar to the perceptions around the implementation of Networks. It was acknowledged that there is one active network (Access Open Minds), another one ramping up (Primary Care) and the next five (Chronic Disease) not yet in place.

Key informant and case study interviewees identified moderate progress in engaging stakeholders when responding to the closed-ended rating question. This moderate rating does not mirror the qualitative elaborations provided. All interview data, supporting performance data submitted by the SUPPORT Units, as well as the governance and administrative network analysis support a very great extent of stakeholder engagement taking place. The SUPPORT Unit case study results have identified that stakeholders have been heavily supported to engage in research and decision-making processes and there are a number of ties between and among researchers, clinicians, policy makers and patients.

However, results from the key informant interviewees and the SUPPORT Unit case study also identified further (and continuing) opportunities for more and better communications to engage stakeholders. There were perceptions that SUPPORT Units are not well known by the research community, and that researchers do not understand what platforms are being established, what services are offered and how researchers can leverage what is being done. There also appears to be more effort required to achieve effective PE.

Advancement in methods and capacity development opportunities were the two outcomes rated the lowest by key informant and case study interviewees. For advancement in methods, again it is the early stage of SPOR that prohibited key informants and case study interviewees in identifying significant advancements in POR being made, however, some examples were provided. Capacity development was the area with the greatest number of respondents (10%) identified that no progress had been made in this area.

Further elaboration on the findings against each of the immediate outcomes follows.

Stakeholders, including patients, are engaged

Virtually all key informant and case study interviewees noted very extensive efforts in this sphere being made both by SPOR at the national level and by SUPPORT Units at the provincial/territorial level. Thus SPOR’s stakeholder engagement activities are seen as strong not only because they ensure the needs of the research, clinical, and policy communities are being met, but that these sometimes siloed communities have increased opportunities to interact with and learn from each other.

The view from international key informants also support this point. Interviewees in the UK viewed SPOR as having been relatively more successful in garnering diverse public engagement than NIHR, and felt that there could be lessons learned for NIHR from SPOR.

A unique blog-based enterprise started by the mother of a young boy with a rare, serious bowel/colon disease was intended to connect with, support, and obtain information from families of other children with this disease. This mother is now a co-investigator on research to investigate the condition, and has engaged with many in the research and clinical communities.

The SUPPORT Unit case study identified participation in cross-sectoral discussions on research and government policy that were nearly unthinkable before, including with stakeholders from other jurisdictions and through the Networks case study, ACCESS Open Minds and PIHCI both have engaged stakeholders extensively in their design.

“Never before in my long career have I been around the table with provincial counterparts from other regions. NEVER. This is transformative.”

Epidemiologist

The total number of personnel reported as actively engaged within the SUPPORT Unit core functions was 2,127 personnel in 2014-2015, an increase from the 409 reported in 2013-2014Footnote 44. At this point the four initial SUPPORT Units launched in 2013 have progressed through their initial set-up and two additional SUPPORT Units are active. All SUPPORT Units active for longer than one reporting period experienced an increase in the number of personnel actively engaged within the SUPPORT Unit core functions over the periods, with the exception one SUPPORT Unit’s Training and CD core function, which reported a decrease from 382 to 62 personnel.

The three most engaged categories of stakeholders, as reported by the SUPPORT Units, are researchers or academics, health system/care practitioners or professionals and patient representatives as shown in Exhibit 10.

Exhibit 10 – Numbers and types of stakeholders engaged with SUPPORT Units 2013 - 2015 (FTEs)

  Patient representatives Researchers or academics Graduate students or fellows Health system / care practitioners or professionals Policy makers Other
Total 423 882.9 211 553.25 137 331.05
Min 0 10 2 0 0 1
Mean 71 147 35 92 23 55
Max 244 553 157 334 108 214

The SUPPORT Unit stakeholders are engaged across all core SUPPORT Unit functions. The main areas of engagement include consultation and research services, health systems implementation research and knowledge translation core functions as depicted in Exhibit 11. Less engagement is occurring in the clinical trials area, methods support and development, and data platform areas.

Exhibit 11 – Numbers of stakeholders engaged with SUPPORT Units by core function 2013 - 2015 (FTEs)

  Patient Engagement 2015 Data Platform 2014 Data Platform 2015 Consultation and Research Services 2015 Methods Support and Development 2015 Real World Clinical Trials 2014 Real World Clinical Trials 2015 HR, Implementation research KT 2014 HR, Implementation research KT 2015 Training Capacity Development 2014 Training Capacity Development 2015
Total 283.08 15 159.8 623.15 143.33 2 29 10 607.75 382 283.1
Min 3 4 4 0 1 2 0 10 3 382 1
Mean 47 8 27 104 24 2 5 10 101 382 47
Max 245 11 92 404 58 2 19 10 481 382 188

The ACCESS Open Minds network also demonstrates stakeholder engagement across multiple categories. Of particular interest is the reported relatively high percentages of youths and families involved at both the governance and site team level.

The governance and administrative network analysis also shows engagement across and between stakeholder groups. The network graph shown in Exhibit 12 depicts all the SUPPORT Units and the PIHCI research network as having a mix of ties to research, clinical and policy areas. The Ontario SUPPORT Unit and the SPOR National Steering Committee are two structures that include ties to all categorized roles.

Exhibit 12 – Numbers of stakeholders engaged with SUPPORT Units by core function 2013 - 2015 (FTEs)

Legend: Dark blue = Research; Light blue = Clinical; Dark green = Policy; Light green = Industry; Red = Charity; Orange = Patient

Exhibit 12 long description

Exhibit 12 is a visual representation of an analysis of the connections between and across SPOR governance and administrative networks. There are dots connected by one or more lines. Each line and dot connection represents a relationship across or between organizations connected to SPOR. The larger dots indicate an organization has many connections to other organizations (represented by smaller dots at the end of a line). The larger the dot, the greater number of connections it has in place. There are larger dots for patients, for each of the SPOR SUPPORT Units, the PIHCI network, the SPOR Working Group, the SPOR SUPPORT Unit Council and the SPOR National Steering Committee. The dots are color-coded to represent different stakeholder groups. Dark blue dots are research, light blue dots are clinical, dark green dots are policy, light green dots are industry, red dots are charity, and orange dots are patients.

The Ontario SUPPORT Unit and the SPOR National Steering Committee are two structures that include ties to all stakeholder groups.

In addition to the available public information on the engagement of patients in the SPOR governance structures, the SPOR program provided further detailed information related to the engagement of patients in the SUPPORT Units. It is a requirement of SPOR that the SUPPORT Units have patients engaged in their governing bodies. The total number of patient seats allocated by SUPPORT Units to patient representatives is 74. This ranges from two seats up through 17 seats depending on the SUPPORT Unit. These patient positions are part of different governance structures, such as a steering committee or advisory council supporting the SUPPORT Unit overall or a specific component area (e.g., capacity development). Through data provided by the program, it is reported that 59 (~80%) of the available 74 seats are filled with the rest in recruitment. Note that one patient can fill more than one seat so these data are not reflective of the number of participants.

However, it was identified that the development of all relevant partnerships and connections would be an ongoing process. Key informant interviewees identified that some key institutions/organizations were not connected to SPOR and that the connection to the research community was not clear. Opportunities were noted to be available to better engage with other key stakeholders (e.g., health charities) already working for better care for stakeholder communities through POR. The PE case study interviewees echoed this finding in identifying a need to increase and enhance collaborations with various external networks and institutions. They responded that various external networks, such as the Patient Voices Network and Patients Canada, could be further leveraged by the PE element of SPOR for the provision of resources and expertise. SPOR was encouraged to continue to determine what resources are already available and which stakeholders can be leveraged going forward.

Equally common was the observation that in some regions there is a long way to go to improve the literacy of policy makers and the understanding of researchers as to how to participate fully in the research decision-making process, as well as, quite importantly, supporting the effective engagement of patients.

Part of the challenge is due to definitional issues. What is stakeholder engagement vs PE? Results from the SUPPORT Units and PE case studies indicate the definitions are not clear. PE can mean any or all of: engagement of patients in SPOR governance, engagement of patients in SUPPORT Units governance and decision-making, engagement of patients in individual initiatives (e.g., research projects, SPOR Networks), and engagement of patients less formally through asking them about research priorities (even if they have no other participation in the research) or satisfaction with clinical processes and outcomes. Each of these different aspects likely requires different engagement mechanisms, has best practices, and has different pros and cons.

The CD, CT and PE case studies identified that enhancements would be required to support the effective engagement of patients within the research and decision-making process highlighting the need to provide training and potentially mentoring to patients to enable them to fully contribute. The PE case study particularly identified the need for enhanced support of patients to further enable them in their roles, identifying potential gaps in supporting the representation of patients in the governance structures. The CT case study also identified a need to further involve patients in governance and increase communication and engagement of patient representatives. The CD case identified that there was more discussion needed on how PE should be conducted as opposed to application and uptake by the research community, thus providing room for improvement as SPOR matures.

Extensive “good will” effort has been reported through the SUPPORT Units and Networks case studies in conducting PE, although with highly variable intensity and success. The issuance of the PE Framework nationally by SPOR in 2014 has been used moderately by all SUPPORT Units, mainly by those individuals responsible for PE. It is either as a model for SUPPORT Units to develop their own PE plans, or to (less likely) to refine PE plans already in place. The ACCESS Open Minds Network is aware of the Framework but have not leveraged it to any great extent, mainly because of unique issues involved with youth.

Interviewees in the Networks case study identified that patient engagement efforts to date were incomplete, not sufficiently planned, not fully representative of patients and unclear as to the extent the patient voice is valued and acted upon. At the same time, difficulties in grasping the best ways to involve and educate patient representatives along with struggles in patient capabilities for participation were also raised.

While many respondents across the key informants and case studies believe that PE needs to be strengthened, it was noted that in order to support PE, considerable work needs to be done as to how to do PE, when to do PE, why do PE, and what capacity building needs to be completed – on both the patient and the researchers/clinician sides. Within the SUPPORT Units case, one PE lead, for example, reported that researchers have yet to come to them with questions or identify requirements for help on recruiting patients as partners in research – although this was a SUPPORT Unit operational for only a year and a half.

The Networks case study identifies significant challenges being encountered in developing appropriate, useful, best practices for involving patients and families in the design, review, and conduct of research programs – these are evolving constantly, and while being based on similar underlying principles, will almost certainly require customized approaches for each individual research situation; e.g., taking into account the experience and resources of both the patient and researcher/clinician communities.

Multi-disciplinary, integrated research platforms established

At the time of evaluation data collection, six SUPPORT Units were established and are all at different stages of implementation. During the course of the evaluation, however, an additional three SUPPORT Units were funded at the end of the 2015-16 fiscal year and out of scope for the evaluation. The first year of operation for most SUPPORT Units consists of planning and development. The second year typically consists of the creation of standard policies, procedures and processes for each core function and the beginning of service delivery. Two SUPPORT Units, Ontario and Manitoba, are the most fully developed.

Exhibit 13 depicts all the SUPPORT Units; those approved, and those under discussion. Once completed, there will be SUPPORT Units across all provinces and territories in Canada.

Exhibit 13 – SPOR SUPPORT Units

Exhibit 13 long description

Exhibit 13 depicts Canada’s map with all the SUPPORT Units: those approved and those under discussion.

The approved SUPPORT Units coloured in green are: British Columbia, Alberta, Saskatchewan, Northwest Territories, Manitoba, Ontario, Quebec, Maritimes and Newfoundland & Labrador.

Two SUPPORT Units are in ongoing discussions, coloured in red on the map: Yukon and Nunavut.

A gold line connects all SUPPORT Units and represents the governance across SUPPORT Units through the SPOR SUPPORT Unit Council.

Through implementation, the SUPPORT Units have been able to establish a number of partnerships that align with the provincial/territorial focus. The majority of the partnerships established are with public funders from government (e.g., the various provincial Ministries of Health), though various SUPPORT Units have also established partnerships with academic institutions, and a couple have maintained private partnerships (IBM and the Cochrane Centre).

The total number of multi-disciplinary and cross-sectoral partnerships being reported across SUPPORT Units in 2014-2015 is 37 (Exhibit 14), up from 17 reported by those SUPPORT Units active during the 2013-2014 reporting period.

Exhibit 14 – Number of multi-disciplinary and cross-sectoral partnerships established, reporting period 2014-2015

  Public Academic Private
Min 1 1 1
Mean 4 2 1
Max 10 4 1
Total 26 9 2

Additionally, there is increased activity in SUPPORT Units core functional areas including the provision of supports and services and work around data platforms.

Data provided through the individual Annual Performance Reports for SUPPORT Units was reviewed for the 2013-2014 and 2014-2015 reporting periods. Key results are summarized as follows:

Indicator Total across 4 SUPPORT Units (mean, range) in 2013-14 Total across 6 SUPPORT Units (mean, range) in 2014-15
Data sets 253 (126.5, 16 to 237) 534 (89, 0 to 252)
Supports and services 376 (94, 43 to 332) 1,872 (312, 0 to 1,183)

On the data platform side, the SUPPORT Units case study identified that many of the provinces are putting high priority on building data platforms, facilitating data access and data linkages, within which researchers and policy makers will be able to access and use a wide variety of relevant provincial data (e.g., potentially including clinical data, healthcare administrative information, prescription data and genomics findings.). Additionally, some SUPPORT Units already include non-health data (e.g., environmental variables, justice statistics) while some intend to do so. Many SUPPORT Units and non-SUPPORT Unit initiatives are being planned to use these expanded “Big Data” capabilities (see example impact from Newfoundland & Labrador in section 3.3.2.6).

“We’re trying to “liberate” all data housed in [our region’s] research and clinical labs.”

SU Executive Director

These initiatives will also be able to link some databases owned by individual researchers, thus providing users with easy access (e.g., cloud-based), interoperability across different data sets, and high levels of security and privacy. Further, the SUPPORT Units provide expert advice as to the nature of these data (because there is so much, it is difficult for researchers to even know what is available), how to access them, and help conduct and interpret the analysis. The Newfoundland and Labrador SUPPORT Unit has the IBM Canada Research & Development Centre as a major funding partner and provider of e-solutions and “deep analytics”. IBM is also involved in the Newfoundland Translational and Personalized Medicine Initiative (TPMI).

The SUPPORT Units case study identifies SPOR to have been a critical factor in at least the smaller provinces in driving these initiatives and allowing them to hire key personnel and make cross-appointments with universities. (Very large provinces usually had significant initiatives pre-SPOR, so the additional effect of SPOR is far less clear). Many (possibly all) of these data platform initiatives are tied to existing institutes that house healthcare data, such as the Newfoundland and Labrador Centre for Health Management and Informatics, or Ontario’s Institute for Clinical Evaluative Sciences. Even where there was an existing initiative, however, in smaller provinces the SPOR funding accelerated the development process: e.g., the Data Science Centre in the George & Fay Yee Centre for Healthcare Innovation in Manitoba was reportedly only a “one person shop” before SPOR funding.

Consultation and research services provided by the SUPPORT Units centered around data analyses and/or data management activities, knowledge translation support, study design (including methods) assistance, and support for research proposal/grant application submissions. Least accessed services included scientific writing experience, literature review and/or metadata review, as well as, perhaps surprisingly, PE support.

All key informant and case study interviewees acknowledged each SUPPORT Unit was at varying levels of maturity and progressing to outcomes at varying rates. Progress was identified at the provincial level, as each SUPPORT Unit is operationalizing its business plan.

Pan-Canadian networks are mobilized

ACCESS Open Minds and PIHCI are established networks, although both have developed more slowly than originally expected and are not fully operational.

The ACCESS Open Minds Network was officially announced in June of 2014 so at the time of the evaluation, has been in development for one and a half years. As identified through the Networks case study, ACCESS Open Minds topics were seen to be in an area of significant need for some type of large, multi-faceted, initiative, to demonstrate how youth mental health care could be transformed. ACCESS Open Minds is intended to gather evidence on how to do so, analyze it, and apply it to communities. Once fully operational, expectations are high among case study respondents that needs in this area will be met and that ACCESS Open Minds will be capable of transforming services for youth with mental illnesses at sites across Canada. Respondents believe the benefits for youth will be earlier and more comprehensive access to services, based on real-world experimental evidence. The principal investigator believes ACCESS Open Minds has been able to put youth mental health services and needs “on the map”.

While ACCESS Open Minds intends to address services for young people in underserviced areas across Canada, the first site will only be fully operational in December 2015 (a First Nation community in Nova Scotia), and, although there is a deliberate multi-site design to this Network, there is significant site-by-site variation in how the model will be implemented.

ACCESS Open Minds has reported in its 2014-15 annual report that central office space has been secured at the Douglas Hospital Research Centre in Montreal, Quebec and main governance structures began to be put in place during the reporting year. This included the establishment of a Youth Council, a Family/Careers Council, Executive Committee and Governing Council. It is also reported that key staff have been hired for the central office including a Scientific & Clinical Director, Senior Project Manager, Training and KT, Research & Evaluation and Inter-site and Governance coordinators, two administrative assistances and one research assistant.

The PIHCI Network is still developing with the launch of its “quick strike research projects” in late 2015. PIHCI survey respondents noted that the function of PIHCI Network still appears to be in its incipient stages, and to date no space or structure has been created for collaboration and discussion between provinces and territories, although some structures are being established informally and more organically (e.g. monthly calls between coordinators).

Both Networks, while somewhat operational, are too new to have assessed outcomes at the time of the evaluation and as noted in Section 3.2 are still undergoing design and delivery adjustments.

Capacity development opportunities are provided

CD was seen to be the least developed SPOR element. Key informant and case study interviewees identified the lack of priority being given to this element as a hindrance, in part due to the only recently released SPOR CD Framework (August 2015). However, the release of the Framework was identified as a clear success in the development of a pan-Canadian approach, vision and framework regarding CD (and also PE). The CD and PE case studies identified the effectiveness of SPOR bringing together CD personnel within provinces, and beginning to open the conversations across provinces.

Findings from the CT and CD case studies indicate progress in this element to be tied to the activities underway at the local level though the SUPPORT Units. Knowledge and use of the SPOR CD Framework varied substantially across the SUPPORT Units and the ACCESS Open Minds Network, with, of course, all component leads responsible for CD being very familiar with it. The ACCESS Open Minds Network has not used the Framework to date, but intends to support capacity development through an extensive training program now in preparation, though very targeted in nature. The intent is for development of certification level courses; e.g., trainees will be certified as an ACCESS Open Minds youth mental health clinician, trained to do mental health assessments.

The PE Case study identified capacity development as an issue within the PE element of SPOR. Specifically a gap in the development and provision of training in PE for both patients and researchers to engage with each other (different than the governance training mentioned above). The PE case study noted that interviewees were unaware of a deliberate link being made between the PE Framework and the CD Framework.

Through the international comparative review, training was noted as an area of stress for NIHR so SPOR is not alone here. NIHR was described as consisting of various entities, each with their own training for researchers, as well as the public/patients. INVOLVE was viewed as having taken the role of ensuring public/patient stakeholders are made aware of the training available, though it was noted that there is no one entity that reviews the public/patient training under NIHR for gaps or duplication and no clear development path for public/patient stakeholders who wish to be involved. This was compared to CIHR SPOR, which was viewed as having a strong approach at the provincial level, but potentially requiring a strengthened national level approach.

However, there are examples of some activity (e.g., studentships, training programs within the SUPPORT Units related to PE and/or POR). SUPPORT Units reported a total of 422 capacity development supports and services across two annual reporting periods (2013-14 and 2014-15). The largest volume of activities during both reporting periods were conducted in the research methods key competencies area, as illustrated in Exhibit 15. Knowledge translation activities experienced the highest level of growth between the periods, with the least degree of growth seen in patient partnership activities.

Exhibit 15 – Number and type of supports and services – Training and CD – total number of activities 2013-2014, 2014-2015

Exhibit 15 long description
Number of activities 2014 2015
Research methods key competencies 17 135
Patient partnerships 8 25
Leadership in research 1 39
Leadership in health systems 0 29
Knowledge translation 0 126
Other 8 34

Note: Only one SUPPORT Unit was reporting on these areas in 2014.

  Research methods key competencies Patient partnerships Leadership in research Leadership in health systems Knowledge translation Other
2015 Min 0 0 0 0 0 0
2015 Mean 23 4 7 5 21 6
2015 Max 99 20 28 22 99 24

CD plans being developed by the SUPPORT Units include elements such as PE and POR studentships, presentations by patients as to how they became involved in research and how they like/dislike being approached, new methods for POR, use of Big Data, and knowledge translation of science and how to apply it.

However, key informant interviewees, the SUPPORT Units and CD case study respondents commented that there was a lack of direction from CIHR as to where they want to go nationally with respect to CD, especially with respect to concrete guidelines. At the moment, these respondents believed this initiative was more loose-knit than one would ideally want, and that the national Framework’s wording was too broad.

Additionally, there is confusion as to capacity development vs. career development. Both are referred to as “CD,” increasing the confusion – different SUPPORT Units appear to be interpreting this component in different ways. Capacity development is mainly interpreted as increasing capacity of patients to participate in research, and of researchers and clinicians to effectively use patient input – with the SUPPORT Units case study identifying that knowledge of best practices here is still poor, as are metrics of success. However, capacity development can also mean the understanding of, and competence of, researchers to conduct POR – assuming they know exactly what “POR” is. Career development appears to mainly involve training students in POR methods, although this is unclear.

Through the SUPPORT Units case study it was noted that building capacity represent a range from dealing with beginning students up to leadership training, and CD in some regions explicitly addresses how to build leaders in POR. For example, one SUPPORT Unit is building a “ladder” in POR techniques, with certificates and diploma programs up to the Masters level, following which leadership training will be provided. It was further noted that individual POR elements such as Knowledge Translation or Clinical Trials would be included within the CD modules.

The main needs to be addressed identified in the CD case study were:

CCTCC is established

The CCTCC has been operational for over a year, although still in the preliminary stages of implementation. It is perceived by CT case study interviewees to have made progress as anticipated on the nine recommendations defined in the Canadian Clinical Trials Summit Action Plan. Achievements against the nine recommendation areas are as follows:

CCTCC Recommendation Status Recommendation Achievements to Date

Achieved

1. National Headquarters to oversee Action Plan & to coordinate related clinical trial activities

On April 24, 2014, the Government of Canada announced the creation of the Canadian Clinical Trials Coordinating Centre (CCTCC) – a collaborative effort of the Canadian Institutes of Health Research, Innovative Medicines Canada, and the merged organizations of the Association of Canadian Academic Healthcare Organizations and the Canadian Healthcare Association (ACAHO/CHA).

In terms of progress being made on the nine recommendations, all CT case study interviewees identified this recommendation as being the “single greatest success” to date.

In Progress

2. Metrics to evaluate & market progress on Action Plan’s goals

Current metrics are assessed manually, based on manual time/activity entry as well as survey results. This activity is time consuming and data has the potential to be inaccurate. Discussions on this issue have been raised to the executive group and national advisory group to review the need for further analysis.

The CT case study identified performance measurement as an area requiring improvement, both within the CCTCC as well as across all aspects of the CT element of SPOR.

In Progress

3. Bold vision for health care and research ecosystems

Definition expanded to include the goals of the broader SPOR and recommendations of the National Task Force on the Future of Academic Health Science Centres to bolster receptor capacity. This activity remains ongoing.

In Progress

4. Improve ethics review efficiency, and study feasibility of common consent, application, and harmonization

Progress has been made on the research ethics review. In April 2015, the CCTCC implemented a working group to explore the development of a pan-Canadian accreditation system for Research Ethics Boards (REB) that review clinical trials.Footnote 45 The CT case study also identified, however, that the majority of progress towards this initiative was being conducted outside of SPOR and the CCTCC, identifying provincial reviews as mainly responsible for progressing this agenda.

In Progress

5. National patient registries & recruitment plan

The National Patient Recruitment Strategy was noted to be making good progress, and is following timelines as anticipated.

In Progress

6. Common standard operating procedures, training & certification

Creation of Canadian standards has been completed, as has the development of common tools. Again, the CT case study has identified other external sources contributed to progress in this area.

Achieved

7. A model clinical trials agreement (mCTA) to help streamline the negotiation process and reduce start up time

The model Clinical Trial Agreement (mCTA) provides a standard model contract for use by clinical trial sites and sponsors in negotiating phase II and phase III clinical trial agreements. Though a difficult concept to bring through to realization, the mCTA is now complete. Currently the team is creating a web-based tool from which clients will be able to access their provincial contract. It is anticipated this will be completed by the end of January, 2016. Additionally, an implementation team for the mCTA is being established for communication of the agreement and supervision of the web-based tool, with the goal of extending the program to consist of an educational tool element.

Thirdly, though this was not within the original plan, the team is adding a budgeting component to the contracts. This work is anticipated to be completed in the spring.

In Progress

8. Optimized intellectual property protection & SR&ED Tax Credits

This activity remains ongoing.

Achieved

9. National Clinical Trials Asset Map

The CCTAM is a unique, robust, searchable web-based database designed to communicate Canada’s clinical research strengths to all stakeholders, including clinical trial sponsors. The CCTAM software is completed, though it will remain an ongoing project. It is anticipated that the new website will be up shortly.

It was noted that there may be a need for the CCTCC to further prioritize the recommendations it is tasked to address.

The CT case study documented the difficulty that case study interviewees had in ascertaining the incremental impact of SPOR on the progress of the CCTCC, since in a number of instances it was felt progress in certain areas was completed prior to the establishment of the CCTCC. Thus the specific importance of the CCTCC role going forward in SPOR is less clear.

The Clinical Trials element has also been incorporated within SPOR as a core function of each provincial/territorial SUPPORT Unit as well as through the clinical trials activities planned to be conducted across the SPOR Networks, however very little activity (almost none) has taken place in this area due to the early stages of development.

Methods in POR are advanced

In all cases it was viewed as being too early to identify specific methods being developed that are attributable to SPOR. However, there are activities taking place that will contribute to the eventual advancement of methods and the POR environment.

A few interesting examples were provided through the SUPPORT Units case study that are clearly examples of POR, including:

Ontario

Newfoundland & Labrador

Manitoba

In the Ontario example, patient feedback and experience directly informed a change in a selected outcome measure for a major clinical study moving the study focus from hospital readmissions, which mattered to the clinicians, to measuring ER visits, which mattered to patients. The home-based patient monitoring technology in Newfoundland and Labrador is offering early detection of issues so health services can intervene with patients sooner providing better patient outcomes; and for Manitoba, getting the right treatment (Massive Transfusion Protocol) to patients faster and more efficiently resulted in decreases in mortality.

There is also research output being produced through the SUPPORT Units. The total number of research output across SUPPORT Units was 203 publications or papers during the 2014-2015 reporting period. As demonstrated in Exhibit 16, the largest volume of research outputs during the period were from scientific publication.

Exhibit 16 – Research output of SPOR Network supported researchers, reporting period 2014-2015

  Plain language publications Scientific publications Educational materials
Min 0 0 0
Mean 4 20 10
Max 20 99 52
Total 25 118 60

Further, the case studies do document a change in POR activities by those involved in SPOR.

“For myself, I’ve always worked closely with First Nations communities. But our thinking has expanded to involve decision-makers . . . the sit-down with the policy makers is now easier because SPOR forces us all to think more carefully about what you want at the end of the day.”

Clinician/scientist

The SUPPORT Units and Networks case studies identified that many stakeholders involved with SPOR had at least some leanings towards POR before SPOR’s roll-out, although the extent of this varied enormously. With SPOR present, these activities have become even more intense, creating a pressure to do more of this type of work, partnership-based, and across sectors (e.g., with the public health and government departments of health services and health policy.) The SUPPORT Units and CD case identified more discussion on how PE should be conducted, knowing previously that PE is important, but now see it needs a value proposition in order to achieve sustainability.

The SUPPORT Units case study also found a significant shift towards accounting for the priorities and challenges of stakeholders in other sectors and working with research groups they had not worked with before – and in some cases did not know even existed. This includes the integration of research topic areas and the creation of shared research opportunities where previously these tended to be investigated in silos.

The Network case, in particular the results of the PIHCI survey, also identified that SPOR has demanded researchers to think more deliberately about engaging the end-users of the primary health care services in a meaningful way. The national scope of the Network and the requirement for partnership across jurisdictions, has enabled work on projects that involve more than activities in a single province.

For ACCESS Open Minds, patient input was noted to make the research more practicable and able to be effectively applied.

3.3.2.2 Movement toward longer term impacts (Evaluation Question #6)

One of SPOR’s key achievements is making all relevant stakeholders respond to “the pressure of ideas,” expanding the recognition of the value of multi-sector, multi-stakeholder, focus on results to patients; really considering SPOR principals and goals. It is now reported to be very difficult to plan major clinical research initiatives without at least taking POR-related concepts into account; e.g., PE, stakeholder engagement, evidence-based clinical policies and procedures.

Although there are identified gaps and it is re-iterated that SPOR is still very early in its lifecycle, the average rating provided by key informant and case study interviewees indicated, on average, that SPOR has contributed to a change in Canada’s health research landscape to a moderate extent. Many (57%) rated SPOR’s contribution to be at least to a moderate extent or greater and equally many (59%) rated SPOR’s contribution to be moderate or less as shown in Exhibit 17. The SUPPORT Unit case study respondents were those mainly identifying this change taking place to a great extent.

Exhibit 17 – Extent that Canada’s health research landscape changed because of SPOR

Exhibit 17 long description
  Not at all
(1)
To some extent
(2)
To a moderate extent
(3)
To a great extent
(4)
To a very great extent
(5)
(η) Average Rating
Key Informants 0 4 4 4 1 13 3.2
SUPPORT Units 2 6 1 12 1 22 3.2
Networks 0 2 0 1 0 3 2.7
Capacity Development 0 0 0 0 1 1 5.0
Patient Engagement 0 4 3 1 0 8 2.6
Clinical Trials 0 4 0 0 0 4 2.0
Total 2 20 8 18 3 51 3.0
Not at all (1) To some extent (2) To a moderate extent (3) To a great extent (4) To a very great extent (5)
Key Informants 0% 31% 31% 31% 8%
SUPPORT Units 9% 27% 5% 55% 5%
Networks 0% 67% 0% 33% 0%
Capacity Development 0% 0% 0% 0% 100%
Patient Engagement 0% 50% 38% 13% 0%
Clinical Trials 0% 100% 0% 0% 0%
Total 4% 39% 16% 35% 6%

*Rounding has been forced to add to 100%.

*Average rating has been calculated as a standard weighted average based on the number of respondents.

The most frequently mentioned area of change, attributed to SPOR, is the increased involvement of all stakeholders in health research. Broadening the scope of stakeholders involved to span the researcher, clinician, policy maker and patient communities was noted to be one of the key change drivers to POR research. This was identified by key informant interviewees as well as through the Networks, CT, CD and SUPPORT Units case studies.

The other area identified was the focus on the relevance of PE. Through the key informant interviews and the CD and PE case studies, SPOR was seen to have fostered and perpetuated a conversation on the relevance of PE in research and the release of the PE Framework was viewed to set the tone and understanding for what the research community could expect regarding patient-oriented research going forward.

It is still very early days, however, and SPOR is seen by virtually all key informant interviewees to have shifted expectations, and built themes and goals that are – if not perfectly coordinated and clear – at least much more widely understood than was previously the case. Because of the SUPPORT Units priority-setting exercises, projects supported by the SUPPORT Units are focused on provincial/territorial priorities, and because of SPOR they focus more strongly on outcomes of importance to patients. This is a distinct change from the traditional CIHR biomedical model.

Similar results are seen when looking at the perceptions of SPOR’s contribution to a culture change towards increased buy-in to POR. All key informant and case study interviewees identified that SPOR has had an influence on the expected culture change required for the conduct of patient-oriented research to a moderate extent on average, with 66% identifying SPOR’s contribution to be moderate or lower and 68% rating SPOR’s contribution to be moderate or higher. Exhibit 18 identifies the proportion of respondents by rating selection.

Exhibit 18 – Extent of SPOR’s influence on culture change to enable POR

Exhibit 18 long description
  Not at all (1) To some extent (2) To a moderate extent (3) To a great extent (4) To a very great extent (5) (η) Average Rating
Key Informants 0 6 4 1 2 13 2.9
SUPPORT Units 0 6 9 10 1 26 3.2
Networks 0 1 1 0 0 2 2.5
Capacity Development 0 0 0 2 0 2 4.0
Patient Engagement 1 1 4 3 0 9 3.0
Clinical Trials 3 0 1 0 0 4 1.5
Total 4 14 19 16 3 56 3.0
Not at all (1) To some extent (2) To a moderate extent (3) To a great extent (4) To a very great extent (5)
Key Informants 0% 46% 31% 8% 15%
SUPPORT Units 0% 23% 35% 38% 4%
Networks 0% 50% 50% 0% 0%
Capacity Development 0% 0% 0% 100% 0%
Patient Engagement 11% 11% 44% 33% 0%
Clinical Trials 75% 0% 25% 0% 0%
Total 7% 25% 34% 29% 5%

*Rounding has been forced to add to 100%.

*Average rating has been calculated as a standard weighted average based on the number of respondents.

However, the qualitative elaborations supporting the ratings indicate a lower level of influence being perceived by key informant and case study interviewees. Culture change was consistently noted to be a long term endeavor and SPOR was noted to be too young for such change to have taken place at this point in time. There were a number of consistent themes that emerged through the key informant interviews and case studies where SPOR would need to continue efforts in order to influence a change in culture, and it was noted that should SPOR “stay the course,” these culture shifts would indeed occur.

Common areas of focus include:

Resistance within the research community

There has been some slow understanding and uptake by some areas/members of the research community who are not convinced of the value of POR or PE, mainly related to the lack of hard evidence as to their value on the quality of the research or the importance of the resulting clinical outcomes. Since POR and PE require significant additional investments of time and energy on the part of researchers, many respondents believe these sceptics will remain unconvinced until hard evidence is available.

Additionally, on the research side, there are disincentives to undertake POR. Institutional tenure and promotion systems may not recognize or reward activities that are primarily directed at practical healthcare challenges, as opposed to fundamental biomedical discovery research and resulting peer review publication. This can be especially discouraging and traditional university systems need to be brought on board to build capacity and leadership in POR areas.

Many interviewees noted that in order to further a culture of PE, it would be beneficial to provide examples of enhanced success due to PE. CIHR needs to keep showing the research community concrete exemplars that POR is better, and why – demonstrating to researchers that POR isn’t a waste of their time. There is a wide-spread belief that these examples are still lacking – there are said to be some pockets of evidence, but more are needed. The SUPPORT Units demonstration projects are helpful, but more of them are needed, perhaps foremost from the SPOR Networks. This also applies to efforts to get patients on board as well. Resistance on the part patients that were unfamiliar with patient-oriented research, both in terms of conduct and potential benefits.

The area of POR understanding and uptake was also identified as a key need internationally to further support the trend of patient-oriented research models. International key informants noted the need for a stronger and increasingly visible generation of evidence supporting the impacts of implementing a patient-oriented research model. It is essential to provide support to researchers, policy makers, and any other potential stakeholders that there is tangible and intangible value to implementing a patient-oriented research model. Further, it was viewed to be essential to provide evidence supporting the optimal level of PE, as it was identified that the optimal level of engagement may vary depending on the type of research being conducted.

Aligned to this were comments on enhancing accountability measures of researchers employing PE. The PE case noted that though the requirement for PE within grant applications is an excellent first step, there is a continued need to ensure accountability of researchers in employing PE. Interviewees noted a concern that researchers are incentivized to employ PE only to the degree that it will fulfill grant application requirements, and there is no accountability to ensure the quality of PE being realized. Suggestions were made to either enhance the performance indicators on the part of researchers, or to allow for patients to report on the level within which they were engaged.

All of these points were raised through key informants, international key informants, and SUPPORT Units, CD and PE cases.

Communicating a clear definition of PE

Despite the definition of PE within the SPOR PE Framework, the SUPPORT Unit and PE case studies noted that those involved in SPOR still operate under different definitions. Interviewees identified that defining the appropriate level of PE is still underway, as each area of research may be optimized through varying levels of PE. However, interviewees also noted that in order to gain further acceptance within the research community, as well as by additional stakeholders such as clinicians and policy makers, PE would need to be better operationalized to provide a coherent understanding of how patients can be engaged in a meaningful way for the enhancement of research outcomes.

Communicating a clear definition of SPOR

SPOR is a very large and complex initiative, resulting in many respondents being uncertain as to how all the “moving parts” fit together, not to mention the difficulties encountered in explaining an initiative of such broad scope in plain language. There is a need to be able to present activities in a way that makes them engaging and meaningful to an audience beyond the academic research community and being able to further penetrate down to the “on-the-ground” research, clinical and policy areas in the jurisdictions. These points came through the key informant interviews and the SUPPORT Units case study.

3.3.3 Cost-efficiency Analysis (Evaluation Question #7)Footnote 46

From 2010-11 to 2015-16, $202.83M was allocated specifically to SPOR by Treasury Board. Resources were targeted for SPOR SUPPORT Units, Networks, a Clinical Research Coordinating Centre, and other relevant research investments. Over this same time period, CIHR spent a total of $356.86M on SPOR grants and awards (Exhibit 21), thus increasing the overall envelope of funding allocated to SPOR and SPOR-related research.

Exhibit 19 – SPOR Funding from Treasury Board (TB), 2010-11 to 2015-16

TB Submission Year 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16
2010-11 G&A $6,000,000 $6,000,000 $6,000,000 $6,000,000 $6,000,000 $6,000,000
2010-11 O&M $455,000 $455,000 $455,000 $455,000 $455,000 $455,000
2011-12 G&A $0 $9,000,000 $9,000,000 $9,000,000 $9,000,000 $9,000,000
2011-12 O&M $0 $820,000 $820,000 $820,000 $820,000 $820,000
2012-13 G&A $0 $0 $13,936,000 $13,936,000 $13,936,000 $13,936,000
2012-13 O&M $0 $0 $1,064,000 $1,064,000 $1,064,000 $1,064,000
2013-14 G&A $0 $0 $0 $14,000,000 $14,000,000 $14,000,000
2013-14 O&M $0 $0 $0 $1,000,000 $1,000,000 $1,000,000
2014-15 G&A $0 $0 $0 $0 $5,000,000 $5,000,000
2014-15 O&M $0 $0 $0 $0 $0 $0
2015-16 G&A $0 $0 $0 $0 $0 $0
2015-16 O&M $0 $0 $0 $0 $0 $0
 Cumulative Total G&A $6,000,000 $21,000,000 $49,936,000 $92,872,000 $140,808,000 $188,744,000
Cumulative Total O&M $455,000 $1,730,000 $4,069,000 $7,408,000 $10,747,000 $14,086,000
Cumulative SPOR Funding from TB $6,455,000 $22,730,000 $54,005,000 $100,280,000 $151,555,000 $202,830,000

The financial data reflect that all SPOR core elements now have committed funding at scale. When considering the fidelity assessment, the financial data converge with data from elsewhere in the evaluation and confirm the delays in the implementation of the SPOR SUPPORT Units and Networks.Footnote 47 Official program documents indicate there was a planned delay through the first two years of SPOR with activities identified for 2011-12 to include the establishment and engagement of SPOR’s National Steering Committee, the determination of priorities, the creation of the funding opportunities for the Networks and SUPPORT Units, as well as some specified work in the Clinical Trials area.

The unplanned delays were attributed by the program to the considerable amount of time needed to engage partners and to design and deliver the elements in a way that would achieve the greatest outcomes as follows:

SPOR Grants and Awards expenditures are allocated to SPOR core elements as well as Foundational investments. Official program documents identified existing CIHR investment in POR areas prior to the release of SPOR that includes, as examples, funding randomized control trials, various research networks and teams as well as methods centers for clinical research.

There were a number of relevant and important programs already underway that helped to set the stage/prime the Canadian clinical and patient-oriented research environment for SPOR’s launch and implementation. These Foundational investments were and continue to be important as most play a role in the development of SPOR-relevant research capacity and are directly aligned with SPOR. For example:

CIHR has planned to gradually refocus this Foundational spending as the core elements of SPOR are implemented.

The investment towards the Foundational elements has been decreasing over the evaluation timeframe. On average Foundational investments have decreased 15% annually, or by 56% in total from 2010-11 through 2015-16. Official program documents note this is a planned refocusing of the spending (Exhibit 20).

Exhibit 20 – Extent of SPOR’s influence on culture change to enable POR

Exhibit 20 long description
TB Submission Year 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16
2010-11 G&A $6,000,000 $6,000,000 $6,000,000 $6,000,000 $6,000,000 $6,000,000
2010-11 O&M $455,000 $455,000 $455,000 $455,000 $455,000 $455,000
2011-12 G&A $0 $9,000,000 $9,000,000 $9,000,000 $9,000,000 $9,000,000
2011-12 O&M $0 $820,000 $820,000 $820,000 $820,000 $820,000
2012-13 G&A $0 $0 $13,936,000 $13,936,000 $13,936,000 $13,936,000
2012-13 O&M $0 $0 $1,064,000 $1,064,000 $1,064,000 $1,064,000
2013-14 G&A $0 $0 $0 $14,000,000 $14,000,000 $14,000,000
2013-14 O&M $0 $0 $0 $1,000,000 $1,000,000 $1,000,000
2014-15 G&A $0 $0 $0 $0 $5,000,000 $5,000,000
2014-15 O&M $0 $0 $0 $0 $0 $0
2015-16 G&A $0 $0 $0 $0 $0 $0
2015-16 O&M $0 $0 $0 $0 $0 $0
Cumulative Total G&A $6,000,000 $21,000,000 $49,936,000 $92,872,000 $140,808,000 $188,744,000
Cumulative Total O&M $455,000 $1,730,000 $4,069,000 $7,408,000 $10,747,000 $14,086,000
Cumulative SPOR Funding from TB $6,455,000 $22,730,000 $54,005,000 $100,280,000 $151,555,000 $202,830,000

In turn, it is evident there has been an increase in spending on SPOR’s core elements over the last three fiscal years, reflecting the launch of SUs in 6 jurisdictions and funding of 7 Networks across three priority areas (Mental Health; Primary and Int0grated Health Care Innovations; and Chronic Diseases).

Exhibit 21 – CIHR Grants and Awards Expenditures on SPOR, 2010-11 to 2015-16

Grants and Awards (CIHR Expenditures) 2010-11 2011-12 2012-13 2013-14 2014-15  2015-16 TOTAL
SPOR Networks $0 $0 $0 $898,220 $3,472,925 $8,244,145 $12,615,290
SUPPORT Units $0 $0 $0 $13,554,918 $21,087,625 $33,291,588 $67,934,131
Capacity Development - Core $0 $0 $401,667 $927,243 $879,788 $514,166 $2,722,864
Capacity Development - Foundational $9,088,962 $7,988,277 $6,765,368 $5,719,213 $5,166,603 $4,725,392 $39,453,815
Patient Engagement $0 $0 $0 $0 $164,796 $280,160 $444,956
Clinical Trials - Canadian Clinical Trials Coordinating Centre $0 $0 $0 $62,500 $250,000 $250,000 $562,500
Clinical Trials - Foundational $32,564,377 $28,100,558 $24,864,038 $23,431,606 $20,354,681 $14,981,390 $144,296,650
Additional Foundational Investments $21,908,452 $18,838,927 $16,986,970 $12,987,579 $9,645,716 $8,442,400 $88,810,044
TOTAL SPOR GRANTS and AWARDS $63,561,791 $54,927,762 $49,018,043 $57,581,279 $61,022,134 $70,729,241 $356,840,250

As the SPOR core elements have committed funds at planned scale, there have been commensurate increases in administration costs associated with its delivery, reflecting a growth in staff tied to the Strategy. The increases in operating and maintenance costs reflect implementation of the design features of the Strategy such as its governance structures, face-to-face meetings with provinces and territories to support SUPPORT Unit applications, iterative peer review processes, Strengthening Workshops for the Networks delivered between Letter of Intent and Full Application stages and the annual SPOR Summit, intended to enable cross-element learning and collaboration.

Exhibit 22 – CIHR Administration Costs on SPOR, 2010-11 to 2015-16

DIRECT ADMINISTRATION COSTS (CIHR Expenditures) 2010-11 2011-12 2012-13 2013-14 2014-15  2015-16 TOTAL
Direct Salary $259,029 $604,402 $949,774 $1,791,620 $1,893,393 $1,923,264 $7,421,482
EBP - 20% $51,806 $120,880 $189,955 $358,324 $378,679 $384,653 $1,484,296
Accommodation - 13% $33,674 $78,572 $123,471 $232,911 $246,141 $250,024 $964,793
Direct Operating and Maintenance $187,445 $95,710 $309,529 $286,592 $270,230 $213,403 $1,362,909
Internal Services (Indirect Administration Costs) $2,536,115 $2,191,618 $1,955,820 $2,297,493 $2,434,783 $2,822,097 $14,237,926
TOTAL - Administration Costs (Direct + Indirect) $3,068,069 $3,091,182 $3,528,548 $4,966,940 $5,223,226 $5,593,441 $25,471,406

SPOR has been able to increase its efficiency by using the existing CIHR program delivery mechanisms to invest SPOR Grants and Awards dollars in Foundational investments. There is no net new incremental salary or O&M expenditures related to the delivery of the Foundational elements, as SPOR retains the tracking and oversight of these investments.

Therefore, in calculating the annual operational efficiency of SPOR no direct salary or O&M data for the Foundational investments have been included. The total internal services (indirect administration costs) have been calculated on the entire Grants and Awards envelope which would cover additional small amounts of effort related to Priority Announcements directly under a core element (e.g., Doctoral Research Awards in Patient Oriented Research – Biostatistics under SPOR’s core element of Capacity Development) where needed.

The operational efficiency of SPOR conservatively ranges between 4.6-7.9% (Exhibit 23).

Exhibit 23 – CIHR Expenditures on SPOR, 2010-11 to 2015-16

TOTAL SPOR (CIHR Expenditures) 2010-11 2011-12 2012-13 2013-14 2014-15  2015-16 TOTAL
TOTAL SPOR Grants and Awards $63,561,791 $54,927,762 $49,018,043 $57,581,279 $61,022,134 $70,729,241 $356,840,250
TOTAL - Administration Costs (Direct + Indirect) $3,068,069 $3,091,182 $3,528,548 $4,966,940 $5,223,226 $5,593,441 $25,471,406
TOTAL SPOR Expenditures $66,629,860 $58,018,944 $52,546,591 $62,548,219 $66,245,360 $76,322,682 $382,311,656
Administration Costs as % of Total Expenditures 4.6% 5.3% 6.7% 7.9% 7.9% 7.3%  

An important principle of SPOR is the leveraging of non-federal partner funds to match, at a 1:1 ratio, the investments made from the federal government toward the core elements of SPOR. For the SUPPORT Units, Networks, Patient Engagement, and Canadian Clinical Trials Coordinating Centre core elements it can be seen that for every $1.00 invested from CIHR, the same or more was invested by a partner over the full time-frame of the investments. For the investments made in capacity development core element, since these were primarily priority announcements for awards (not grants) for New Investigators, Fellows, and Doctoral Researchers, there were no expectations of matching funds.

For those Foundational Investments where such partnership contributions were not required, partnership funding is still evident, however at a much lower rate. This suggests that requiring partner matching funding is an effective mechanism to extending CIHR investments in SPOR core elements.

Exhibit 24 – CIHR and Partners’ Commitments on SPOR, 2010-11 to 2020-21

Program Funding Source 2010-11 2011-12 2012-13 2013-14 2014-15  2015-16 2016-17 2017-18 2018-19 2019-20 2020-21 TOTAL Ratio of CIHR: Partner Contribution
SPOR Networks CIHR $0 $0 $0 $898,220 $3,472,925 $8,244,145 $17,818,040 $16,584,689 $15,818,119 $12,226,777 $8,167,025 $83,229,940 $1.00:$1.73
Partner $0 $0 $0 $828,750 $2,224,750 $4,470,257 $30,745,266 $28,922,019 $28,196,385 $24,529,302 $23,831,424 $143,748,153
SUPPORT Units CIHR $0 $0 $0 $13,554,918 $21,087,625 $33,291,588 $39,988,290 $41,583,997 $26,726,134 $14,627,034 $4,075,507 $194,935,093 $1.00:$1.12
Partner $0 $0 $0 $28,533,328 $29,047,090 $39,856,060 $38,423,916 $37,305,683 $25,830,267 $14,627,034 $4,075,507 $217,698,885
Capacity Development - Core CIHR $0 $0 $401,667 $927,243 $879,788 $514,166 $132,500 $110,000 $0 $0 $0 $2,965,364 $1.00:$0.00
Partner $0 $0 $0 $0 $0 $0 $0 $0 $0 $0 $0 $0
Capacity Development - Foundational CIHR $9,088,962 $7,988,277 $6,765,368 $5,719,213 $5,166,603 $4,725,392 $1,959,554 $1,350,761 $498,523 $185,552 $58,046 $43,506,251 $1.00:$0.23
Partner $1,841,950 $1,666,960 $1,558,892 $1,371,182 $1,002,457 $785,966 $675,251 $507,271 $175,259 $127,823 $78,804 $9,791,815
Patient Engagement CIHR $0 $0 $0 $0 $164,796 $280,160 $0 $0 $0 $0 $0 $444,956 $1.00:$1.47
Partner $0 $0 $0 $0 $286,906 $366,028 $0 $0 $0 $0 $0 $652,934
Clinical Trials - Canadian Clinical Trials Coordinating Centre CIHR $0 $0 $0 $62,500 $250,000 $250,000 $187,500 $0 $0 $0 $0 $750,000 $1.00:$1.06
Partner $0 $0 $0 $62,500 $265,000 $265,000 $202,500 $0 $0 $0 $0 $795,000
Clinical Trials - Foundational CIHR $32,564,377 $28,100,558 $24,864,038 $23,431,606 $20,354,681 $14,981,390 $9,492,585 $6,492,618 $4,527,355 $1,884,031 $298,925 $166,992,164 $1.00:$0.00
Partner $0 $0 $0 $0 $0 $0 $0 $0 $0 $0 $0 $0
Additional Foundational Investments CIHR $21,908,452 $18,838,927 $16,986,970 $12,987,579 $9,645,716 $8,442,400 $7,343,297 $6,992,705 $2,117,797 $1,364,667 $619,751 $107,248,261 $1.00:$0.32
Partner $11,752,117 $10,664,589 $8,982,758 $1,388,413 $586,856 $535,796 $298,996 $176,510 $93,608 $67,917 $29,167 $34,576,727
Sub-total: CIHR investments $63,561,791 $54,927,762 $49,018,043 $57,581,279 $61,022,134 $70,729,241 $76,921,766 $73,114,770 $49,687,928 $30,288,061 $13,219,254 $600,072,029 $1.00:$0.68
Sub-total: Partner investments $13,594,067 $12,331,549 $10,541,650 $32,184,173 $33,413,059 $46,279,107 $70,345,929 $66,911,483 $54,295,519 $39,352,076 $28,014,902 $407,263,514
Total SPOR investments $77,155,858 $67,259,311 $59,559,693 $89,765,452 $94,435,193 $117,008,348 $147,267,695 $140,026,253 $103,983,447 $69,640,137 $41,234,156 $1,007,335,543  

3.3.4 Conclusions

Although SPOR is in its initial phases of implementation, it has advanced toward achievement of its stated immediate outcome areas. The main areas of achievement include the establishment of six SUPPORT Units that are providing support and services to the POR communities, establishing partnerships, increasing the availability of data to inform and accelerate research, and beginning to undertake POR projects and producing research outputs. Seven national research Networks have been launched, and are at the stage of undergoing implementation. The CCTCC has been operational for just over one year and has delivered against two of its key objectives, with the completion of a model clinical trials agreement and a clinical trials asset map.

SPOR is now committing funding at full scale to its core components and has leveraged partner dollars, meeting is own requirement for 1:1 matching and in most cases exceeding this ratio. SPOR is refocusing its efforts to supporting the core elements as planned, evidenced by a significant decrease in funding directed towards Foundational investments.

SPOR has actively engaged stakeholders across sectors and across participant communities. Extensive efforts have been made at the national and provincial/territorial levels, tying researchers, clinicians, policy makers and patients together across sectors and geographies. However, there are ongoing opportunities to continue to strengthen stakeholder engagement with other networks and institutions involved in POR to further leverage resources and expertise. More important is the requirement to further strengthen SPOR’s PE element through increased work on capacity development (patients learning how to participate in research, and researchers learning how to effectively engage patients) as well as expanding the national lens applied to more consistent and well-communicated development of PE and capacity development efforts.

Through SPOR’s successes in aligning with provincial/territorial priorities, increasing the involvement of all stakeholders in health research, promoting the importance of PE and establishing an environment where the focus is on outcomes of importance to patients, SPOR has somewhat “moved the needle” in terms of influencing the health care research landscape and shifting the culture towards POR. There is still much to do in this area, with culture change being a very long term endeavor and SPOR’s continuing efforts to address the resistance within the various stakeholder communities are important. SPOR is not alone here; there is an international need for stronger and more visible generation of scientific evidence assessing the impacts of implementing a patient-oriented research model.

In terms of progress toward intermediate outcomes, the alignment and integration of the specialized services and methodological expertise of all SPOR core elements will be critical to providing support services responsive to health system needs, generating and translating research to inform health care policy and practice, and improving the clinical trial environment in Canada.

Underlying these core outcomes is the ongoing need for increased awareness of and expertise among researchers in patient-oriented research and enhanced engagement of patients – essential for patients to become active partners in research and clinical practice thereby strengthening Canadian patient-oriented research capacity.

The alignment and integration of the activities of SPOR’s core elements, enhanced engagement of patients and greater clarity in communicating key concepts, future plans and priorities, and early wins to increase buy-in for POR are necessary to complete implementation and achieve intended results.

There are three overarching issues that must be addressed for SPOR to move forward:

4 Recommendations

Overall, SPOR is relevant, addresses an ongoing need for POR, and performance is on track given the stage of implementation –with some adjustments needed to design and delivery.

The following recommendations are identified that have the potential to improve program effectiveness, performance and design/delivery:

1. CIHR should increase efforts to strengthen SPOR’s role in a common agenda for change to patient-oriented research.

2. CHIR should provide strategic guidance regarding how SPOR elements are to work together toward achieving SPOR’s intermediate and long-term outcomes.

3. CIHR should communicate plans for moving beyond the initial five-year funding period to manage sustainability expectations for CIHR investments in SPOR.

4. CIHR should strengthen approaches to enable cross-learning, sharing of best practices, and collaboration; this should occur within and across SPOR elements and between CIHR and Canadian and International organizations.

5. CIHR should continue to support effective management and administrative functions within funded SPOR SUPPORT Units and Networks and across these elements.

6. CIHR should revise the existing SPOR performance measurement strategy to balance administrative/operational outputs with outcomes/impacts.

Appendices

Appendix A – SPOR Logic Model

SPOR Logic Model

SPOR Logic Model long description

3.0 Logic Model

3.1 Logic Model Overview

SPOR’s logic model provides a graphic illustration of how its activities will contribute to the achievement of the expected outcomes.

Since SPOR aims to change the health research culture in Canada, it is estimated that it will take at least ten years to get evidence-based patient-oriented research integrated in the health care system, at the clinical practice and decision-making levels. Over its ten-year span, the Strategy will make patient-oriented research a major focus of health research at CIHR and in Canada.

Due to its complexity and the extensive stakeholder engagement that is required, there have been delays in its implementation. This delay was necessary in order to allow key partners, including provincial governments, time to mobilize resources, identify matching funding and develop strong business cases. The ten year life cycle of SPOR and the delay in its implementation are important factors to consider in performance measurement and evaluation.

The following figure depicts the Strategy’s logic model, which focuses on the strategic level outcomes. It is followed by a narrative description of its activities, outputs and outcomes.

3.2 Logic Model Narrative
3.2.1 Activities

The main SPOR activities are as follows:

Patient & stakeholder engagement in governance, decision-making processes, research & knowledge translation: Patients and stakeholders are engaged at all stages of SPOR, from the development of the strategy through to the implementation of its components. SPOR was developed after a series of consultations were held with relevant stakeholders, including officials from the federal, provincial and territorial governments and provincial health research funding agencies. To implement SPOR, CIHR is actively working with these stakeholders on an ongoing basis. Most importantly, patients are involved in key governance structures and decision-making processes in all SPOR elements and are supported by a variety of tools and resources to participate meaningfully in all stages of research reprocess. .

Design and manage SPOR funding tools and platforms: The core components of SPOR include complex funding tools and platforms. It is important to collaborate with stakeholders in their design and management so that SPOR outcomes are fully achieved. In addition, innovative approaches will be used to facilitate research across the full spectrum of patient-oriented research.

Provide support to, and facilitate links between, SPOR elements: CIHR organizes meetings and workshops to bring together participants and stakeholders of SPOR to provide opportunities for knowledge sharing and to facilitate greater collaborations. CIHR also co-chairs and provides secretariat support to a national SPOR SUPPORT Unit Council which has been established to coordinate activities across the various SUPPORT Units and ensure a sharing of best practices and lessons learned.

Oversee the implementation of the National Clinical Trials Summit Action Plan: The National Clinical Trials Summit in 2011 resulted in “An Action Plan to Help Attract More Clinical Trials to Canada.” Together with the Association of Canadian Academic Healthcare Organizations (ACAHO) and Canada's Research-Based Pharmaceutical Companies (Rx&D), CIHR has established a coordination centre to implement the action plan.

Develop, implement and maintain the governance structure: A National Steering Committee was established to oversee the development and implementation of SPOR. A Working Group was also struck to provide leadership within CIHR for the development, implementation and oversight of SPOR. Both the National Steering Committee and the Working Group meet regularly to address emerging issues and to make decisions on the direction of SPOR activities.

Ongoing monitoring, performance of SPOR overall and its elements: CIHR collects on-going data for performance measurement and reporting to Parliament, such as in the annual Departmental Performance Report. In addition, CIHR is working with the SUPPORT Units and the SPOR networks to collectively develop relevant performance measurement frameworks for the ongoing collection and analysis of performance information. CIHR will lead the evaluation of the SPOR, including the SUPPORT Units and the SPOR networks, to assess the Strategy’s design, delivery and performance to date and meeting federal government evaluation coverage requirements.

Communications and promotion: CIHR works closely with the office of the Minister of Health to promote the SPOR program to Canadians, stakeholders, and researchers. Communications efforts are focused on explaining patient-oriented research and its benefits to Canadians, engaging stakeholders, and promoting funding opportunities to researchers and partners.

3.2.2 Outputs

The primary SPOR outputs are as follows:

Multi-disciplinary and cross-sectorial partnerships: The core components of SPOR require matching funding from non-federal government partners. This funding can come from a variety of public, private for-profit and/or private not-for-profit sources, including ministries of health, social services, and/or education, provincial health research funding organizations, health charities, and private foundations.  

Funding issued for SUPPORT units, Networks, CCTCC and Patient Engagement: At capacity, SPOR will provide funding to up to nine SUPPORT Units across Canada. Up to ten networks will also be funded to link the best researchers across Canada to undertake patient-oriented research in designated health areas. Funding has been issued to the Canadian Clinical Trials Coordinating Centre to address the recommendations outlined in the Summit Action Plan. A variety of funding opportunities will build capacity for more meaningful patient engagement in research process.

Meetings and Workshops: As part of SPOR’s governance structure, the National Steering Committee meets up to two times per year. The SPOR Working Group also meets regularly. In addition to these governance meetings, workshops will provide substantial opportunities for exploring potential synergies, linkages, and partnerships for those involved in different SPOR elements, including SPOR Networks and SUPPORT Units.

Patient Engagement and Capacity Development frameworks: A SPOR Patient Engagement Framework has been developed to guide active and fruitful collaborations between patients, researchers, health care professionals, decision-makers and other stakeholders throughout the entire health research continuum. A SPOR Capacity Development Framework will be developed after consultations with relevant stakeholders.

Knowledge translation products & events: CIHR will undertake knowledge translation activities to ensure the results of SPOR-related research are disseminated to a wide audience. Knowledge translation products may include: plain language publications, scientific publications, educational material, mass media products or social media products.

Performance measurement & evaluation report: CIHR requires annual performance reporting related to progress towards, or achievement of, the SPOR outputs and outcomes from each of the SUPPORT Unit and Network elements. These elements are also required to submit mid-term and final evaluation reports. In alignment with TBS requirements, a program evaluation report for SPOR will be produced.

Communication & promotion products: Apart from maintaining a website to provide accurate and up-to-date information regarding SPOR, products are developed and maintained to support SPOR communication activities, including roll-up banners, brochures, fact sheets and newsletters.

3.2.3 Immediate Outcomes

The immediate outcomes that are expected to result from the SPOR activities and outputs are as follows:

Key stakeholders, including patients, are supported to engage in research & decision-making processes: In accordance with the SPOR Patient Engagement Framework CIHR will collaborate with relevant stakeholders to identify mechanisms to support patients, researchers, health care professionals and decision-makers to work, CIHR will provide funding opportunities, facilitate seminars, workshops and other events, and share knowledge and best practices. CIHR will also work closely with SPOR Networks and SUPPORT units to support them in embedding patient engagement in their work.

Multidisciplinary, integrated and accessible research platforms are established: As the funding is released, SUPPORT units will be established across Canada to provide the necessary expertise to pursue patient-oriented research and help lead reforms in response to locally-driven health care needs. Each of these Units will have its own governance structure, will be staffed with multi-disciplinary professionals to deliver highly specialized data, to provide integrated and accessible methodological and research services, and be equipped with both a performance measurement plan and a sustainability plan.

Pan-Canadian networks are mobilized to address health challenges: The SPOR Networks will be mobilized as collaborative alliances that incorporate the perspectives of all stakeholders, including patients, health care professionals and decision makers, in research and knowledge translation activities that lead to integration of evidence into practice. These networks will be national in scope and implement a coordinated nationwide patient-oriented research agenda in collaboration with relevant stakeholders, particularly patients. Each network will include a governance and management structure with appropriate mechanisms to ensure patient engagement, and will be equipped with both a performance measurement plan and a sustainability plan.

SPOR elements provide capacity development opportunities that align with the SPOR Capacity Development framework: In accordance with the SPOR Capacity Development Framework, indirect training, mentoring and career support will be integrated into the SPOR Networks and SUPPORT Units. The unique research and research support environments created will be leveraged to provide unique cross-disciplinary training environments for the next generation of patient-oriented researchers.

Canadian Clinical Trials Coordinating Centre is established: The established CCTCC will complete the summit action plan’s recommendations, serve as a coordinating function for relevant provincial and national initiatives, and determine next steps upon completion of the action plan. The CCTCC will be guided by an Executive Committee comprised of the Summit partners.

Methods in patient-oriented research are advanced: The core functions of SPOR SUPPORT Units include providing specialized services and methodological expertise to stakeholders within the clinical trial environment. The responsiveness of SUPPORT Unit services and expertise to the needs of stakeholders will result the advancement of knowledge in methods in patient-oriented research. Further advancement of knowledge and methods will occur as a result of patient-oriented research conducted by SPOR Networks.

3.2.4 Intermediate Outcomes

The intermediate outcomes that are expected to result from the SPOR activities, outputs and immediate outcomes are as follows:

Patients, health care providers and decision-makers are active partners in both research & implementation of evidence-based improvements: As new resources and mechanisms are developed, and linkages to the existing ones fully utilized, patients, health care providers and decision-makers will be informed about the broad spectrum of research-related activities and actively involved in those. With the infrastructures in place, and strong commitments from the stakeholders, patients, health care providers and decision-makers will be involved in both the research and decision-making processes.

Infrastructure and support services are responsive to health systems needs: As a result of the establishment of the SUPPORT Units, timely access to data and specialized and multidisciplinary methodological expertise will be provided. These units will also respond to the needs of health systems by identifying and addressing the needs of patients and knowledge users, facilitating research; and assisting decision makers to apply the knowledge generated by research.

Research evidence is generated, published, disseminated and integrated into health care practice and policy: SPOR Networks are designed to be national research collaborations involving patients, researchers, health care professionals and policy makers across Canada. Together, they conduct research and help bridge the gap between research evidence and health care practice. Networks aim to generate evidence and innovations that will lead to transformative and measurable improvements in patient health, health care, and the efficiency and effectiveness of service delivery.

Strengthened Canadian capacity in patient-oriented research: With enhanced training and career support provided to a full spectrum of patient-oriented researchers including clinical investigators, methodologists, epidemiologists, biostatisticians, health economists and health-related social scientists, Canada will be more able to attract, train and mentor health care professionals and patient-oriented researchers, as well as to create sustainable career paths in patient-oriented research.

Improved clinical trial environment in Canada through CCTCC, specialized services & methodological expertise: As the Summit Action Plan is implemented, Canada will achieve trust and profitability through operational efficiencies in those areas that impact clinical trial start up times and costs. The clinical trial environment will also be improved because the SUPPORT Units will provide expert assistance in areas such as innovative trial design, data management, statistical analysis, ethics approvals and multi-site management for clinical trials.

3.2.5 Ultimate Outcomes

Ultimately, SPOR aims to improve health outcomes and enhance patients' health care experience through integration of evidence at all levels in the health care system.

Cultural change in patient-oriented research that values the involvement of, and interaction between, all stakeholders: When all participants in patient-oriented research – patients, researchers, health care professionals and decision makers – are trained and supported to work together, and when it happens routinely, the value of collaboration will become apparent and a culture shift will occur.

Improved patient health outcomes: Patient health outcomes will be improved when the patient receives the right care, in the right place at the right time. This will happen when SPOR enables health care professionals and policy makers integrate the latest evidence in their practices and decision making processes.

Enhanced patient health care experience: When patients participate in all aspects of the research and decision making processes, from setting research priorities to implementing evidence into practice, policy makers and health practitioners will be better able to understand their needs and deliver high-quality services responding to those needs. Ultimately, the patient health care experience will be enhanced when health services and products are truly patient-oriented.

Improved Canadian health care system through evidence-based practice: SPOR will enable the timely and efficient translation of research innovations from the research setting to patient care settings, the evaluation and synthesis of existing knowledge, and the proper transfer of existing knowledge to the clinical setting. This will ultimately lead to evidence-informed improvements to the system.

Canada is recognized internationally as a favoured environment for the conduct of clinical trials: By overcoming the barriers identified at the Summit, Canada’s competitiveness in conducting clinical trials will be increased, and it will once again be recognized as an international player in conducting clinical trials.

Appendix B – CIHR Institutes: Engagement with SPOR

CIHR Institute Governance Financial Contributions Scientific Leadership Strategic Linkages
IA

Member of Network Management and Administrative Office PIHCI

PIHCI Network

IA IAB input into the SPOR Networks in Chronic Diseases Funding Opportunity.

IAPH

IAPH initiatives with strategic linkages to SPOR include:

  • Fellowships and New Investigator FOs in FNIM health,
  • Intersectoral prevention research Integrating Indigenous Traditional Ecological Knowledge (Environments.
ICR

 Patient Engagement Champion

ICRH

Clinical Trials Champion (until transition of Scientific Director in 2015)

Consulted with various provincial and territorial stakeholders and engaged various groups in the development of SUPPORT Units

Consulted with various clinical trials experts within ICRH fields to inform the SPOR strategy

Member of an internal working group that evaluated the funding and impact of the Canadian Clinical Trials Coordinating Centre (CCTCC).

ICRH Emerging Networks represent key investment in initiatives with key strategic linkages to SPOR.

IG

Working with Newfoundland and Labrador SPOR SU, which will play an important role with the PerMed RAF initiative.

They will present at the Canadian Human and Statistical Genetics Meeting 2016, IG’s main annual meeting.

IGH

Sex/gender requirements in SU annual performance reporting

Developed sex/gender content for the Networks in Chronic Diseases funding opportunity (e.g. requirements for 'Sex/Gender Champions')

IHDCYH

New Networks Champion (2015).

PIHCI Network

SPOR Networks in Chronic Diseases

Support of SPOR workshop (NeuroDevNet)

Support of SPOR network strengthening workshop

IHSPR

SUPPORT Unit Champion;

Member of the Network Management & Administrative Office - PIHCI

PIHCI Network

Working with IPPH on how best to link PIHCI with SUPPORT Units.

Lead institute on planning for funding opportunity for clinical trial/s of pay-for-success models of delivery

Co-lead on establishment of expedited access to CIHI Dynamic Cohort of high system users. Lead institute on Embedded Clinician Researcher Salary Award

III

Federal Action Plan for Vaccine Research compliments the stated objectives of SPOR.

IMHA

Former Networks Champion

Clinical Trials Champion (2015)

Facilitated the future investment in arthritis research – along with the Arthritis Alliance of Canada in the beginnings of the Networks in Chronic Diseases

INMD

Champion for Capacity Development

PIHCI Network.

SPOR Network in Chronic Diseases

INMD hosted Diabetes and Related Complications National Research Network Strengthening Workshop in January 2014

Met with various SUPPORT Units and presented the SPOR Capacity Development Framework.

INMD Initiatives with Strategic Linkages to SPOR include:

  • A collaboration with the CHILD Foundation (Children with Intestinal and Liver Disease) to support a National Pediatric IBD Network and Data Platform.
  • INMD played a convener role between Health Policy and Patient Engagement Canada Research Chair, and Kidney Foundation of Canada (KFOC) to engage of process of setting research priorities among people on or nearing dialysis using a priority-setting method of James Lind Alliance
INMHA

Member of the Oversight Committee for the ACCESS Open Minds Network.

Played an important role in the launch of ACCESS Open Minds Network by facilitating the partnerships attending meetings and workshops, funding opportunity writing and relevance review

INMHA committed to the SPOR Network in Chronic Disease (Pain) and will be building upon that for a national Pain Consensus Conference.

IPPH

Member of the Network Management & Administrative Office - PIHCI

PIHCI Network

Provided a presentation on implementation science to SUPPORT unit Leads

Working with IHSPR on how best to link PIHCI with SUPPORT Units

Contribute to the development and implementation of Network related activities including several funding opportunities (e.g. FO content development and peer reviewer briefings), performance measurement framework, strengthening workshops with member networks, meetings of interim leadership council,

Fostering linkages between PIHCI and other initiatives (e.g. CBPHC Signature Initiative).

Date modified: