Evaluation of the Drug Safety and Effectiveness Network Program

Table of Contents

List of Common Acronyms

CADTH – Canadian Agency for Drugs and Technologies in Health

CIHI – Canadian Institute for Health Information

CIHR – Canadian Institutes of Health Research

CNODES – Canadian Network for Observational Drug Effect Studies

DSECT – Drug Safety and Effectiveness Cross-Disciplinary Training

DSEN – Drug Safety and Effectiveness Network

DSEN CO – Drug Safety and Effectiveness Network Coordinating Office

DSEN SC – Drug Safety and Effectiveness Network Steering Committee

FCSAP – Food and Consumer Safety Action Plan

F/P/T – Federal/Provincial/Territorial

GoC – Government of Canada

HPFB – Health Products and Food Branch (Health Canada)

HC – Health Canada

INESSS – L'Institut national d'excellence en santé et en services sociaux

KT – Knowledge translation

MHPD – Marketed Health Products Directorate (Health Canada)

OOGP – Open Operating Grant Program

PAA – Program Alignment Architecture

PHAC – Public Health Agency of Canada

PMDSE – Post market drug safety and effectiveness

RRS – Research Reporting System

SPB – Strategic Policy Branch (Health Canada)

STIHR – Strategic Training Initiative in Health Research

TBS – Treasury Board Secretariat of Canada

Executive Summary

The Drug Safety and Effectiveness Network (DSEN) was established by the Government of Canada in 2008-09 as one of a number of initiatives that contribute to meeting the objectives and goals of the 2007 Food and Consumer Safety Action Plan (FCSAP).Footnote 1

DSEN generates and promotes the use of evidence on post-market drug safety and effectiveness (PMDSE) and contributes to increasing capacity to undertake quality PMDSE research. The DSEN program links existing research centres in PMDSE across Canada. The DSEN Coordinating Office is located within the Canadian Institutes of Health Research (CIHR).

As the first evaluation of DSEN, the evaluation focused on assessing the design and delivery of DSEN, measuring progress toward the achievement of its immediate expected outcomes, and identifying areas for improvement. It was agreed that the scope of the evaluation would not include a review of the impacts of a policy and decision-making support enterprise in a research agency. The evaluation covers the time period from the creation of DSEN in fiscal year 2008-09 to end of fiscal year 2012-13 and meets the requirements of the Treasury Board’s Policy on Evaluation (2009) and the Financial Administration Act.

Key Findings

Recommendations

The following recommendations are intended to support DSEN in positioning the program to achieve its objective of better informing pharmaceutical post market drug safety and effectiveness decision-making across the Canadian health care system.

  1. The DSEN CO, in consultation with its partners and stakeholders, needs to examine key design and delivery features of DSEN to identify areas where efficiency and effectiveness can be enhanced. Key areas for further examination include:
    1. Clarify key aspects of DSEN’s operation with program stakeholders
      The clarification and common understanding of several aspects of DSEN’s operation would help strengthen the program:
      • Effective formats for communicating research results in response to Queries to stakeholders according to their different mandates and information needs (i.e., knowledge translation).
      • The prioritization of submitted Queries.
      • The appropriate level of independence of DSEN researchers in the context of their interactions with query submitters to define research protocols in response to Queries.
    2. Establish service standards for the Query submission and response process to clarify expectations for Query submitters and support performance measurement.
      • Decision makers require clear and transparent timelines for the delivery of research results in response to Queries to make timely decisions on the safety and effectiveness of marketed drugs.
      • There is a need for improved dialogue between Query submitters and DSEN researchers at the outset of submission to clarify the information required to respond to the Query and the context and timelines of the decision-making process the research result will inform or support.
      • The DSEN CO should develop and implement service standards, such as target response times, to enable submitters and research teams to establish common expectations and agreement on timelines and milestones, taking into consideration the methodology and scope of the proposed research, at the outset and throughout the Query submission and response process.
  2. The DSEN CO, in consultation with Health Canada and CIHR, should review the current performance measurement strategy to identify changes to better monitor performance against expected outcomes.
    • The DSEN CO should identify the indicators to collect and track information to better monitor, assess and communicate the performance and impact of DSEN. In particular, additional indicators should be developed relating to timeliness, program expenditures, indirect training and capacity development, and the longer term benefits of query responses on the Canadian health care system.
    • In terms of program expenditures, DSEN needs to track and report on expenditures in greater detail in order to: separate out DSEN related expenditures from the broader Targeted Oversight activities of FCSAP; and, map the use of grant funds by research teams to support PMDSE research, capacity development, knowledge translation and network activities.

Management Response and Action Plan

Recommendation Response (Agree or Disagree) Management Action Plan Responsibility Timeline

1. The DSEN CO, in consultation with its partners and stakeholders, needs to examine key design and delivery features of DSEN to identify areas where efficiency and effectiveness can be enhanced. Key areas for further examination include:

Agree

Management supports the findings that the program has been implemented as planned and is showing progress towards meeting its immediate and intermediate outcomes. It is recognized that all the findings raised are valid, but reiterates that they need to be taken in context of an implementation period defined by changes in process. Only now is the DSEN entering a period marked by stability of process and increased knowledge creation/dissemination. As the DSEN program continues to mature, it is Management’s intent to use the findings of this evaluation to streamline and refine process to address efficiency while maintaining and improving delivery to DSEN’s key stakeholders.

Chief Scientific Officer/Vice–President, Research, Knowledge Translation and Ethics Portfolio

September 2015

a. Clarify key aspects of DSEN’s operation with program stakeholders.

The clarification and common understanding of several aspects of DSEN’s operation would help strengthen the program:

  • Effective formats for communicating research results in response to Queries to stakeholders according to their different mandates and information needs (i.e., knowledge translation).
  • The prioritization of submitted Queries.
  • The appropriate level of independence of DSEN researchers in the context of their interactions with query submitters to define research protocols in response to Queries.

Agree

CIHR is committed to continuous improvements in the design and delivery of the DSEN program. The DSEN Coordinating Office (DSEN CO) will continue to focus on increasing understanding across the network to minimize the gap between the distinct stakeholder cultures. The highly integrated knowledge translation model on which DSEN operates will continue to advance researcher and decision maker understanding of the key elements and roles of each player in the overall DSEN program.

An agreement with the Canadian Agency for Drugs and Technologies in Health (CADTH) will allow for greater engagement of provincial and territorial drug plan managers and policy makers working in jurisdictional pharmaceutical policy. Moving forward, CIHR and CADTH will collaborate to see greater responsiveness to the information needs of decision makers responsible for drug plan management and optimal use programs across Canada. This agreement will also increase efficiency in the dissemination and targeting of research outputs to the program’s various stakeholder audiences.

CIHR will also work with the DSEN Steering Committee to gain additional insight on prioritization of safety versus effectiveness research and balancing these investments against capacity development to support future capacity needs.

Chief Scientific Officer/Vice– President Research, Knowledge Translation, and Ethics Portfolio

Implementation of the CADTH agreement occurred between Oct 2013 – Oct 2014.

A review of the DSEN research investment portfolio will be presented to the DSEN SC in Fall 2014.

b. Establish service standards for the Query submission and response process to clarify expectations for Query submitters and support performance measurement.

  • Decision makers require clear and transparent timelines for the delivery of research results in response to Queries to make timely decisions on the safety and effectiveness of marketed drugs.
  • There is a need for improved dialogue between Query submitters and DSEN researchers at the outset of submission to clarify the information required to respond to the Query and the context and timelines of the decision-making process the research result will inform or support.
  • The DSEN CO should develop and implement service standards, such as target response times, to enable submitters and research teams to establish common expectations and agreement on timelines and milestones, taking into consideration the methodology and scope of the proposed research, at the outset and throughout the Query submission and response process

Agree

The DSEN CO and CIHR have continuously taken steps to improve communications and reduce unnecessary bureaucracy throughout the implementation phase. The institution of the Science Advisory Committee has established working relationships that continue to allow for improved alignment between submitted Queries and the conduct of corresponding research. Increased understanding by both decision makers and researchers is leading to improved responsiveness to submitted Queries and building confidence in the network.

On a risk minimization basis, the DSEN CO is examining further reductions in the time to onset of research through the design of its funding opportunities. Where appropriate CIHR will fund teams to conduct research projects from within their team grant (i.e. the model of CNODES). The Rapid Funding for Targeted DSEN Research grant tool will allow for distribution of funds to unanticipated research gaps and mitigate risk from high cost – long term projects.

CIHR is committed to working continuously to improve the timeliness, relevance and utility of DSEN funded research for application in decision making. Building on a workshop held March  2013 to establish agreed principles in knowledge translation (KT) for application across the network, the DSEN CO is developing a network wide strategy to set the stage for dissemination of research results to key stakeholders and beyond to a broader audience. CIHR will champion the strategy’s roll out across the network to see the establishment of collaborative, structured and interactive processes whereby DSEN research results can be taken up and put into action across the health care system.

Chief Scientific Officer/Vice– President Research, Knowledge Translation, and Ethics Portfolio

Starting September 2014, funding agreements for DSEN collaborating teams’ engagement stipulates that teams will produce project management action plans (PMAPS) outlining key milestones and target achievement dates for individual research projects.

CIHR will produce, in collaboration with partners and stakeholders, guidance documents and templates to support and improve network wide KT activities by March 2015. Tracking of their uptake will continue through 2015 and beyond.

2. The DSEN CO, in consultation with Health Canada and CIHR, should review the current performance measurement strategy to identify changes to better monitor performance against expected outcomes.

  • The DSEN CO should identify the indicators to collect and track information to better monitor, assess and communicate the performance and impact of DSEN. In particular, additional indicators should be developed relating to timeliness, program expenditures, indirect training and capacity development, and the longer term benefits of query responses on the Canadian health care system.
  • In terms of program expenditures, DSEN needs to track and report on expenditures in greater detail in order to: separate out DSEN related expenditures from the broader Targeted Oversight activities of FCSAP; and, map the use of grant funds by research teams to support PMDSE research, capacity development, knowledge translation and network activities.

Agree

CIHR (and the DSEN CO) is accountable to Parliament in the allocation of the grant funds which it releases to researchers. It must always balance the efficient operation of its programs with the risks associated with seeing those programs achieve their desired impact. Thus CIHR is committed to establishing, measuring and evolving the design of those elements necessary to match the DSEN program design to the evidence needs of its stakeholders in order to maximize the impact of its investments.

As the DSEN CO is responsible to facilitate the overall collaboration between the various players in the network, and as CIHR and HC are responsible to report on the performance of the DSEN program in general, various performance measurement strategies have been established to track the performance of the operations both internal to CIHR and HC as well as the performance of the collaborating parties and funded researchers. In follow up to the evaluation, the DSEN CO will work with its program partners, CIHR and HC’s evaluation staff, researchers, and stakeholders to review elements of the existing performance measurement strategy to support ongoing program performance monitoring and the next evaluation, to be undertaken in no more than five years’ time.

Chief Scientific Officer/Vice-President Research, Knowledge Translation, and Ethics Portfolio

The DSEN program partners will revise the DSEN Performance Measurement Strategy by end of FY 2015-16.

1. Evaluation Purpose

1.1 Evaluation Purpose, Issues and Scope

The purpose of the evaluation was to assess the program’s design and delivery, performance and relevance to date through a targeted study. Subsequent to the onset of the evaluation, the scope was expanded to ensure that the evaluation would meet TBS evaluation coverage. The evaluation assessed the extent to which the current delivery model facilitates the achievement of the program’s outcomes to identify potential improvements; however, it was agreed that the scope of the evaluation would focus on assessing the achievement of immediate outcomes and not include an assessment of the impacts of a policy and decision-making support enterprise in a research funding agency. The issues and questions addressed by the evaluation were developed based on program considerations and information needs and aligned with core issues outlined in Treasury Board’s Policy on Evaluation and Directive on the Evaluation Function (see Appendix A). The evaluation meets the requirements of the Financial Administration Act and the Treasury Board’s Policy on Evaluation (2009) and covers the activities of the DSEN program from its creation in 2008-09 to the end of fiscal year 2012-13.

1.2 Methodology

In line with Treasury Board’s Policy on Evaluation and recognized best practice in evaluation, a range of methods - involving both quantitative and qualitative data - were used to triangulate evaluation findings (see Appendix A for a full description of the data collection methods used). The data collection cut-off point was March 31, 2013.

1.2.1 Document Review

The review of documents provided information on the history and objectives of the DSEN program as well as insight into changes in the program’s implementation and activities, which contributed evidence addressing some of the evaluation questions. In addition, the document review assisted with the development and formulation of the interview questions. Some of the documents reviewed included: background documentation related to the development of DSEN; Senate reports; peer-reviewed journal articles; and, reports and presentations.

1.2.2 Review of Research Reporting System Progress and Annual Reports

The Canadian Institutes of Health Research (CIHR) formally launched the Research Reporting System (RRS) instrument on March 31, 2011.Footnote 2 CIHR requires Nominated Principal Investigators (NPIs) to report on their CIHR-supported research results via the RRS. The intention is to use these RRS reports for a variety of internal and external purposes in order to obtain better evidence on the effectiveness of CIHR funding programs, advance CIHR's Knowledge Translation mandate, and contribute evidence to enhance CIHR’s accountability within the Federal Government and to Canadians for their investment in health research. CIHR also uses the RRS data for a variety of internal uses, including how to better manage the process of funding health research.

All NPIs on Open Grants and selected Priority Announcements, where the authority to spend funds expires as of March 31, 2011 or later, must submit end of grant reports via RRS. NPIs have 18 months after the end of their grant period to complete their reports, and CIHR provides on-going support to assist NPIs in this task.

The DSEN program was an early adopter of the RRS reporting system. This evaluation uses CIHR’s RRS as one of its lines of evidence. The RRS reports provided data on the outputs and outcomes of both completed research projects from the first cohort of funding and the outputs and outcomes of the seven DSEN supported methodological teams as reported in their first annual reports which also made use of the RRS instrument. These reports were received in the 2012-13 fiscal year.

It is important to note that the submitted RRS responses used by this evaluation are not representative of the overall number of DSEN-related grants and awards, and the following reported data should be interpreted with caution. A limited amount of time has elapsed since program inception and only the 2009 DSEN catalyst grant recipients have completed their projects. This has resulted in not only the availability of a limited number of RRS reports for DSEN/CIHR-supported projects where funding has expired (10), but also only a limited number of Queries being completed (seven) by March 31, 2013.Footnote 3 A further limitation is that the RRS reports completed by DSEN researchers, as well as DSEN’s seven teams’ progress reports, are abbreviated versions of the full RRS instrument, with a reduced set of questions.Footnote 4

Finally, it is important to note that information in the RRS reports is not directly indicative of the responsiveness of the program to its primary stakeholders who submit research questions for address by DSEN.

1.2.3 Key Informant Interviews

A total of 17 interviews were completed with key informants and provided information to address all of the evaluation questions. The interviews were structured to address issues that other lines of evidence could not, or had limited ability to, address. Table 1.1 presents the breakdown of interviews by the three respondent groups.

Table 1.1: Breakdown of Key Informant Interviewees and Response RateFootnote 5
Respondent Groups # of Interviews Completed # of Participants Invited % of Participants Invited Interviewed
Steering Committee Members 5 8 63%
Executive Working Group MembersFootnote i 4 4 100%
PartnersFootnote ii 8 11 73%
Footnote i

Working Group members include individuals from the DSEN CO and Health Canada’s Marketed Health Products Directorate (MHPD) and Strategic Policy Branch (SPB).

i

Footnote ii

Organizations that will assist in the implementation of the initiative and that have complementary interests in the regulatory or research area linked to post-market drug safety and effectiveness. These organizations will directly enable the DSEN in a variety of ways. Partners may include: the research community, particularly organizations associated with the DSEN-funded centres or working in collaboration with the centres on DSEN funded Projects; other publicly funded organizations such as the Canadian Institute for Health Information (CIHI) and the Canadian Agency for Drugs and Technologies in Health (CADTH); data holders both public and private (e.g., P/T drug plans, health insurance providers, academic institutes, health care providers, industry, etc.); and, international organizations (e.g., U.S. Food and Drug Administration, European Medicines Agency); and, associations (e.g., Canadian Public Health Association, Consumers’ Association of Canada).

ii

1.2.4 Case Studies

Two purposively selected case studies were conducted involving a total of six interviews as well as document review to support the background context of the case studies. For each case study, one of the lead researchers was interviewed along with two decision makers engaged with the development of the research question (Query). In addition to interviews, the case studies were informed by a review of the query submission, research report as well as other information provided by the key informants. Table 1.2 presents the breakdown of interviews by the two respondent groups.

Table 1.2: Breakdown of Case Study Interviewees and Response Rate
Respondent Groups # of Interviews Completed # of Participants Invited % of Participants Invited Interviewed
Researchers 2 2 100%
Decision Makers 4 4 100%
1.2.5 Citation Analysis using Google Scholar Data

Raw citation data was used as a proxy to determine the “quality” of the evidence generated or updated in the peer-reviewed journal articles published by DSEN researchers. Google Scholar was used to collect this raw citation data on the approximately forty articles published by DSEN-supported researchers that were related to funding received by the program.

1.3 Limitations

It is important to highlight several limitations, including:

2. Program Profile

2.1 Overview of the DSEN Program

The DSEN program was established by the Government of Canada in 2008-09. The DSEN is one of many initiatives that contribute to meeting the objectives and goals of the 2007 Food and Consumer Safety Action Plan (FCSAP). DSEN generates and promotes the use of evidence on post-market drug safety and effectiveness (PMDSE), and contributes to increasing the capacity to undertake quality PMDSE research within Canada. The DSEN program is a virtual network that links and facilitates coordination among existing research centres in PMDSE across Canada, with its Coordinating Office located within the Canadian Institutes of Health Research (CIHR).

DSEN is a health portfolio initiative, funded by the Government of Canada (GoC) through Health Canada (HC) and CIHR. The DSEN program was allocated $32 million for the period from 2008 to 2013, and ongoing funding of $10 million per year. DSEN has distributed a total of $17,253,145 in grants and awards from FY 2009-10 to 2012-13 (Figure 2.1).Footnote 6

Figure 2.1: Grants and Awards Expenditure by Fiscal Year

Long description: Grants and Awards Expenditure by Fiscal Year

Table 2.1 outlines the DSEN program’s two primary objectives and key activities for achieving these two objectives. The DSEN Logic Model in Appendix B provides a full description of the program’s activities and expected outcomes.

Table 2.1: DSEN Program Objectives and Key Activities
Objective Activities

Increase available new evidence on post-market drug safety and effectiveness to inform drug regulation, public reimbursement and optimal\ prescribing and use of drugs

  • Fund: Collaborative Centres, research projects (Queries / Capacity Development)
  • Facilitate: linkages across the network and between researchers and knowledge users, submission of DSEN Queries
  • Coordinate: network meetings, collaboration between knowledge users and researchers
  • Partner: HC, CIHR Institutes, external stakeholders
  • Communicate: disseminate research results/newsletters/KT products

To increase capacity within Canada to undertake high-quality post-market research in this area

  • Fund: Doctoral Research Awards and New Investigators
  • Facilitate: trainees’ development and participation in network meetings and in Collaborative Centres
  • Partner: Drug Safety and Effectiveness Cross-Disciplinary Training (DSECT) (a former STIHR program)
  • Promote: discussion on national curriculum development for drug safety and effectiveness research

Source: DSEN Dashboard, November 7, 2012

The evidence that is generated by DSEN-supported research complements existing, as well as new post-market surveillance and reporting activities included under the FCSAP and other Canadian research initiatives. This evidence is intended to enable key healthcare decision makers to better determine safety and effectiveness profiles for drugs and to implement measures that will promote their optimal use. A diagram of the life cycle approach model of pharmaceuticals regulation in Canada and DSEN’s role is provided in Appendix C.

As of March 31, 2013, the DSEN program has utilized a variety of funding mechanisms to support PMDSE-related research, including: catalyst grants, team and operating grants; rapid funding grants; salary awards; fellowships; and, doctoral awards.

In 2011, DSEN established seven methodology-specific research teams (research teams) which receive funds to support network activities, capacity development and knowledge translation activities. One team, the Canadian Network for Observational Drug Effect Studies (CNODES), receives funds to support research activitiesFootnote 7; whereas the other six teams must apply for additional funding to support the query-related research they undertake on a query by query basis.

The following are identified partners and stakeholders for the DSEN Program:

Table 2.2 details the status and number of Queries received, as well as the relevant query submitters, as of March 31, 2013.

Table 2.2 Number of Queries Received by Status
Query status Number of Queries Query Submitters
Reported back to decision maker 7 HC, Provincial Ministries of Health, DSEN research teams
Under research 22 HC, Provincial Ministries of Health, CADTH, DSEN research teams
Out of scope 11 HC, CADTH
Pre-query (under development) 6 HC, PHAC, F/P/T drug plan
Awaiting resources to become available 7 HC, CADTH

2.2 Query management process

The DSEN CO has established a formal query management process (Appendix E) with a supporting framework and guidance documentation. Several decision making organizations are currently able to submit queries (e.g., HC, provincial and territorial health bodies, CADTH).

The DSEN CO facilitates discussion of a potential query between the submitting party and its seven funded research teams. These teams then respond, confirming if they are capable of answering it adequately. These discussions lead to the decision makers either formally submitting their final query to DSEN or pursuing another avenue to address it. The final queries are then considered by the DSEN SC for advancement to the DSEN prioritized research agenda.

One of the roles of the DSEN SC is to endorse the results of the query prioritization process to ensure that the most pressing and relevant queries are examined first. The DSEN CO has developed a Multi-Criteria Decision Analysis (MCDA) tool to assist the SC when evaluating and ranking query submissions.

The MDCA tool is a set of weighted criteria that is used to rate feasible queries to be recommended to the DSEN SC for prioritization to the DSEN research agenda. Through extensive consultations between DSEN and provincial and federal representatives, the MCDA tool was developed and piloted by a working group comprised of members from the DSEN CO, HC and provincial health organizations.

As of March 31, 2013, the DSEN SC has accepted all submitted queries, due to the limited number of queries submitted. DSEN expects that the volume of queries submitted by stakeholders will increase as the provinces and other eligible groups gain the awareness, capacity and experience to submit feasible queries. Thus, in time, the SC will consider all new queries and prioritize them based on the available funds.

All DSEN research teams undertaking research in response to a query must develop a proposal for answering the query to be discussed with the query submitter. Once the approach is developed to support the needs of decision makers, the research team submits, as necessary, an application for a rapid funding grant to support the research they will conduct in relation to the accepted query, which is then peer reviewed.Footnote 8

In October, 2012, the Science Advisory Committee (SAC) was formed to help expedite the early stages of query submission by convening preliminary reviews of potential queries. The SAC is charged with providing an informal discussion forum to decision makers and researchers for the development of DSEN Queries. This facilitates the development and submission of scientifically, methodologically, and technically feasible DSEN Queries to the DSEN CO, increases efficiency of the Query management process and enhances the direct interaction and communication between DSEN researchers and decision-makers.

If a proposal is recommended for funding by peer review, the team receives a rapid funding grant to support the research they will conduct in relation to the accepted query. If no proposals are received from the seven DSEN research teams, an open call may be made by the DSEN CO to the general research community, which would also be peer reviewed. All proposals rejected by peer review, as well as queries with no responses, are returned to the submitter, who then decides to resubmit the query or, resolve the query through other methods.

3. Program Design, Delivery and Efficiency

3.1 Key Findings

3.2 Design and Delivery

This section of the report relating to program design and delivery is divided into two parts:

  1. An analysis of the existing design and delivery of the DSEN to assess the extent to which the program has been implemented as planned, including the establishment of a monitoring and management framework, facilitators and hindrances, and identified areas of improvement; and,
  2. An analysis of the cost of program delivery, as planned and in practice to date.
3.2.1 Governance and the extent to which roles, responsibilities, and accountabilities of the health portfolio partners (HC and CIHR) are clearly defined and followed

The DSEN program is a health portfolio integrated partnership between CIHR and HC. Figure 3.1 details the roles and responsibilities of the three partners. HC’s Strategic Policy Branch (SPB) acts as portfolio secretariat for the program providing policy oversight and monitoring the performance and achievement of outcomes of DSEN, and reports this progress under the FCSAP.

Figure 3.1: CIHR and HC DSEN roles and responsibilities
Drug Safety and Effectiveness Networks
Health Canada CIHR

Health Products and Food Branch (HPFB): Lead - Marketed Health Products Directorate

Leads activities to develop HC and PHAC input into the DSEN program prioritized research agenda; and, works towards effective use of DSEN program research findings interdepartmental regulatory decision-making activities

Strategic Policy Branch (SPB): Lead – Office of Pharmaceuticals Management Strategies

Provides policy oversight and horizontal health portfolio coordination for the overall DSEN program

DSEN Coordinating Office

Implements, facilitates and coordinates the DSEN network operations and establishes the research funding mechanisms

Although the DSEN CO has developed Terms of Reference and other governance-related documentation, there is evidence of some confusion among some DSEN SC members regarding the role and function of the SC, which may undermine its functionality.

The opinions of interviewees were mixed as to whether the current governance structure and responsibilities are working, particularly with respect to the SC. Nine out of 15 SC members were interviewed and some interviewees indicated that they were not satisfied with the present role of the SC, nor with the types of discussions and advice being generated, and expressed doubts on whether their opinions were being heard. These same individuals felt that the SC was not fulfilling its role to provide strategic advice and direction. One SC member felt that the discussions were highly technical in nature while a few others felt that a better balance in discussions between safety and effectiveness was needed. A few members expressed the need for a greater level of transparency throughout the process and of not overloading SC members with information at meetings.

Some interviewees felt that the current membership of the SC was appropriate, while others felt that it did not necessarily have the right mix. For those who questioned the mix of current membership, no suggestions were provided about what they would suggest to be a more appropriate mixture of members.

3.2.2 A Planning and Management Framework has been Established

The DSEN CO has been proactive in establishing and documenting the activities of the network (see Appendix B: DSEN Logic Model; Appendix D: DSEN Organizational Chart; DSEN Steering Committee Terms of ReferenceFootnote 9) and developing processes for the management of Queries (see Guidance Document for Submitters of DSEN QueriesFootnote 10; and Query Submission Process Appendix E). While DSEN is still in the process of evolving, the DSEN CO continues to formalize processes to avoid ad hoc decision-making, and to operate in a transparent manner.

3.2.3 Performance Measurement and Evaluation Framework

A Performance Measurement and Evaluation Framework (PMEF) for the DSEN program was developed by the DSEN CO and partners in collaboration with CIHR/HC, and post-market evaluation experts and approved in August 2010.

The PMEF makes use of CIHR’s Research Reporting System (RRS) data for its annual reporting.Footnote 11 Whereas CIHR requires grant recipients to complete an RRS report within 18 months of expiry to use funds, the DSEN CO has initiated a policy that requires DSEN grant recipients to complete their abbreviated RRS reports within six months of expiry to use funds, to expedite the collection of data for reporting purposes.Footnote 12 DSEN supported research teams are also required to submit annual RRS update reports electronically.

In November 2012, the DSEN CO presented its DSEN Steering Committee (DSEN SC) members with a draft dashboard which it plans to use for reporting performance measurement data to the committee on an annual basis.

While the program has taken active steps in developing a detailed performance measurement strategy and while most interviewees felt that the established mechanisms kept them informed, a few interviewees suggested that DSEN was overly “bureaucratic” and that more emphasis, in terms of performance reporting, should be on outcomes, rather than process. Furthermore, the data collection activities undertaken by this evaluation have revealed performance measurement gaps that have reduced the ability to accurately determine program performance and, in particular, the efficiency of the program (both in terms of delivery of query responses/evidence and financial). An assessment of the current utility of the data collected by the PMEF to support future evaluations should be undertaken to identify areas in the framework which need improvement.

The findings from this evaluation will help inform how to best revise the PMEF to ensure that it provides the DSEN SC, DSEN program partners and stakeholders with the information needed to make program related decisions and support the strategic priorities of DSEN.Footnote 13

3.3 DSEN Program Costs and Efficiency

3.3.1 DSEN Program Costs

The DSEN program receives an annual budget to support the activities of the network, the DSEN CO, and the DSEN-related activities of HC. Table 3.1 presents the program delivery costs associated with the program from fiscal year (FY) 2008-09 to 2012-13. It is important to note that the FCSAP reporting structure is such that DSEN specific spending by HC is aggregated with spending on other Targeted Oversight activities (e.g., periodic safety update reporting) and, as result, this analysis uses the Treasury Board Secretariat of Canada (TBS) allotments as proxy estimates of DSEN-related expenditures. While not preferred, the approach is acceptable and any variance is not expected to be material.

The proportion of funding spent on program delivery as a whole is higher than other programs delivered at CIHR (Table 3.1) but is largely in line with the percentage approved in the program’s TBS submission. The initial high percentage of delivery costs to total program expenditures can be explained by the resources required to start up a novel research network that was designed to meet and respond to the needs of specific stakeholders, rather than supporting investigator driven research. Resources were used to refine and develop the network’s structure in terms of funding instruments, the type of methodology required to address Queries and the establishment of the seven teams. The percentage has decreased over the period under review; the proportion spent by the DSEN CO and by HC on the delivery of the DSEN is approximately evenly split, with the most recent year’s proportions being 10.6% and 9.3%, respectively.

Table 3.1: Program delivery costs of the DSEN program, 2008–2013
FY CIHR Actual Program Delivery Expenditures HC TBS Program Delivery AllotmentsFootnote iii Combined CIHR and HC Delivery Figures Grants & Awards Expenditures Percentage of Total Program Expenditures Spent on Program Delivery TBS Approved Percentage of Total Program Expenditures Spent on Program Delivery
2008-09 $365,750Footnote iv $634,250Footnote iv $1,000,000 $0 100% 100%
2009-10 $614,864 $701,272 $1,316,136 $1,350,000 49% 66%
2010-11 $1,033,143 $1,046,510 $2,079,653 $1,144,089 65% 31%
2011-12 $1,122,463 $1,022,132 $2,144,595 $6,066,726 26% 26%
2012-13 $1,129,949 $986,343 $2,116,292 $8,692,330 20% 26%
Total $4,266,169 $4,390,507 $8,656,676 $17,253,145 33% 31%
Footnote iii

Note that DSEN specific amounts were unable to be separated from the totals reported by HC. Therefore, this analysis used the TBS amounts awarded as proxy estimates of DSEN-related expenditures at HC. Program expenditures/allotments include funding for the following elements: salary, operations and management, Employee Benefit Plan (20%), and accommodation (13%). Grants and awards are excluded.

iii

Footnote iv

$365,750 was transferred to CIHR for initial setup costs related to the DSEN CO in February, 2009.

iv

While data availability restricted the granularity of our analyses, more detailed costing information is available for fiscal year 2012-13. Table 3.2 presents the planned and actual operational expenditures for DSEN for fiscal year 2012-13.

Table 3.2: DSEN Program delivery expenditures (Planned and Actual) 2012-13
Item Planned Actual
DSEN/CIHR programming $1,578,141 $1,129,949
Health Canada programming $862,166 $845,997
DSEN/CIHR internal services $0 $0
Health Canada internal services $124,178 $18,928
Total expenditures $2,564,484 $1,994,874

Table 3.3 details the ratio of expenditures dedicated to the administration and delivery of the program. The program realized a retained surplus of $704,480, with the majority of the retained surplus coming from the DSEN CO, including reduced salary requirements (the DSEN CO has one of its allocated full-time equivalent staff (FTE) unfilled due to CIHR’s 2012-13 vacancy management policy, and a second FTE absorbed by CIHR corporate), reduced Operations and Management costs, and reduced salary related costs due to the vacancies (i.e. benefits and accommodation).Footnote 14

Table 3.3: Cost of program delivery related expenditures as a percentage of total budget (2012-13)
Element Amount
Allocated budget (A) $10,000,000
Actual Spend (B) $10,687,204
Overspend $687,204
Re-profiling of funds (C) Footnote 15 $1,391,684
Retained surplus ([A+C]-B) $704,480
Overall program delivery costs as a percentage of total 19.9% ($1,994,874/$10,687,204)
CIHR-related program delivery costs as a percentage of total 10.6% ($1,129,949/$10,687,204)

In the future, the savings realized by the program (largely related to staffing vacancies both within CIHR and HC) may be reduced if the DSEN CO receives increased numbers of Queries which would require hiring the program’s full allocation of FTEs in order to ensure the quality of the support for network activities and as short of a turn-around time as possible for submitted Queries. Ongoing program monitoring and the next evaluation should examine the impact this may have on the effectiveness of the program.

It is important to note that the $8,692,330 in grants and awards was used to support both the program’s query-related research and capacity development activities. While the DSEN CO has provided funds to support its seven research teams to conduct PMDSE research in Canada, it also invested $3,343,219 in direct capacity development, awarding bridging grants to early career PMDSE researchers and supporting doctoral students through doctoral awards (38.5%). Furthermore, five of the DSEN research teams reported spending $551,292 on stipends to provide training opportunities for students from all levels of postsecondary education (e.g., Bachelors, Masters, Doctorates and Postdoctoral Fellows).Footnote 16 These funds came from their respective Team Grants.

Previous evaluations of CIHR programs have also included opportunity costs related to the time peer reviewers spend reviewing for CIHR. Peer reviewers spent, on average, approximately six hours reviewing DSEN-related proposals per Rapid Funding review panel convened (each review panel has three reviewers/readers).Footnote 17 This amounts to $372 per reviewer per application (rounded to the nearest dollar), or, a monetized cost of $1,117 per Rapid Funding application (rounded to the nearest dollar). DSEN held review sessions for 13 rapid grant applications in the 2012-13 fiscal year,Footnote 18 which represents a total monetized cost of peer reviewer time spent reviewing applications for funding to support prioritized query research of $14,510. It is important to note that this cost is not paid by DSEN, CIHR or HC, but represents a sacrifice made by the peer reviewers.

While the monetized cost of peer reviewer time is negligible in comparison to the overall budget for DSEN and, as such, has a limited impact the efficiency of the program, the costs represent time taken away from the peer reviewers’ primary task — to conduct research (most peer reviewers are PMDSE researchers). This time is in addition to other peer reviewer commitments these individuals may have. Most of the researchers who serve as reviewers for DSEN also participate in other peer review committees for CIHR and other organizations. Therefore, it is important to keep the time reviewers spend on peer review to a minimum, so that they can devote their time to research.

3.3.2 Efficiency of the DSEN Program

The assessment of the efficiency of the DSEN program involves assessing the program’s resource utilization in relation to the production of outputs and progress toward expected outcomes. As outlined in the TBS Directive on Evaluation Function, the assessment of program performance requires the demonstration of efficiency and identifies the need to compare program outputs and outcomes with program expenditures as one way to assess efficiency (see Appendix A).

Previous evaluations of CIHR programs have used the number of applications received by a program in a given fiscal year to determine the cost-efficiency of the program (i.e., program costs/# of applications). This methodology is not appropriate in the context of DSEN because unlike many of CIHR’s programs, such as the Open Operating Grant Program (OOGP) which has two regular annual competitions with application deadlines in March and September, the DSEN program receives applications on an ad hoc basis and holds funding competitions on an as needed basis.Footnote 19

This evaluation estimates the program’s efficiency by looking at the total operational budget for the period between FY 2009-10 to 2012-13 and key outputs of the DSEN CO for the program. The total expenditures for this period were used as the denominator, while the total numbers of Queries received/accepted/answered were used as numerators. The results provide an indication of the costs, to date, associated with the production of indicators (i.e., number of grants and awards, number of Queries received, accepted and answered) related to two key program outputs: funded research activities; and new post market drug safety and effectiveness evidence.

It is also important to note that the costs per output may seem rather high, but that these figures include the costs associated with the start-up of the DSEN program and that its seven supported research teams were only formed over the 2011 calendar year (one was formed in January while the remaining six teams were established in September). Given that, it is expected that subsequent efficiency analyses conducted as part of program monitoring and the next evaluation should reveal a lower cost per output.Footnote 20

In total, the DSEN CO has distributed a total of 101 grants and awards, as of March 31st, 2013. The total cost, per output (grants and awards), was $79,430, with the total cost per query received being $151,367 and the cost per query answered being $1,146,061 (Table 3.4).

Table 3.4: DSEN efficiency calculations of the program
Output NumberFootnote v Cost per output (total expenditure for FY 08–09 to 12–13 divided by number of outputs)
Total number of grants and awards 101 $79,430
Number of projects related to Queries undertaken 62 $129,394
Number of Queries received 53 $151,367
Number of Queries accepted 36 $222,845
Number of Queries answered 7 $1,146,061
Footnote v

Some Queries generated multiple research projects.

v

It is important to note that the costs associated with Query responses are likely over-estimated due to the nature of the program and the activities undertaken by DSEN CO. Integrated knowledge translation is a fundamental component of the delivery of DSEN and requires regular interactions amongst the DSEN CO, the methodological research teams and research-users to ensure that the research conducted addresses identified information gaps and is disseminated to relevant user groups. The DSEN CO facilitates and tracks the exchange of information and results between the research teams and relevant decision makers as well as hosts regular events (e.g., semi-annual Network meetings) to disseminate research results and to provide further opportunities for discussions between DSEN researchers, decision makers and relevant stakeholders. Integrated KT approaches to research are also required within the research teams. It is expected that the teams contribute to the development of strategies for knowledge translation within DSEN and will continue to participate in KT activities supported through existing and established methods and channels. The allocation of FTE activity related to the extensive KT activities is not currently tracked which prevents the exclusion of these activities and their associated costs in the efficiency calculations presented above.

3.4 Factors Facilitating or Hindering Program Design and Delivery

3.4.1 Innovative Query research funding design

The DSEN program currently uses two primary funding instruments for maintaining its network and supporting query research — team grants and rapid funding grants. Since 2011, DSEN has supported seven teams of researchers with various areas of specialization in different research methodologies of PMDSE.

Each research team receives funds to support networking, knowledge translation activities, and capacity development. The Canadian Network for Observational Drug Effect Studies (CNODES) team receives additional funds to support research activities.Footnote 21 The maintenance of these seven teams provides HC and other query submitters a readily available pool of experts that can respond rapidly to emerging drug safety and effectiveness issues.

The program initially used CIHR’s Catalyst Grant funding instrument between 2009 and 2011 to engage PMDSE researchers in responding to Queries until such time as the DSEN research teams could be established. While it was expected that this funding instrument would be more responsive than other CIHR funding instruments in extant, there were substantial delays to the onset of funding, due to the ad hoc nature and timing of query submissions and the time required to launch corresponding catalyst competitions.

After the establishment of the DSEN funded research teams, the DSEN CO subsequently developed its own funding instrument, the Rapid Funding grant, to support the research activities of the six teams that must apply for funds to address Queries on an as-needed basis. The Rapid Funding granting instrument is designed to deliver PMDSE research funds in a more expedient manner as compared to the Open Operating Grant Program or Catalyst grants (see section 4.3. for a discussion of the timeliness of query responses).

To do this, the Rapid Funding grant has more tightly specified application requirements and requires less time for review while retaining the academic rigor of the review process. For example, peer review by teleconference reduces the time it takes for a prioritized query to be researched by one of DSEN’s established research teams. In 2012-13, DSEN CO received and reviewed 16 applications through the Rapid Funding tool, of which nine projects were funded for a total cost of $1.3 million over three years. The DSEN CO continues to fine-tune the review process to strike a balance between reduced application completion burden yet ensuring that applications have sufficient data for the peer reviewers to adequately evaluate the proposals.

3.4.2 Safety versus Effectiveness

Health Canada was the lead for the Treasury Board submission for the DSEN program, in partnership with CIHR. As the lead, HC began to accumulate potential Queries, which were submitted to DSEN in its first year of operation. As outlined in Table 3.5, HC has submitted the majority of Queries to date, with many of the Queries submitted by HC focused on drug safety due to HC’s role to regulate drugs. Table 3.6 presents the focus of prioritized Queries from 2009-10 to 2012-13, with sixteen Queries focused on drug safety, seven focused on safety and effectiveness and eight focused on effectiveness.

Table 3.5: Number and Origin of Submitted Queries as of March 31st, 2013
Year HC Other DemonstrationFootnote vi
< April 1, 2010 6 0 0
2010–2011 6 2 2
2011–2012 2 5 6
2012–2013 11 2 0
Total 25 9 8
Footnote vi

The first queries that were researched by DSEN were actually developed prior to the launch of the network. These initial queries were used as a demonstration of the network’s ability to respond to decision-maker submitted PMDSE research needs.

vi

Table 3.6: Focus of Prioritized Queries – 2009-10 to 2012-13
Focus of Prioritized QueriesFootnote vii Number of Queries
Safety 16Footnote viii
Safety and Effectiveness 7
Effectiveness 8
Footnote vii

Query submitters indicated whether their query was safety or effectiveness related, or both, upon submission. Not all submissions indicated their focus.

vii

Footnote viii

This total includes some Queries that were initially prioritized but later found to be not feasible or no longer relevant. Therefore, they total more than 25.

viii

The opinions of interviewees were mixed about the appropriateness of the blend between safety and effectiveness of prioritized Queries. Some of the interviewees suggested that the current focus of the studies undertaken by DSEN researchers is appropriate, while other interviewees expressed concern that there is not enough balance between the two types of Queries. Furthermore, one interviewee expressed concern that the MDCA tool developed by the DSEN CO may favour the prioritization of effectiveness studies over safety studies. Therefore, it would be of benefit to re-examine the MDCA tool to determine if the tool could be revised or if another approach is applicable. The DSEN CO is aware of this issue and is currently discussing the review of the MDCA tool with HC and other stakeholders.

3.4.3 Funding Mix

While interviewees agreed that DSEN must fund both post market research and capacity/infrastructure development, there was no agreement on what should be the appropriate distribution of funds. Based on data for the 2012-13 fiscal year, the current distribution of funds between post-market research and capacity/infrastructure development is approximately two-thirds (62% or $5,349,111) to one-third (38% or $3,343,219), respectively; it is expected this ratio will likely increase in favour of research support as more Queries are submitted and prioritized.Footnote 22 It is important to note that the research funds given to the CNODES team are not included in this calculation because the research funds for CNODES were included in their initial Team Grant and were distributed in 2010-11. The addition of these funds would increase the ratio in favor of research.

There has not been much concern about the mix of how funding is distributed up to now but it may become a concern as DSEN approaches full capacity (in terms of responding to Queries). A few interviewees suggested that there may be too much emphasis on capacity development.Footnote 23

As well, interviewees suggested that there may not be sufficient funds to meet the expectations of all stakeholders. While the current level of funding is adequate for the current level of prioritized Queries, it may not be sufficient to support long term demand if the number of Queries submitted increases. The first few years of the DSEN program saw a surge of Queries submitted, due to submitters developing a list of possible Queries in anticipation of DSEN’s launch, but the more recent years have seen a slight decline in the overall number of both submitted and prioritized Queries, with a decrease in the Query rejection rate. Furthermore, there are some Queries which have multiple responses. Ongoing performance measurement and future evaluation will be better able to determine the steady state of the annual Query submission rate; thereby, allowing for improved tracking of changes in the numbers of Queries being submitted and prioritized as well as allocation of resources required to meet program delivery requirements.

3.4.4 Autonomy, independence, and the location of DSEN

The understanding and definition of what independence means varies from stakeholder to stakeholder. Testimony provided to the Senate Standing Committee on Social Affairs, Science and Technology, which held hearings on pharmaceuticals over 2012 and 2013, suggests that some stakeholders are concerned that the pharmaceutical industry may have an influence on CIHR policy decisions due to their partnerships with the national research funding agency. One witness testified that he had several misgivings/reservations about DSEN’s degree of independence from industry (GoC, 2012a). This witness expressed concerns that CIHR’s increasing collaboration and partnering with industry may adversely affect the independence of DSEN. On the other hand, a few witnesses from industry suggested that they were uninformed of DSEN’s research until the network releases its findings. They expressed significant concern that they had no influence in the monitoring of post-market drugs by DSEN. The Senate’s report on the testimony they received stated that industry representatives expressed a desire, at a minimum, to at least be informed when one of their drugs is under investigation (most witnesses were satisfied with the current practice of not informing manufacturers) (GoC, 2013a, 15).

There was some disagreement among interviewees on the appropriate level of independence of DSEN funded researchers in terms of the involvement of stakeholders in defining specific research protocols to respond to submitted Queries. Some interviewees expressed a desire for greater involvement in the development of the objectives and methodology used for researching prioritized Queries. While, on the other hand, there was some concern among other interviewees that decision makers may not have enough technical knowledge to be engaged in the research process in a detailed manner. Additionally, one of the evaluation’s interviewees suggested that DSEN may require increased resources in the future (to address increasing numbers and potentially, more complex, prioritized Queries) and that it could find partnership funding from industry since they felt that there was a sufficiently robust firewall between partnered industry funding contributions and decisions relating to how the contributions are utilized. While there may be divergent views about the independence of DSEN from the influence of industry and the autonomy of DSEN within CIHR, there seems to be sufficient evidence that this concern may be unsubstantiated.

Related to the issue of independence is the location of DSEN within CIHR. The selection of CIHR as a host was arrived at through a systematic process where it was determined that locating DSEN in CIHR was more compatible than the alternatives because it enabled DSEN to take advantage of CIHR’s existing relationship with researchers working in post market drug research, expertise in supporting health-related research, and ability to support the highest quality of research to ensure that the DSEN-supported findings would be seen as rigorous. Evidence collected suggests that the majority of key informants interviewed for this evaluation did not have an issue with DSEN being located in CIHR. Only four of the key informant interviewees discussed this issue with differing opinions about the locating of DSEN in CIHR, with their responses equally falling on a continuum that ranged from it being “critically important” that DSEN remain in CIHR to the opposite extreme where one interviewee opined that it may be better to have DSEN as a separate entity in and of itself.

While the one interviewee’s comments on their support for the continued location of DSEN in CIHR was brief and stressed the need for DSEN to remain part of CIHR, the other three comments explained their positions in greater detail. One interviewee suggested that some stakeholders, during the development of DSEN, did not fully understand the procedures and processes in place at CIHR which affect the timeliness and responsiveness of the network which resulted in early challenges. Another interviewee posited that one way to increase the timeliness and responsiveness of the network would be to have CIHR continue to fund DSEN research, but to have DSEN established as a separate entity which would result in greater independence. Finally, one interviewee suggested a new and independent organization be set up which would be free of any limitations caused by CIHR’s structure and the restrictions around how it provides funding to support research. The interviewee posited that this would result in a more timely and responsive body which would allow it to enter into research contracts that could potentially increase the timeliness of the delivery of research results.Footnote 24

Therefore, while the opinions and arguments of the two interviewees who discussed the relative merits of moving DSEN out of CIHR are relevant, they do not appear sufficiently strong to necessitate a reconsideration of the host analysis conducted. Furthermore, the Canadian Senate has re-affirmed the decision to locate DSEN within CIHR. The recently completed Standing Senate Committee on Social Affairs, Science and Technology’s examination and review of prescription pharmaceuticals in Canada announced, after hearing lengthy testimony from various experts, that it felt that DSEN should remain a part of CIHR since, the opinion of the members of the Senate committee, CIHR is a trusted organization in Canada and that there were no pressing need to re-locate DSEN at the time (GoC, 2013A, 19).

3.4.5 Provincial and Territorial Engagement

The DSEN CO has identified the need to increase provincial and territorial (P/T) engagement in the query process in its November 2012 Dashboard presented to the DSEN SC. Some interviewees suggested that the difficulty in engaging provincial decision makers results from the fact that the capacity to meaningfully engage with the program varies significantly across provinces and territories.

Another factor that may contribute to this challenge is that the provinces, while integrally involved in the work leading towards the establishment of a national PMDSE network, had limited involvement in developing the implementation plan and timeline for DSEN’s establishment, which was initially led by HC. Furthermore, one interviewee also posited that an additional cause could be that provincial and territorial decision makers are less familiar with the DSEN query process.

While there is some evidence that provincial engagement is a challenge, the DSEN CO (and CIHR) has taken actions to better engage them. The Network Advisory Committee (NAC), established after the TBS submission, provided advice for DSEN’s early implementation and engaged regional, provincial and international stakeholders. In addition to more actively engaging provincial and territorial decision makers, the DSEN CO has adopted a number of strategies, including: creating guidance documents that describe how to submit quality QueriesFootnote 25; including provincial and territorial decision makers in the development of the MDCA query prioritization tool; and, establishing links with the CADTH to increase awareness among provincial decision-makers (e.g., CADTH has included DSEN as a point of discussion on the agenda for some of its meetings with the provinces).

4. Program Performance

4.1 Key Findings

4.2 Progress Towards the Achievement of Immediate Outcomes

4.2.1 New knowledge of drug safety and effectiveness has been generated

The review of grant recipients’ progress and final reports that CIHR has received to date reveals that the knowledge creation stemming from DSEN funded projects, as measured by journal articles and other dissemination activities, has been substantial (see Table 4.1). Eight of the ten projects related to Queries from the 2009 initial cohort reported publishing at least one article (the remaining two reported submitting articles). Overall, there were a total of 161 presentations conducted on these 17 projects over the span of three years, with each grant recipient reporting at least one presentation (this could include poster or oral presentations at symposiums, conferences, workshops, etc.). One-third (33.5%) or 54 of these presentations were given in countries other than Canada, including: the United States of America, Mexico, the United Kingdom, Spain, Germany, Switzerland, France, China, Australia and Singapore. This productivity is on par with the performance of other CIHR-supported researchers, as reported in collected end-of-grant reports.

While the publication of journal articles is a proxy that is used to measure knowledge creation, it does not indicate whether or not the knowledge is useful and is making an impact on decision-makers. One way to measure whether the knowledge generated by a specific project is having an impact on others is to look at whether it has been cited in other articles, using citation analyses. The assumption made here is that the more an article is cited, the more it is influencing others, be it positively or negatively. Therefore, the raw citation scores of the articles listed by respondents were collected, using Google Scholar, as an indicator of whether the listed articles were having an impact on the academic community.Footnote 26 The analysis of Google Scholar data suggests that the knowledge generated by DSEN-supported research is having an impact on the broader academic community (see Table 4.1).

It is important to note that the number of citations an article receives is affected by how long it has been publically available. For example, an article that was published in 2011 has had 2.5 years to gain citations while an article published in 2013 has had a maximum of six months to gain citations. This, as well as the merit of the article/usefulness of the information it reports are the two reasons why there is great variation in the citation counts (as indicated by the large standard deviation values). The raw citation count ranges from zero citations to 40, with zero, one and four being the most frequent number of citations received.

Table 4.1: Number of Knowledge Products by TypeFootnote ix
Group Journal articles Journal articles submitted Average number of raw citations (standard deviation, number of articles) Book/Book chapter published Published Reports Invited Presentations Other Presentations
2009 Catalyst grant cohort
(n=10)
21Footnote x 8 7.75 (6.18,
n=20)
1 1 37 12
Team grants - annual report
(n=7)
22Footnote x 12 10.26 (13.4,
n=23)Footnote xii
0 2 87 26
2009 catalyst cohort and team and directed grants
(n=17)
43 20 8.36 (9.58,
n=42Footnote xi)
1 3 124 48
Footnote ix

Note that these are sums, and not averages. The average is not given since the n is too small.

ix

Footnote x

One RRS respondent reported two articles in both a final report and a progress report. It was decided to retain these two counts even though it may seem like double counting. Researchers use multiple sources of funding to support their research and therefore, it was assumed by the evaluator that funding from the catalyst grant and the larger team grant both contributed to the project from where the research results stemmed. This conclusion was arrived at by determining that the researcher did not double report articles on a consistent basis, and restricted it to just these two titles.

x

Footnote xi

Note that one article was listed twice. Therefore, while it was included in the analyses for the Catalyst grant cohort and the Team grant separate calculations, it was only included once for the combined totals.

xi

Footnote xii

Note that one of the articles reported for a Team grant was only released in June of 2013. Therefore, it has not generated any citations to date, although its articles on its findings have appeared in numerous media sources (i.e., newspapers and magazines).

xii

Note that some Queries are submitted to DSEN by decision makers in order to confirm internal analysis, or to validate synthesis of research findings. Therefore, not all research outputs from DSEN will be of interest for journal editors, so in addition to the articles captured by RRS reports, other content will rest unpublished and must be disseminated directly to relevant stakeholders.

Furthermore, it is important to note that in the particular context of DSEN’s mandate and objectives, these commonly employed measures do not directly map the utility of or impact of DSEN research outputs to those submitting Queries or health care decision-makers more broadly. The utility of the research outputs rests on the specificity of the research result in responding to the central uncertainty underlying any query submitted, which can only be captured by qualitative measures of the submitter’s satisfaction with the research product.

Finally, it should also be noted that much of this data is derived from the initial years of DSEN’s operations whereby a portion of research was conducted in response to open calls to the research community to test the success of researchers proposing research projects to highly targeted questions from decision makers. These open calls did not place requirements on researchers to provide their research outputs directly to the submitter of each research question, which later became a condition of funding for the research teams which comprise the DSEN core methodological capacity. Although the DSEN program partners have worked to identify the research outputs from these early open grants, which are directly relevant to the Queries submitted, not all knowledge produced by these early responses were shared directly with decision makers. Rather, most of this knowledge was disseminated through traditional academic mediums. Furthermore, these data represents knowledge generation and dissemination to a very broad audience and, as such, it is only partially representative of new knowledge provided to the submitters of DSEN Queries.

4.3 Responsiveness of Network and Timeliness of Query Responses

As of March 31, 2013, the DSEN program has received a total of 53 Queries (with the majority of the Queries originating from HC).Footnote 27 Of the total number of Queries received, 36 Queries (or 68%) have been prioritized with the remaining Queries either under development (six or 11%) or out of scope/misaligned with the network’s capacity to conduct the required type of research (11 or 21%). Of the 36 prioritized Queries, 22 are under research, seven are completed and seven are prioritized with proposals for funding under review.

Most key informants interviewed agreed that the delivery of DSEN funded research findings have not met the expectations of decision makers with respect to timeliness to date. Interviewees, primarily decision makers, cited several factors that may explain the timelines, including:

It is important to note that there may be differing interpretations of what timeliness means. The first DSEN Funding Opportunity’s (FO) call for proposals cautioned that any proposed research would need to be completed within a short time frame; however, this was defined differently across grant type. For example, the 2009 catalyst grants supported one-year projects whereas the Team grants were defined as demonstration projects to be completed within a two year timeframe.

While there may be some variation in defining timeliness, there are indications that the DSEN CO is taking actions to make the query response process more time efficient. The DSEN CO has taken steps to reduce the time between query submission and the onset of research. It has established a Scientific Advisory Committee which reviews newly submitted Queries to ensure that they are feasible and specific enough to be responded to by the network teams. As well, the DSEN CO has streamlined the application submission process for securing rapid funding grants. While there is no way to confirm that these actions are solely responsible for reducing the time between query submission and the onset of research, an analysis of data indicates that the gap has decreased substantially (Figure 4.1). The average gap has dropped from a high of approximately 24 months in 2009, the first year of operation of DSEN, to approximately five months in 2013.

Figure 4.1: Number of months between query submission and onset of research by submission yearFootnote xiii,Footnote xiv

Long description: Number of months between query submission and onset of research by submission year

Footnotes

Footnote xiii

These data are based on projects with the related to prioritized Queries which had the shortest gap between submission and onset of research. Furthermore, work on one 2011 query response was begun prior to its prioritization. The principal investigator was aware of the query prior to its submission resulting in a gap from submission to onset of research of zero months. Therefore, it was decided to remove this case from the analysis since it was not reflective of the other queries.

xiii

Footnote xiv

Note that these values are based on the first response to a prioritized Query, regardless of the number of responses it had generated. Furthermore, DSEN receives Queries which are sometimes returned to the submitter due to their not being feasible, given the network’s resources.

xiv

Modifications to the current performance measurement indicators are necessary in order to collect the required data that can better determine if the program is resulting in the timely delivery of PMDSE information to query submitters.

It is also important to recognize that early in the implementation of DSEN there was a need to develop lines of communication between decision makers and the DSEN-supported researchers. These parties have spent much time discussing the decision-makers’ needs and expectations in relation to their submitted Queries. Although this creates some delay in the onset of examining the Query due to the level of engagement needed, this is a value added activity, which over time has been becoming increasingly efficient.

The following case study of a query pertaining to Isotretinoin illustrates the importance of collaboration between HC and DSEN in refining a research query.

Isotretinoin is a medication used mostly for severe acne. While highly effective, it has significant documented adverse effects, the best-known and most dangerous being birth defects due to in utero exposure. There are measures in place to decrease the risk of pregnancy in women using Isotretinoin. However, Isotretinoin-exposed pregnancies are still reported internationally. The purpose of HC’s research proposal was to determine more accurately how often pregnancy occurs among Canadian women using Isotretinoin.

HeaIth Canada submitted this query in August 2010. It was the last of 23 topics submitted in the first two years of DSEN’s roll out, received five months prior to the identification of the first DSEN funded team – CNODES. CNODES subsequently identified the research question as one of six potentially actionable Queries to be undertaken in its start-up phase. In early 2012, CNODES contacted HC to discuss possible refinements to the research question. In summer, 2012, following correspondence between DSEN, CNODES and HC, CNODES indicated that it could take on the query as redefined by HC. Most recently CNODES indicated that it had begun to work with various provinces to assess feasibility and limitations around data collection.

Some of the lessons learned from this query include the following:

While the researchers have not completed their research at the time of data collection, they have identified unintended benefits stemming from the work and DSEN as a whole. The DSEN network has allowed researchers to better access medical data that enables them to undertake complex studies that had not been feasible prior to the establishment of DSEN. DSEN therefore provides a way for HC and other P/T health authorities to obtain answers to important drug safety questions that it has identified, answers which may not otherwise be obtainable at the time of decision-making.

4.4 DSEN Responsiveness and Delivery of Evidence – Suitability of methods and mediums

The RRS and qualitative data collected by the evaluation identify four primary methods used to share DSEN research results with stakeholders: reports; direct communication between researchers and decision makers/query generators; academic publishing; and, presentations.

To date, most of the information shared with present query generators (i.e., HC) has taken the form of academic papers which are highly technical in nature. The current format of the responses to Queries has not challenged the capacity of the query submitters. For example, one interviewee noted:

Right now the majority of the research coming back is in the format of a scientific journal paper… which is not particularly problematic for people working in the decision-making realm who have that same level of scientific training.

While this format suits the needs of the primary query submitter, it may not be suitable for other decision makers such as representatives from the provinces or the Canadian Agency for Drugs and Technologies in Health. Further, future decision makers may require different response formats to meet their needs and capacity.

Additionally, the current media/fora used to convey research findings are not necessarily suitable for broader dissemination, to other stakeholder groups. As one interviewee suggested, one of the deliverables that should be produced by researchers is “a lay person understandable press release”. The DSEN CO is currently developing a knowledge translation strategy that includes plain language findings documents that can be posted on DSEN’s website to broaden the dissemination of their network’s research results beyond the query submitter and improve transparency for the program. Furthermore, some interviewees expressed a desire for all DSEN SC members to receive query responses, regardless of whether or not their constituency submitted the initial query.

The following is an example of one of DSEN’s research teams, CNODES, being proactive in their dissemination of findings beyond the query submitter. CNODES developed a publication announcement for an article they produced on the results of a DSEN-supported study examining statins and published by the British Medical Journal, one of the world’s most prestigious medical journals. News of their results was published in such places as the UK’s Daily Mail, Heartwire; Pharmacology Weekly; Forbes; CBC; and Pulse.Footnote 28

Another example of a proactive dissemination strategy implemented by DSEN-supported researchers is an examination of the use of second-generation anti-psychotics by children (usually in treatment of Attention Deficit Hyperactivity Disorder). The researchers found that “there is good evidence that second generation antipsychotics cause both metabolic and extrapyramidal side effects in children. This evidence comes from randomized controlled trials of these medications in children, and prospective cohort studies”.Footnote 29 Besides providing the query submitter with a response, the research team has published several journal articles and has developed several drug specific safety-related clinical guidelines which have been made available to the public on a publically accessible website.Footnote 30 Each guideline provides drug-specific recommendations for monitoring drug safety in children, since “the risk and timing of side-effects vary by medication”.Footnote 29

Findings from key informant interviews also suggest that no single response format or template will meet the needs of all decision makers/query generators. In light of this, some interviewees suggest that increased dialogue is necessary between researchers and the query generators to better ensure that the response to each query is provided in the appropriate medium, format and has the necessary content to meet the needs a specific decision makers/query generator.

4.5 Use of DSEN-generated Evidence to Inform Decisions

As of March 31, 2013, the DSEN research teams had delivered responses to seven Queries. There was only a limited amount of evidence collected through key informant interviews and document reviews relating to how this information provided evidence for the decision-making of query submitters. Available evidence suggests that DSEN query responses have had a limited impact on decision making. Members of the Executive Working Group and SC members from HC identified a few areas where DSEN-generated evidence had an impact on HC decisions, policies and practices. Two of the examples provided by interviewees were:

Findings from the case study of the response to a query regarding the comparative effectiveness of anticoagulants, used to reduce the risk of thrombosis in at-risk patients, indicates that DSEN-generated evidence contributed to CADTH’s decision to issue a recommendation, through its Common Drug Review for the continued use of the standard anticoagulant. The query was submitted by a provincial drug plan manager on August 31, 2011, and focused on the comparative effectiveness of new oral anticoagulants (NOACs), such as Dabigatran, Apixaban and Rivaroxaban to Warfarin which is the standard anticoagulant and has been used for over 50 years. These drugs are used to reduce the risk of thrombosis in at-risk patients. There were several reasons for the submission of the query including:

One of DSEN’s seven research teams, the Canadian Collaboration for Drug Safety, Effectiveness and Network Meta-Analysis, submitted a proposal and were funded to conduct a comparative study of the cost-effectiveness and safety questions of NOACs in comparison to the standard - Warfarin. The Query response was completed and submitted to CADTH in April 2012; CADTH participated in the study since it felt that the research was of interest to one or more P/T jurisdictions. Warfarin was found to be just as effective as other anticoagulants at a much lower cost (a dose of Warfarin is approximately 1/3 the cost of a similar dose of Dabigatran). The research team disseminated the results of their study through a detailed report available on CADTH’s website (Wells, 2012), through presentations and through CADTH’s one-page summary of the study’s key findings.Footnote 31

The results contributed evidence informing a Common Drug Review recommendation, in September 2013, on the continued use of Warfarin. The study’s results, through the recommendation, are informing provincial drug plans in their negotiations with respect to the price of NOACs. As well, the findings have been translated into specific clinical recommendations targeted towards prescribers. Furthermore, the project resulted in capacity development by directly involving several PhDs and Master’s students; one of whom is exploring knowledge dissemination techniques to make the results of network meta-analysis and economic evaluations more understandable.

It is also important to note that the limited evidence of contribution towards or influence on policy for all seven submitted responses does not mean that the responses have not had any influence. For example, evidence submitted by DSEN researchers could result in no modification or impact on policy if their findings suggest there is no cause for concern in relation to the topic of the submitted query; however, the decision to take no action is still a policy decision that was based, in part, on evidence submitted by DSEN. Furthermore, it is important to note that the DSEN program is relatively new and the research to respond to many of the submitted Queries was initiated by DSEN teams in 2011 and later. In addition to the relatively recent launch of DSEN program, is the time required for findings submitted in response to a query to influence decision-making at a query generator’s organization. As one interviewee noted:

“[Findings have a] trickle-down effect in terms of policy changes… is sometimes a bit slow so first of all, the users will have to think about the data to decide what kind of regulatory changes might be necessary so I would guess it would probably be a year anyway till we see major regulatory changes on the basis of this data.

The time between the submission of the query results for the NOAC study to CADTH and the issuing of the recommendation seems to support this statement.

4.6 Coordination, Collaboration and Linkages to Address PMDSE Evidence Gaps

DSEN supported research has been conducted by researchers working in collaborations with each other. The ten completed catalyst grants, issued before the establishment of the seven methodological teams, had an average of approximately four co-applicants/principal investigators. In 2011, the DSEN program distributed multi-year grants to support seven teams of researchers distributed across Canada that focus on five specific methodologies, (see Figure 4.2). Furthermore, DSEN researchers collaborate with researchers from other countries (such as Greece, the United Kingdom and the United States of America). The seven teams are comprised of approximately 170 researchers.

Figure 4.2: Geographic Distribution of the DSEN Teams’ Researchers

Long description: Geographic Distribution of the DSEN Teams’ Researchers

The DSEN program has also taken proactive steps to expand its network, by conducting numerous networking and engagement activities, both nationally and internationally (see Table 4.2). The DSEN has also engaged stakeholders from HC, the Provinces and Territories, health technology assessors (CADTH and L'Institut national d'excellence en santé et en services sociaux), and international organizations (such as bilateral meetings with the US Food and Drug Administration and the European Medicines Agency, and the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance and the USFDA Mini-Sentinel program).

Table 4.2: Number and Type of Network Activities
Type of Network Activity Number of Occurrences
DSEN Steering Committee Meetings 6
Network Development (Semi-Annual Network meetings) 4
Scientific Discussions (ISAC Meeting, Query Teleconferences SAC) 7
Internal Meetings (HC Science Forum, Presentations to HC directorates, Presentations to HC DSEN Team, HC Discussion Forum) 6
Conferences (CADTH, CAPT, CSPT, Priorities, SAPEC, DIA, Post Approval Summit) 14
Outreach Events (Funding Opportunity Workshop, Methodology Workshop, Query Submission workshop) 4
Webinars (Query Submission, Capacity Building) 2
Publications (CIHR DSEN Newsletter, Framework and Guidance Documents for the Submission of DSEN Queries) 6

All key informant interview participants agreed that DSEN has made an important contribution towards increasing communication between decision-makers, CIHR, HC and PMDSE researchers. Interviewees reported that DSEN has created a national forum/body in PMDSE that has brought greater coordination to these previously fragmented bodies/groups. The key achievements of DSEN identified by interviewees include: better leveraging of expertise across various groups of PMDSE researchers; developing PMDSE capacity in Canada; and, fostering culture change by encouraging its researchers to be more willing to share preliminary results with decision-makers.

In addition, DSEN has succeeded in better coordinating and facilitating access to drug databases. To date, DSEN has secured access to the drug use records of over 40 million individuals for its research teams through access to data captured by provincial drug plans in Canada, the United Kingdom’s Clinical Practice Research Datalink (CPRD) and the United States’ MarketScan Database. For example, one query examining Isotretinoin that is nearing completion relies on data from several registries from multiple provinces. The lead investigator suggested that the use of these collated data would not have been possible in the absence of DSEN.

To date, DSEN has established a limited number of linkages and partnerships with other national and international organizations in the domain of PMDSE research. Most interviewees acknowledged that only a few years has passed since DSEN’s launch and that it takes time to establish solid partnerships and linkages, but note that there has been some progress in this area. For example, the DSEN CO is in the process of developing a collaboration agreement with CADTH to support P/T decision makers by identifying and refining jurisdictional evidence needs, producing research (clinical and economic) and disseminating findings to the broader decision making community. Specifically, DSEN and CADTH will collaborate on a joint project to:

The intent of this partnership is to have CADTH act as a point of contact for the provinces and territories. It is expected that having CADTH as a central query submitter will facilitate P/T engagement, allowing them to rely on one entity with relevant expertise that can work with them to develop appropriate query submissions. Using CADTH to represent provincial issues will likely reduce the challenge DSEN currently has in engaging the provinces individually, given that each province is autonomous and has its own policies, priorities and infrastructure. Findings from the case study of the response to the NOAC query provide a good example of the synergies that can be realized by having CADTH act as a point of contact for federal/provincial/territorial drug plans PMDSE questions and Queries.

4.7 Direct and Indirect Capacity Development

Although the evaluation focused on the program’s outputs and immediate outcomes, data was collected that indicated that there has been some progress towards achieving the intermediate outcome of increased capacity in Canada to undertake high-quality post-market research. DSEN supports capacity development directly through awards for individual researchers, and indirectly, by providing funding for research projects that develop capacity through the involvement of students, trainees and other researchers/stakeholders. Capacity is further developed by negotiating access to additional data sources (which increases the network’s ability to undertake a variety of analyses) and by developing new methodologies which can then be applied to Queries that were previously unanswerable.

As outlined in Table 4.3, DSEN has invested approximately $4 million in direct support of capacity development. Recently, the network has invested $1.8 million over five years in the career development of six new investigators, starting in 2012-13. Furthermore, DSEN assumed responsibility for funding the Drug Safety and Effectiveness Cross-Disciplinary Training (DSECT) program in 2012, which supported 28 fellows in drug safety and effectiveness research in 2012.Footnote 32 In addition, DSEN has further supported capacity development by providing opportunities for trainees to participate in conferences, promoting trainees and new investigators participation in network events and by supporting the development of a national curriculum for training new drug safety and effectiveness researchers.

Table 4.3: Investment in direct support of capacity development (2008–2012)
Type of grant Number of applications funded Total dollars invested
PA - New Investigators Awards 6 $1,800,000
Catalyst Grant - Post-Market Drug Safety and Effectiveness in Populations Underrepresented in Clinical Trials 15 $996,000
STIHR - DSECT 1 $970,476
PA - Bridge Funding 6 $572,743
Total 28 $4,339,219

The seven DSEN research teams also have a responsibility to provide indirect capacity development within their team funding envelope through funding research projects involving, and providing training opportunities to students, trainees and researchers.

Given the focus of the evaluation on the immediate outcomes and the information available via the Research Reporting System (RRS) reports submitted, there is limited evidence to assess indirect capacity development that may have occurred. In the case of RRS reports, the limited information regarding capacity development is due to a reduction in the amount of information recipients of smaller grants, which include many of DSEN’s early grants, are required to report on in the end of grant RRS report.Footnote 33 A few RRS annual research team reports however did include some brief information on indirect capacity development in the narrative portion of the report. For example, the Canadian Network for Observational Studies (CNODES) reported, in their first annual report, that they had 35 trainees involved with the work they have conducted so far. Overall, there are insufficient data to assess the full performance of DSEN’s indirect capacity development initiatives to date.

It is important to note that DSEN also interprets capacity development as increasing data access for its researchers (thus improving their capacity to conduct larger studies). An example of DSEN’s success in this area was the establishment of a harmonized database of provincial pregnancy cohorts which has the potential to increase the speed with which DSEN’s research teams can respond to Queries that involve/target pregnant women (see Isotretinoin case study in Section 4.3). Furthermore, DSEN also includes methodological advancements in its definition of capacity development. Seven papers related to methodological advancement have been published by its teams since 2011, with four having between 7-12 citations each (as reported by Google Scholar). This is further evidence that DSEN is showing progress towards achieving this intermediate outcome.

5. Relevance

5.1 Key Findings

5.2 Continued Need

The evaluation evidence indicates there is a continued and likely increasing need for active surveillance to ensure the safety and effectiveness of marketed drugs. The DSEN program works to address this need by coordinating, generating and promoting the use of PMDSE research to inform F/P/T decision-making. Active post-market surveillance mechanisms, such as DSEN, are important for patient safety because decision makers want to ensure that the most effective pharmaceuticals for the safe treatment of illnesses are used in order to support the cost-effective, evidence-informed operation of medical drug plans.

The Canadian Institute of Health Information has estimated that Canadians spent $29.3 billion on prescription drugs in 2013 (CIHI, 2014), which represented 13.9% of all health expenditure for that year. Furthermore, the recent Canadian Rx Atlas, 3rd Edition (Morgan et al, 2013) re-affirms the importance of pharmaceuticals. It reports that three out four Canadians fill out at least one prescription every year. The report also predicts that the costs associated with prescription drugs in Canada will sharply increase with the introduction of increasingly specialized, and expensive, drugs to treat specific ailments. In this context, there is a need to identify the most efficacious drugs to enable P/T drug planners to keep their programs as efficient as possible while ensuring that Canadians can access the safest, most effective pharmaceuticals. Findings from the case study of the response to the Dabigatran Query provide an example of how DSEN is addressing the need for evidence to inform drug plan management by providing evidence that second generation anticoagulants are not necessarily more efficacious than Warfarin, which has the potential for considerable cost-saving for P/T drug plans given that the cost per dose of Warfarin is one-third that of second generation anticoagulants.

Findings from key informant interviews indicate that DSEN is needed, with several interviewees suggesting that it would be detrimental to return to how PMDSE research was conducted prior to the launch of DSEN. According to interviewees, the loss of DSEN would result in fragmented and uncoordinated research which could reduce the availability of credible research for decision makers meaning they would have to make decisions based on the resources and information that are available to them. While some PMDSE research would likely continue, it would be researcher driven and less likely to address the information needs of decision-makers since the research would not be focused to address questions (queries) posed by decision makers.

The increased creation and use of PMDSE knowledge is an important mechanism to reduce the number of Adverse Drug Reactions (ADRs) and their associated harms and healthcare costs. Adverse Drug Reactions (ADRs) are rare, serious and unexpected risks of a drug when used as authorized for the market or in cases where health care professionals make treatment decisions and prescribe pharmaceuticals “off-label” (i.e., outside the manufacturers identified or authorized use for the approved drug).

In most cases, ADRs are voluntarily reported when an adverse reaction to a drug/medical treatment is observed (only manufacturers are required to complete ADR reports) — it is estimated that ADR reports only capture between one percent and ten percent of ADRs (Wiktorowicz et al. 2010; Hazell & Shakir, 2006). The failure of passive ADR reporting to fully capture adverse drug reactions has led to experts in post market drug safety calling for “active” surveillance (see Adler Group, 2006; Health Council of Canada, 2010) to provide early detection of safety problems and prevent greater public harm. In addition, the medical healthcare costs associated with the treatment of ADRs are significant, for example, the estimated medical healthcare costs associated with the treatment of ADRs among seniors (66 years of age and older) was $35.7 million in 2007. DSEN’s active surveillance through its PMDSE research on drugs identified by F/P/T decision makers helps to reduce the uncertainty about drug response, allow for better risk-benefit profiling for decision makers, and identify dangerous issues before large number of users are affected.

The recent Standing Senate Committee on Social Affairs, Science and Technology (SOCI) report on pharmaceutical drugs, Prescription Pharmaceuticals in Canada: Post-Approval Monitoring of Safety and Effectiveness (GoC, 2013b), and the Government of Canada’s response to the report, suggest that PMDSE evidence is still needed by F/P/T decision makers and that DSEN is well situated to provide it. Specifically, the SOCI report recommends the expansion of DSEN’s mandate, citing a need for PMDSE data on prescription drug effects on sub-groups of Canada’s population (e.g., pregnant women, children, Aboriginal people) and indicates that a formal mechanism be established whereupon DSEN can ensure that its recommendations (based on the findings of research conducted by its methodological teams) are translated into action by relevant regulatory bodies; and that the program be made permanent.

5.3 Consistency with Federal Government Roles and Responsibilities

The evaluation findings show that the DSEN program’s role to fund and facilitate PMDSE research is consistent with the roles and responsibilities of the Government of Canada.

The DSEN program closely aligns with CIHR’s mandate, as per the Canadian Institutes of Health Research Act, is “to excel, according to internationally accepted standards of scientific excellence, in the creation of new knowledge and its translation into improved health for Canadians, more effective health services and products and a strengthened Canadian health care system" (Bill C-13, April 13, 2000). DSEN directly contributes to the achievement of this mandate by facilitating the creation, dissemination and use of health-related knowledge, as well as the development and maintenance of Canadian health research capacity, within the area of PMDSE. In addition, the aforementioned SOCI report (GoC, 2013a) acknowledges the federal government’s role in supporting PMDSE research and recommends that DSEN be established as a permanent entity and have on-going and sustained funding which is consistent with a 2010 report of the Health Council of Canada which stated: “The federal government needs to commit to secure, stable and ongoing funding in order for the DSEN to be able to adequately plan and carry out long-term research” (2010, p. 42). The government’s response to the Senate report agreed, in principle, with the committee’s findings. Key informant interviews and case studies also corroborate these findings.

5.4 Consistency with Government and Agency Priorities

The FCSAP has three action areas: Active Prevention; Targeted Oversight; and, Rapid Response. The DSEN program is one key element which assists in ensuring that FCSAP’s goal of achieving “Targeted Oversight” for health products. DSEN does this by generating new evidence to support the ongoing assessment of the risk-benefit profile of drugs after they enter the market and by building capacity. Health Canada’s recent Report on Plans and Priorities (RPP) for 2014-15 affirms their regulatory role in the governance of the safety of products including food, pharmaceuticals, medical devices, natural health products, consumer products, chemicals, radiation emitting devices, cosmetics and pesticides.Footnote 34 Therefore, DSEN’s strategic direction continues to support HC’s priorities.

In addition to the HC priorities, the goals of the DSEN program support and contribute, either directly or indirectly, to two of the three advantages outlined in the federal government’s Mobilizing Science and Technology to Canada’s Advantage (S&T Strategy) (Industry Canada, 2007 & 2009) — Knowledge Advantage and People Advantage. The program has contributed to the Knowledge Advantage, which identified basic and applied research in many areas (including health) as a sub-priority, and has helped maintain “Canada’s international reputation for research excellence” (Industry Canada, 2009) through the creation of new health-related knowledge about the safety and effectiveness of drugs in Canada. Its unique and innovative approach has garnered worldwide interest, as evidenced by the numerous countries where DSEN-supported researchers have made invited presentations on their research and by the international news coverage of some of the findings of DSEN research teams.

In terms of the People Advantage, DSEN’s capacity development activities have helped “develop and maintain Canada’s health research capacity” and has contributed to the building of “the best-educated, best-trained and most flexible workforce in the world” (Industry Canada, 2009). For example, DSEN supported research and funding opportunities have directly supported over 170 PMDSE researchers through team grants, funded an additional seven new investigators through awards and bridging grants, and indirectly developed capacity by training graduate and post-graduate students through their participation in the research teams.

The evaluation findings indicate that the DSEN program has contributed to the achievement of CIHR’s goals outlined in its strategic plan Health Research Roadmap: Creating innovative research for better health and health care. The Roadmap outlines a set of activities linked to four strategic directions:

DSEN’s investment in PMDSE capacity development contributes towards maintaining a sustainable world-class research enterprise. Its support of applied research into PMDSE contributes and accelerates the capture of health and economic benefits, for example, the use of DSEN generated PMDSE evidence will likely lead to increased efficacy and efficiencies in the management of F/P/T drug plans. Finally, the continued improvement in the mechanisms used by the DSEN CO to support PMDSE research in Canada contributes towards the CIHR’s overall organizational excellence.

6. Conclusion and Recommendations

6.1 Conclusions

The evaluation found that the DSEN CO has established Canada’s first national PMDSE network within the planned timelines, supported the development of new PMDSE knowledge and capacity building in Canada, and has made considerable efforts at developing and establishing management and performance protocols and procedures to enable DSEN to achieve its immediate and intermediate outcomes.

The establishment of DSEN has resulted in the creation of a national PMDSE forum and has brought greater coordination to PMDSE-related research activities. According to interviewees, this has been achieved through better leveraging of expertise across various groups of researchers, developing PMDSE capacity in Canada, and fostering culture change by encouraging an increased willingness among researchers to share preliminary results with decision-makers. There is, however, evidence that some primary stakeholders are not yet entirely clear on the role and function of DSEN and their relevant roles and responsibilities within the program. Specifically, there are three aspects of the program which require clarification: the roles and responsibilities of the DSEN Steering Committee; the time taken to prioritize and respond to queries; and the level of independence of DSEN’s researchers.

During the period under review, the DSEN CO established seven methodological-specific research teams and received a total of 53 Queries of which 36 have been prioritized for research during the period under review. Of the 36 prioritized Queries, research is complete for seven (12 Queries were completed as of September 2014), is ongoing for 22, and is planned pending resources for the remaining seven Queries. To date, DSEN research has informed a few PMDSE decisions relating to the confirmation of a risk and making recommendations to final labeling for a drug, informing a policy decision and contributing to a Common Drug Review recommendation; however, this should be considered in the context of the relatively recent launch of the DSEN program, the small number of completed Queries and the time required for the response to a Query to influence decision-making. Overall, the DSEN program has distributed a total of 101 grants and awards, with 62 of these being used to support projects related to prioritized Queries and 39 used to support networking, knowledge translation and capacity development.

While most interviewees expressed opinions about the timeliness of the delivery of evidence and the need to track timeliness, there are no data on timeliness that is systematically collected to date. Therefore, there is a need to strengthen the communication and tracking of information processes around the timelines of query submissions and responses. In addition, many of the DSEN grants analyzed in the reporting period were only required to complete abbreviated versions of the RRS report, which do not collect training and capacity development related data. This makes it difficult to estimate the total number of trainees (a key measure of capacity development). As a result, there is a need for the identification and implementation of indicators to better measure the program’s impact in the area of training and capacity development.

The assessment of program efficiency conducted by this evaluation were limited due to the inability to separate out the DSEN component of HC expenditures related to Targeted Oversight and, as a result, had to be estimated, with the exception of the 2012-13 fiscal year. Over the period under review, the estimated total delivery costs of DSEN were 33% ($8,656,676/$25,275,571) of total program expenditure and decreased from 100% in 2008-09 (the first year of operation) to 20% in 2012-13. The DSEN CO and HC related delivery costs were approximately equal over this period: 16% ($4,266,169/$25,909,821) and 17% ($4,390,507/$25,909,821) of program expenditures, respectively. For fiscal year 2012-13, the delivery costs of the DSEN CO were 10.4% ($1,129,949/$10,808,622) of total expenditures and total delivery costs, including both DSEN CO and HC expenditures were 19.5% ($2,116,292/$10,808,622) of total expenditures. Going forward, 2012-13 cost data can be used as a baseline for the program within the Treasury Board allocations for operating and grants and awards expenditures and improvements should be made to the tracking and reporting of program expenditures across program activities areas, such as in capacity building, evidence generation, and knowledge translation.

The DSEN program is working to address the continued need for the active surveillance of drug safety and effectiveness in Canada. Its goals and objectives are consistent with the roles and responsibilities of the federal government and align with Health Canada (FCSAP) and the federal governments’ priorities on targeted oversight of health products, as well as CIHR’s strategic directions.

6.2 Recommendations

The following recommendations are intended to support DSEN in positioning the program to achieve its objective of better informing pharmaceutical post market drug safety and effectiveness decision-making across the Canadian health care system.

  1. The DSEN CO, in consultation with its partners and stakeholders, needs to examine key design and delivery features of DSEN to identify areas where efficiency and effectiveness can be enhanced. Key areas for further examination include:
    1. Clarify key aspects of DSEN’s operation with program stakeholders
      The clarification and common understanding of several aspects of DSEN’s operation would help strengthen the program:
      • Effective formats for communicating research results in response to Queries to stakeholders according to their different mandates and information needs (i.e., knowledge translation).
      • The prioritization of submitted Queries.
      • The appropriate level of independence of DSEN researchers in the context of their interactions with query submitters to define research protocols in response to Queries.
    2. Establish service standards for the Query submission and response process to clarify expectations for Query submitters and support performance measurement.
      • Decision makers require clear and transparent timelines for the delivery of research results in response to Queries to make timely decisions on the safety and effectiveness of marketed drugs.
      • There is a need for improved dialogue between Query submitters and DSEN researchers at the outset of submission to clarify the information required to respond to the Query and the context and timelines of the decision-making process the research result will inform or support.
      • The DSEN CO should develop and implement service standards, such as target response times, to enable submitters and research teams to establish common expectations and agreement on timelines and milestones, taking into consideration the methodology and scope of the proposed research, at the outset and throughout the Query submission and response process.
  2. The DSEN CO, in consultation with Health Canada and CIHR, should review the current performance measurement strategy to identify changes to better monitor performance against expected outcomes.
    • The DSEN CO should identify the indicators to collect and track information to better monitor, assess and communicate the performance and impact of DSEN. In particular, additional indicators should be developed relating to timeliness, program expenditures, indirect training and capacity development, and the longer term benefits of query responses on the Canadian health care system.
    • In terms of program expenditures, DSEN needs to track and report on expenditures in greater detail in order to: separate out DSEN related expenditures from the broader Targeted Oversight activities of FCSAP; and, map the use of grant funds by research teams to support PMDSE research, capacity development, knowledge translation and network activities.

Appendix A: Evaluation Questions and Methodology

DSEN Evaluation Framework

Evaluation Questions   Indicators Methodology/
Collection Methods
Performance (effectiveness) - Achievement of Expected Outcomes (TBS Core Issue #4)
Immediate
Q1. To what extent is research responsive to priority post-market drug safety and effectiveness evidence needs of decision-makers? Q1.1 #, proportion and range of research questions responding to the research agenda priorities that received funding, as well as the prevalence of unanswered research agenda Queries Document Review
Q1.2 Proportion of key informants who perceive that DSEN program research responds to the priority research questions in a timely manner

Interviews of:

  • key informants
  • FPT decision-makers
Q1.3 Proportion of key informants satisfied with the overall applicability/relevance of DSEN program funded research

Interviews of:

  • FPT decision-makers

Interviews of:

  • Key informants
Q2. To what extent has evidence on drug safety and effectiveness available to decision-makers increased? Q2.1 Number and types of funded research projects that were concluded and contributed to new evidence that address the DSEN program research agenda

Document Review

RRS progress and final reports

Q2.2 Number of knowledge products produced (both raw and normalized by grant duration) in comparison to open grants of similar duration Analysis of RRS progress and final reports
Q3. To what extent has the use of DSEN-generated post-market drug safety and effectiveness evidence to inform decisions increased? Q3.1 Evidence of increased awareness and /or uptake of DSEN program information by key informants

Interviews of

  • FPT decision-makers
  • CADTH staff

Case studies

Document Review

Q3.2 Evidence that the knowledge translation message and medium are tailored to the target audience

Interviews of:

  • key informants
  • FPT decision-makers
  • CADTH staff
  • HC and CIHR Program staff
  • Researchers

Case studies

Analysis of RRS progress and annual reports

Event survey data – reports from fora and workshops

Q4. To what extent is there greater coordination and collaboration to address post-market drug safety and effectiveness evidence gaps? Q4.1 Number and type of linkages established (including national and international)

Document Review

Event survey data – reports from fora and workshops

Analysis of RRS progress and annual reports

Case studies

Interviews of:

  • FPT decision makers
  • CADTH staff
  • Researchers
  • HC and CIHR program staff
Q4.2 Key informant perceptions that there is greater coordination and collaboration to address gaps in post-market drug safety and effectiveness evidence

Interview of:

  • key informants (DSEN SC)
  • DSEN Ex WG
Design, Delivery and Efficiency (TBS CORE Issue #5)
Q5. Has the DSEN program established a clear planning and management framework and are roles and responsibilities clear? Q5.1 Extent to which organizational governance and reporting structures are in place and functioning well

Interview of:

  • key informants (DSEN SC)
  • DSEN Ex WG
  • FPT decision makers
  • HC and CIHR program staff
  • CADTH staff

Document Review

Q5.2 Extent to which roles, responsibilities, and accountabilities of the health portfolio partners (HC and CIHR) are clearly defined and followed Document Review (e.g. DPR/RPP, TBS submission)
Q5.3 Performance measurement strategy exists, has been implemented, and provides management with the information needed to make program related decisions.

Document Review

Interview of:

  • key informants
  • DSEN Ex WG
  • HC and CIHR program staff researchers
  • researchers
Q6. What factors facilitate or hinder the achievement of program results? Q6.1

Program design and delivery facilitators identified

Program design and delivery challenges identified

Interview of:

  • DSEN Ex WG
  • HC and CIHR program staff

Case studies

Document Review

Q7. Are the most appropriate and efficient means being used to achieve the outcomes, relative to alternative design and delivery approaches? How could its efficiency be improved? Q7.1 Perception of alternative approaches

Interview of:

  • key informants (DSEN SC)
  • DSEN Ex WG
  • FPT decision makers
  • HC and CIHR program staff
  • CADTH staff
Q7.2 Key informant perceptions of the efficiency and economy of the DSEN delivery model (e.g., time from addition of questions to research agenda to the availability of findings etc.)

Interview of:

  • key informants (DSEN SC)
  • DSEN Ex WG
Q7.3 Key informant perceptions on whether the program is being managed efficiently and effectively

Interview of:

  • key informants (DSEN SC)
  • DSEN Ex WG
Q7.4 % of funds expended versus the amount received by fiscal year

Document Review

RPP/DPR

Q7.5

Total costs associated with the running of DSEN for FY 2012-13

  • Costs per application
  • Proportion direct costs make of overall cost

Administrative and financial data

Interview of:

  • DSEN Ex WG
  • HC and CIHR program staff

Survey of reviewers and applicants

Relevance (TBS Core Issues #1, #2 and #3)
Q8. Does the DSEN program continue to be consistent with government-wide priorities and the strategic priorities of Health Canada and CIHR? (TBS Core Issue #2) Q8.1 Degree of alignment with Departmental and Agency Strategic Outcomes

Document Review

Interview of:

  • HC and CIHR senior executives
  • DSEN Ex WG
Q8.2 Degree of alignment with federal government priorities

Document Review

Interview of:

  • HC and CIHR senior management
  • DSEN Ex WG
Q9.Does the program address an actual and continued need of Canadians? (TBS Core Issue #1) Q9.1 Key informant perceptions on the actual and continued need for DSEN

Document Review

Interview of:

  • key informants (DSEN SC)
  • DSEN Ex WG
  • FPT decision makers
  • HC and CIHR program staff
  • CADTH staff
  • Researchers
Q9.2 # of drug safety and effectiveness research questions submitted by key informants Document Review
Q10. Is the Government of Canada’s role in the program appropriate? (TBS Core Issue #3) Q10.1 The role and responsibilities of the federal government in delivering the program and the opinion of the key informants on that role.

Document Review

Interview of:

  • key informants (DSEN SC)
  • DSEN Ex WG
  • FPT decision makers
  • HC and CHIR program staff
  • CADTH staff
  • Researchers

Evaluation Methodology Description

Consistent with TBS guidance and recognized best practice in evaluation (e.g., McDavid & Hawthorn, 2006), a range of methods will be used to triangulate evaluation findings. The approach of using multiple methodologies involving both quantitative and qualitative evidence is designed to ensure that the evaluation findings are robust and credible and that valid conclusions can be drawn about the performance of the programs.

Document Review

This component will provide background information on the history and objectives of the DSEN program as well as insight into any important shift or change that may have occurred during the program implementation and delivery of services. It will also contribute evidence for answering some of the evaluation questions and issues. Furthermore, the document review will assist with the development and formulation of the interview questions. The following are examples of the relevant documents that will be included in the review:

The document review may also include relevant documents related to the design, implementation, impacts and evaluations of similar funding programs to allow an analysis and comparison of similar initiatives in other jurisdictions including those internationally, if available.

Interviews

A total of 17 interviews served as an important source of information for the evaluation by providing information that was used as a line of evidence for all evaluation issues and questions. The interviews were structured to address issues to which the other lines of evidence could not, or had limited ability to, contribute. Only two of the 17 interviewees were from the DSEN CO, with the remaining 15 interviewees belonging to various stakeholder groups. The interviewees were divided into three distinct groups:

Participants who were on the Steering Committee but also sat on the working group were included in the analysis for the working group. There were five participant interviews of members of the Steering Committee. Two individuals were senior policy decision makers and three were academics/researchers. The senior policy makers have held their positions for less than three years while the academics/researchers have been in their roles for over ten years. Two of these individuals cited involvement in the conceptual work that preceded the establishment of the DSEN initiative.

There were four participant interviews of members of the Working Group; two DSEN staff and two staff from Health Canada. Both DSEN staff members had been with the initiative from the outset. One of the Health Canada staff members had over 20 years seniority in Health Canada while the other had four years’ experience. Out of the four Working Group members, two had been involved in the conceptual work that preceded the establishment of the DSEN initiative.

There were eight participant interviewees identified as partners; some of whom also sat on the Steering Committee. Two interviewees represented provincial agencies, one a national agency and five Health Canada. Their experience in post-market research ranged from two to 10 years.

Research Reporting System Performance Measurement Data

The Canadian Institutes of Health Research (CIHR) formally launched the Research Reporting System (RRS) instrument on March 31, 2011. CIHR requires Nominated Principal Investigators (NPIs) to report on their CIHR-supported research results via the RRS. The intention is to use these RRS reports for a variety of internal and external purposes, including to obtain richer evidence on the effectiveness of CIHR funding programs, to advance CIHR's Knowledge Translation mandate, and for contributing evidence towards CIHR’s accountability within the Federal Government and to Canadians for their investment in health research. CIHR also uses the RRS data for a variety of internal uses; including to better manage the process of funding health research.

All NPIs on Open Grants and selected Priority Announcements where the authority to spend funds expires as of March 31, 2011 or later must submit end of grant reports via RRS. NPIs have 18 months after the end of the grant period to complete their reports, and CIHR provides on-going support to assist NPIs in this task.

The RRS tool is accessible through CIHR’s ResearchNet platform which enables the pre-population of certain fields in the online survey with data provided at the grant application stage. The tool is organized into seven distinct sections (each with several sub-sections): (a) Profile and Grant Information; (b) Research and Knowledge Translation Practices; (c) Research Findings; (d) Research Capacity and Training; (e) Advancing Knowledge; (f) Broader Impacts; and (g) Research Context.

Google Scholar Data

Google Scholar was used to collect raw citation data on the approximately forty articles published by DSEN-supported researchers that were related to funding received from the program. Google Scholar has been an alternative, and free, source of citation data since 2004. One of the benefits of Scholar, besides being free, is that it captures a wide array of materials within its analyses including “journals, repositories (RePEc, Arxiv and Social Science Research Network), databases (Cochrane database of Systematic Reviews), conference proceedings (IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Proceedings of the ESSCIRC, Proceedings of SPIE, AIP Conference Proceedings) and working papers (NBER Working Paper Series)” (Cabezas-Clavijo & Delgado-López-Cózar, 2012, p. 3).

The broader inclusion of citation sources by Google Scholar makes comparison of raw citations collected through other databases like Scopus or Web of Science difficult, since these data sources include only a selected number of the most prominent journals (most being English-language journals) (personal communication with Jean-Pierre Robitaille, a bibliometrician at Observatoire des sciences et des technologies, July 29th, 2013). While there may be a divergence in the number of raw citation scores an article may have, the relative ranking of the authors are similar, regardless of the data source. One study (Meho & Yang, 2007) suggested that while the absolute number of citations received by a paper may differ between Google Scholar, Scopus and Web of Science, the ranking of researchers through the use of Google Scholar and the merged total number of citations in Scopus and Web of Science were similar. A second study suggested that the increased number of citations recorded by Google are not error, but likely due to the inclusion of a broader source of data sources (books, thesis, etc.) (van Aalst, Jan, 2010). Furthermore, the differences are not dramatic, since all three citation databases use academic journals as the basis of their calculations (personal communication with Jean-Pierre Robitaille, a bibliometrician at Observatoire des sciences et des technologies, July 29th, 2013). The most serious critiques of Google Scholar data are the difference in search results when using key word searchers (there is greater convergence between Scopus and Web of Science), as well as the increased country ranking results within Google Scholar, which may be caused by Scholar’s greater inclusion of non-English data sources; and, the challenges in normalizing results between information sources (i.e. scientific journal versus conference proceedings, books, reports, etc.). These issues make the evaluation and comparison of journal performance difficult (Cabezas-Clavijo & Delgado-López-Cózar, 2012).

While these limitations are valid, they were less applicable to the analysis included in this evaluation, which was to determine whether or not knowledge produced by DSEN-supported researchers was having an impact on the broader community (as measured by the number of citations DSEN-supported papers received). As well, the evaluation was also interested in citations outside of academic journals. Therefore, data collected through Google Scholar was included in the evaluation of the program, given that “Google Scholar citations can provide an acceptable indicator of impact” of the DSEN-related papers on the broader community (van Aalst, 2010, p. 391).

Case Studies

Two purposefully selected case studies were conducted involving a total of six interviews as well as document review to support the background context of the case studies. One principal investigator was interviewed per case study, with an additional two query-related decision makers.

The case studies were written using a standard template (set of key questions) and informed by key informant interviews, review of query submissions, research reports as well as other information provided by the key informants. Due to time constraints, the case studies were not reviewed by the principal investigator as is standard procedure in developing case studies.

Administrative Data Analysis

The administrative/financial data of CIHR and HC was used to determine the cost of the program.

Crosswalk of TBS Core Evaluation Issues by Relevant Sections of DSEN Evaluation Report

TBS Core Issue Section & Page of Report
Performance (effectiveness, efficiency and economy)
Issue 4: Achievement of expected outcomes Pages 26–37
Issue 5: Demonstration of efficiency and economy Pages 13–25
Relevance
Issues 1-3: Continued need for program; alignment with government priorities; alignment with federal roles and responsibilities Pages 39–42

Appendix B: DSEN Logic Model

Long description: DSEN Logic Model

Appendix C: Life Cycle Approach Model

Long description: Life Cycle Approach Model

*Note that this diagram was developed by Health Canada.

Clinical Trial Review

Product Submission

Authorizations

Ongoing Reporting

Re-Evaluation

Appendix D: DSEN Organizational Chart (13–03–2013)

Long description: DSEN Organizational Chart (13–03–2013

Appendix E: DSEN Query Process Map

Long description: DSEN Query Process Map

References

Brookings Institution (2012). Highlights from the Fifth Annual Sentinel Initiative Public Workshop. Accessed 13–03–2013.

Canadian Institute for Health Information (2014). Prescribed Drug Spending in Canada, 2012: A Focus on Public Drug Programs [ PDF (1.4 MB) - external link ]. Accessed 23–04–2014.

Fernandopulle, R. B.; and Weerasuriya, K. (2003). What can consumer adverse drug reaction reporting add to existing health professional-based systems? Focus on the developing world. Drug Safety, 26, 4, 219-225.

Fox, G. and Otten, N (2007). Medicines that Work for Canadians: Business Plan for a Drug Safety and Effectiveness Network. Accessed on 18–12–2012.

Government of Canada (2012a). Proceedings of the Standing Senate Committee on Social Affairs, Science and Technology, Issue 26 - Evidence - November 21, 2012. Accessed on 18–12–2012.

Government of Canada (2013a). Twentieth Report: Prescription Pharmaceuticals in Canada: Post-Approval Monitoring of Safety and Effectiveness. Accessed on 19–07–2013.

Government of Canada (2013b). Twentieth Report: Prescription Pharmaceuticals in Canada: Post-Approval Monitoring of Safety and Effectiveness – Government Response. Accessed on 19–07–2013.

Government of Canada (2012a). Study on prescription pharmaceuticals in Canada - Issue No. 25, November 7 and November 8, 2012 - Transcript (Evidence) of Proceedings - November 8, 2012. Accessed on 19–07–2013.

Government of Canada (2012b). Study on prescription pharmaceuticals in Canada - Issue No. 26, November 21 and November 22, 2012 - Transcript (Evidence) of Proceedings - November 21, 2012. Accessed on 19–07–2013.

Government of Canada (2012c). Study on prescription pharmaceuticals in Canada - Issue No. 24, October 31 and November 1, 2012 - Transcript (Evidence) of Proceedings – October 31, 2012. Accessed on 19–07–2013.

Government of Canada (2012d). Jobs, Growth and Long-Term Prosperity: Economic Action Plan 2012, The Budget Speech [ PDF (298 KB) - external link ]. Accessed on March 28, 2012.

Government of Canada (2011). The Next Phase of Canada’s Economic Action Plan: Low Tax Plan for Jobs and Growth [ PDF (4.6 MB) - external link ]. Accessed on March 28, 2012.

Government of Canada (2007). Generic Drug Sector Study. Accessed on 11–07–13.

Government of Canada (2011). Bill C-13: The Canadian Institutes of Health Research Act. Accessed on 19–07–2013.

Hazell, L.; and, Shakir, S. A. (2006). “Under-reporting of adverse drug reactions: a systematic review”. Drug Safety, 29, 5, 385-396.

Health Council of Canada (2010). Keeping an Eye on Prescription Drugs, Keeping Canadians Safe: Active Monitoring Systems for Drug Safety and Effectiveness in Canada and Internationally [ PDF - external link ]. Accessed on 11–07–2013.

Health Canada (2006). Canada’s Food and Drugs Act & Regulations. Accessed on 07–12–2012.

House of Commons (2012). Drug Supply In Canada: A Multistakeholder Responsibility: Report of the Standing Committee on Health [ PDF (579 KB) - external link ]. Accessed August 2, 2013.

Industry Canada (2009). Mobilizing Science and Technology to Canada's Advantage: Progress Report 2009 [ PDF (528 KB) - external link ]. Accessed February 17, 2012.

Industry Canada (2007). Mobilizing Science and Technology to Canada’s Advantage [ PDF (468 KB) - external link ]. Accessed March 27, 2012.

Lexchin, J., Wiktorowicz, M., Moscou, K, & Eggertson, L. (2013). “Prescription Drugs and Administrative Costs: Provincial Drug Plan Officials’ Views of the Canadian Drug Safety System”. Journal of Health Politics, Policy and Law, 38, 3, pp. 545-571.

McDavid, J C. and Hawthorne, L.R.L. (2006). Program Evaluation and Performance Measurement: An Introduction to Practice. Thousand Oaks, CA: Sage Publications.

Morgan, S; Smolina, K; Mooney, D; Raymond, C; Bowen, M; Gorczynski, C; and, Rutherford, K (2013). The Canadian Rx Atlas, Third Edition. The Centre for Health Services and Policy Research: The University of British Columbia. Accessed on January 14, 2014.

One World Inc. (2005). Working Conference on Strengthening the Evaluation of Real World Drug Safety and Effectiveness: Summary Report. Accessed on 07–08–2012.

Research Canada (2012). Research Canada: An Alliance for Health Discovery Health Research Omnibus Survey Results.

Research Canada (2010). Canada Speaks! 2010: Canadians Go for Gold in Health and Medical Research. Accessed on 18–12–2012.

The Pulse (20 March 2013) - Accessed on 09–07–2013.

Treasury Board of Canada Secretariat (2012). Guidance on the Governance and Management of Evaluations of Horizontal Initiatives. Accessed on 07–08–2012.

Vaidya, S. S.; Guo, J. J.; Heaton, P. C.; and, Steinbuch, M. (2010). “Overview and Comparison of Postmarketing Drug Safety Surveillance in Selected Developing and Well-Developed Countries”. Drug Information Journal, 44, 519-533.

Wiktorowicz, M; Lexchin, J; and, Moscou, K. (2010). “Pharmacovigilance in Europe and North America: Divergent Approaches”. Social Science & Medicine, 75, 165-170.

Footnotes

Footnote 1

The Food and Consumer Safety Action Plan (FCSAP) is an integrated risk based approach which includes a series of initiatives that are premised on three key pillars: active prevention; targeted oversight; and rapid response leading ultimately to reduce adverse health incidents related to health and consumer products. DSEN is intended to contribute to the targeted oversight stream of outputs and outcomes resulting in improved information data and knowledge sharing of post-market drug safety and effectiveness.

1

Footnote 2

The RRS instrument was developed through a collaborative effort of representatives from all branches of CIHR and it went through a rigorous development which included an initial piloting of the instrument by researchers who received an OOGP grant that had an authority to use funds end date that fell between January, 2000 and October, 2008. There were 596 pilot RRS reports submitted.

2

Footnote 3

There were an additional 4 catalyst grants distributed in the 2009 competition, but the researchers have extended the duration of their grants, which has delayed the submission of their final reports.

3

Footnote 4

CIHR decided to not burden recipients of smaller grants (less than $100,000) with a full report, on the assumption that the limited funds awarded would reduce the likelihood of extensive results beyond that which the agency identified. The findings from a recently completed evaluation of CIHR’s knowledge translation program suggest that smaller grants do report many of the same achievements that larger, more long-term grants do, i.e. training, outcomes, impact, etc. The annual progress reports DSEN research teams use are also abbreviated reports. Therefore, this reduction of the RRS should be re-examined in order for CIHR to fully capture the impact these grants are having.

4

Footnote 5

In all, nine out of 15 SC members were interviewed. There were several reasons for the six declines received. Two of the positions were vacant, one declined due to conflict of interest, one declined for unspecified reasons and two never responded to repeated invitations

5

Footnote 6

Note that funding for the 2008 fiscal year was restricted to $1 million which was given to HC. The DSEN CO only began receiving funding in the 2009 fiscal year.

6

Footnote 7

DSEN has committed 47% of its grants and awards budget (up to FY 2017/18) to CNODES.

7

Footnote 8

It is important to note the CNODES research team receives funds to support research through their DSEN team grant and, as a result, does not need to apply for rapid funding grants.

8

Footnote 9

The DSEN Steering Committee Terms of Reference.

9

Footnote 10

The Guidance Document for Submitters of DSEN Queries.

10

Footnote 11

The RRS is CIHR’s end-of-grant reporting system that captures the outputs, outcomes and impacts of supported projects. The RRS was launched in 2011 and this is the first year DSEN-supported projects have submitted reports. The seven DSEN teams use a modified RRS report for their annual progress reports.

11

Footnote 12

CIHR has established a policy which requires grants receiving less than $100,000 only need to complete an abbreviated version of the RRS which has a sub-set of questions from the full RRS.

12

Footnote 13

While CIHR and HC acknowledge that a significant number of parties contribute to the success of the DSEN program, responsibility for the program falls under the FCSAP, with CIHR and HC being accountable for demonstrating that the DSEN program delivers the stated objectives and that the use of funds is appropriate

13

Footnote 14

Note that the DSEN CO has also delayed the implementation of its KT strategy due to a lack of sufficient resources, which has also contributed to the retained surplus.

14

Footnote 15

In order to focus strategically on developing the DSEN network for collaborating researchers, CIHR/DSEN focused efforts on the establishment of the key pan-Canadian network of research centres that delivers the core research capacity for the DSEN initiative. The time and effort required to reach the desired outcome suggested that, to be prudent, DSEN should manage funding between the implementation years to best target the DSEN Grants budget to research that aligns long term with the DSEN mandate and objectives. Thus $4.37M from 2010-11 and $1.2M in 2011-12 was re-distributed within the DSEN budget in equal amounts across 2012-16.

15

Footnote 16

This data was drawn from the methodological team’s annual financial reporting documents related to their relevant DSEN team grants.

16

Footnote 17

Note that each competition usually consists of one application to be reviewed. As well, the data on peer reviewer workloads is drawn from data collected from only three reviewers from the most recent competition and an additional two reviewers from a competition held last winter. Furthermore, salary data was from the 2012 year and was adjusted to account for inflation.

17

Footnote 18

The program has initiated an enhancement to their application operations in order to increase its efficiency. One Rapid Funding competition is held each fiscal year. This competition remains open to address Queries received for the entire year and peer reviewers are brought together to review applications on an ad hoc basis.

18

Footnote 19

The program has initiated a new competition cycle to reduce the costs associated with launching a funding opportunity every time a prioritized query is accepted by a non-CNODES team. The current model has a competition that is open for an entire fiscal year. Reviewers are called in to for ad hoc committees to review submitted applications.

19

Footnote 20

A further confounding factor is that some of the outcomes that can be expected could range from decisions to withdraw licences for specific drugs to adding an additional warning on the label of a drug. The calculation of the value of outcomes will be problematic which, in turn, will make a cost-benefit analysis difficult.

20

Footnote 21

CNODES has received a total of $7,785,000 as of March 31, 2013.

21

Footnote 22

Note that the 2012-13 fiscal year was the first year that DSEN provided a mix of direct research support and capacity development support. Therefore, the data included for this analysis was restricted to the 2012-13 year.

22

Footnote 23

2012 was the first year that DSEN began to provide direct support to trainees, in terms of scholarships and awards.

23

Footnote 24

It is important to note though, that DSEN’s Terms and Conditions (Ts&Cs) identify it as a grants program, the same as its parent organization, CIHR. This limitation is something that led one interviewee to suggest that it may be better to adopt a contract research approach, similar to America’s Mini-Sentinel program. This would require significant modification of DSEN’s Ts&Cs.

24

Footnote 25

The DSEN CO has developed the Framework for the Management of DSEN Queries, a document which provides an overview of the governance of the program and the query submission process and is available to the public. The document is intended to facilitate the submission of Queries from outside of HC.

25

Footnote 26

The use of Google Scholar for collection of bibliometric data has both merits and disadvantages, please see Appendix A under Evaluation Methodology, for a brief discussion of these.

26

Footnote 27

Please note that three Queries were responded to using CIHR catalyst grants prior to DSEN establishing research teams.

27

Footnote 28

See: High potency statins 'linked with increased risk of acute kidney injury'.

28

Footnote 29

The quote was drawn from the end-of-grant report submitted to CIHR.

29

Footnote 30

CAMESA helps parents and doctors manage the side effects of second generation antipsychotics in children

30

Footnote 31

Antithrombotic Therapy for Patients with Atrial Fibrillation.

31

Footnote 32

Information is based on data obtained from the DSECT website.

32

Footnote 33

Results from the evaluation of CIHR’s Knowledge Translation programs suggest that these smaller grants (both Synthesis and Knowledge to Action grants are capped at $100,000) have resulted in capacity development. Recipients of these smaller grants reported, via an online survey, that they trained approximately five individuals, on average (CIHR, 2013). They also reported that they had numerous knowledge translation outputs and outcomes. This type of information, i.e. capacity development, and outputs and outcomes of research, will be required in the next evaluation of the DSEN program. Therefore, it may be beneficial for CIHR to reconsider its policy to require recipients of smaller grants to only complete a reduced RRS report, or, to redefine what dollar value is “small”.

33

Footnote 34

Health Canada 2014-15 Report on Plans and Priorities.

34

Date modified: