What CIHR Heard: Analysis of Feedback on the Design Discussion Document

August 2012

Table of Contents

• Message from the Reforms Task Force
• Executive Summary
  1. Purpose and Introduction
  2. Approach to Collecting and Synthesizing the Feedback
  3. Summary of the Research Community's Feedback
     • 3.1 Feedback on the Proposed Architecture
       • 3.1.1 Foundation/Programmatic Research Scheme
       • 3.1.2 Project Scheme
       • 3.1.3 Integrated Knowledge Translation
       • 3.1.4 New Open Suite of Programs: General Feedback
     • 3.2 Feedback on the Proposed Mechanisms
       • 3.2.1 Multi-phased Competition Process
       • 3.2.2 Application-Focused Review
       • 3.2.3 Structured Review Criteria
       • 3.2.4 Remote (Virtual) Screening Process
       • 3.2.5 College of Reviewers
     • 3.3 Transition
  4. Next Steps
• Annex I
• Annex II

Message from the Reforms Task Force

The Reforms Task Force would like to thank the community for taking the time to voice its opinions through town hall discussions, meetings, emails, letters and the on-line survey. The feedback received reflects the commitment of the Canadian health research community to ensure that we have a system that supports excellence across all research domains.

We now have the challenge of responding to the feedback, which as you will read in this document, reflects the varying perspectives of health research disciplines in Canada.

The architecture that CIHR is proposing for the Open Suite of Programs consists of two broad schemes. The Foundation/Programmatic Research Scheme is about supporting people. It is about providing long-term support to investigators with a demonstrated track record of success. The Project Scheme is about supporting ideas. It is meant to encourage researchers who want to explore specific ideas across the spectrum of health and health systems research and knowledge translation to submit proposals for projects with a specific scope and defined timelines.

Based on the feedback received, we are confident that introducing these two broad funding schemes will help to achieve the overall objectives of the Reforms. CIHR's Governing Council has recently reaffirmed their commitment to this direction. On June 26, 2012 it passed the following motion:

"Governing Council reaffirms the commitment to implementing the Project and Programmatic Research Schemes as part of the Reform to the Open Suite of Programs, that work continues to achieve this, and that this be communicated to the broader research community in July 2012."

To support the adjudication of these two funding schemes, the Design Discussion Document proposed a number of "mechanics" including a multi-phased competition processes, application-focused review, and remote screening process. Opinions regarding the proposed peer review changes are divided and at times polarized. While there are some aspects that the community supports, there are others that are causing great concern. CIHR acknowledges that more work needs to be done to ensure that the peer review process addresses the current challenges in our system and that any changes implemented will not reduce the quality of peer review.

In response to Governing Council's motion, we will continue to work to flesh out additional details around the Foundation/Programmatic Research and Project Schemes. In response to your concerns, we will conduct further analysis over the coming months and work through alternative scenarios to determine the best approach for peer review. We will continue to work with a variety of stakeholders including institutions, the University Delegates, the Institute Advisory Boards, and other advisors to further develop and refine the design details. We expect to release a response to your feedback and announce the design in the fall. This will include a description of a series of pilots to test aspects of the changes and make design changes, where warranted.

Regardless of the final design, we recognize that CIHR must carefully plan the transition to the new schemes. The details on how researchers (both single and multiple grant holders) will transition from the current Open Suite of Programs to the new funding schemes are currently being developed and will also be announced in the fall.

Again, we would like to thank you for all of your feedback and assure you that we are committed to continued communication with the research community.

CIHR Reforms Task Force

Executive Summary

It has been five months since the release of the design document describing the proposed reform to CIHR's Open Suite of Programs and peer review process. The quality and quantity of feedback has been excellent, and CIHR would like to thank Canada's health research community for taking the time to respond.

Based on the feedback received, the research community has signalled to CIHR that it is generally supportive of the need for improvements. There has been significant support for introducing two broad funding schemes – one that focuses on broader programs of research, and one that supports specific projects. The introduction of longer-term funding through the Foundation/Programmatic Research Scheme has been generally well received and was seen as a positive step towards reducing applicant and peer reviewer burden. Respondents were also in favour of introducing application-focused review, the use of structured review criteria, and implementing a College of Reviewers to improve the consistency, reliability, and fairness of review. Opinions were divided on whether the proposed multi-phased competition process would reduce the burden on applicants and peer reviewers, however there is, in general, agreement that the proposed reforms are a positive step forward.

It is in the specific details of the funding schemes, and the specific details of how peer review will be conducted, where opinions were divided (and at times highly polarized) across research fields, career stages, and other stakeholder groups. The feedback received reflects the varied perspectives of health research fields in Canada, emphasizing that the new programs and processes must strive to support excellence across the full spectrum of health research. The concerns and suggestions made by different research communities are described in detail in Sections 3.1 to 3.3 of this document.

Many within the research community indicated that the proposed timelines are ambitious, and recommended that CIHR consider a longer transition period. Similarly, recommendations were made that CIHR rigorously monitor and assess the proposed changes to ensure the quality and efficiency of the new processes. Concerns were also raised about the accessibility of the new funding schemes for early- and mid-career investigators. CIHR agrees with these statements, and is committed to developing a comprehensive implementation plan that would introduce the changes gradually over a longer period of time. This plan would identify and describe potential pilot studies, as well as opportunities to rigorously evaluate the proposed changes to the peer review system before they are fully implemented. The roll-out of the new funding schemes would also be monitored as part of the implementation process, with course corrections applied if needed. Finally, mechanisms would be set in place to give early- and mid-career investigators fair access to both funding schemes.

Many raised concerns about the amount of federal funding available to support health research in Canada. Some felt that any changes to the system would have limited impact in the absence of additional funding. While CIHR appreciates these concerns, it is important to keep the current economic context in mind. CIHR is fortunate that its grants budget has remained virtually intact, in spite of global and national economic pressures that have led to spending reductions throughout the federal government.

1. Purpose and Introduction

The purpose of this document is to provide an account of the feedback received from CIHR's research community regarding the proposed changes to the Open Suite of Programs and peer review process.

This document intends to:

• Share what we heard about CIHR's proposed design for the Open Suite of Programs and peer review enhancements; and
• Clearly layout next steps.

CIHR would like to thank the research community for taking the time to read the Design Discussion Document: Proposed Changes to CIHR's Open Suite of Programs and Peer Review Process, reflect on the design, and provide thoughtful comments and suggestions.

The community's response to the proposal is critical to the finalization of the design, and to informing what kinds of pilots, transition, and implementation plans are needed. Feedback was informally provided to CIHR at various Town Hall meetings, as well as formally submitted by letter, e-mail and feedback form from February 8 to May 1, 2012. Each opinion was taken into account and considered and compared with groups of related opinions, as well as analyzed by respondent research field, career stage, and researcher vs. non-researcher status.

CIHR is committed to keeping the research community informed and involved as it moves forward with reviewing, refining, and implementing the new Open Suite of Programs and peer review processes. We hope this document will be informative to those who provided feedback, and to the broader community.

2. Approach to Collecting and Synthesizing the Feedback

Approach to Collecting Feedback

Feedback from the research community on the proposed changes to the Open Suite of Programs and peer review process was collected from February 8, 2012 to May 1, 2012 using a number of feedback mechanisms (Figure 1):

Figure 1: Feedback Collected from CIHR's Research Community

Figure 1. Mechanisms used to engage the research community and collect feedback on the proposed changes to the new Open Suite of Programs and peer review process.

a. Face-to-Face Discussions

CIHR held 82 discussions with institutions, associations and partners. These discussions included "Town Halls" at various research institutions across the country. Tailored presentations were given to institution administrators and funding partners, and to open forums of researchers, by either CIHR's President or Chief Scientific Officer and Vice-President Research and Knowledge Translation, and were followed by a question and answer period. The feedback received from these Q&A sessions was documented by a CIHR administrator, and submitted by e-mail to a centralized inbox. The list of stakeholders and partners engaged in these discussions can be found online.

b. Web-based Discussion Forum

A moderated Web-based forum was established to capture feedback on the proposed changes to the new Open Suite of Programs and peer review system. The forum focused on major discussion threads, which included the Foundation/Programmatic Research Scheme, Project Scheme, and general peer review enhancements. Researchers were also able to add new discussion topics. There were 186 subscribers to this forum and 22 comments posted.

c. Anonymous Feedback Form

In order to gather structured feedback, a Web-based feedback form (survey format) was developed, and e-mails were sent to a random sample of over 1,690 researchers from all health research fields, inviting them to participate. The availability of the feedback form was also advertised at Town Halls. The form comprised 17 questions, 13 of which required participants to select from a set of structured responses, and 4 of which allowed participants to submit narrative responses (comments and/or questions).

513 completed questionnaires were submitted, representing a response rate of 30% of the randomly selected researchers.

The majority of feedback form respondents were self-identified senior investigators (47%), with a nearly equivalent combined number of early career investigators (24%) and mid-career investigators (24%) (Figure 2).

Figure 2: Proportion of Feedback Form Respondents by Self-Identified Career Stage

Figure 2. Proportion of respondents who completed the feedback form, by self-identified career stage. Totals may not agree due to rounding (N= 513). "Early career researcher" is defined as less than 5 years as an independent researcher (including self-identified graduate students and postdoctoral fellows), "Mid-career researcher" is defined as 5-10 years as an independent researcher, "Senior Researcher" is defined as more than 10 years as an independent researcher. "Other" includes respondents who identified themselves as "Knowledge User" and "Other" (but not including those who self-identified as graduate students and postdoctoral fellows). Note that Knowledge Users represent approximately 46% of the "Other" sample.

Additionally, the distribution of respondents by self-identified pillar of research was generally representative of the random sample selected and of the proportion of funded researchers in the current Open Suite of Programs, by pillar of research. Overall, the majority of respondents were Pillar 1 (Biomedical) researchers (Figure 3).

Figure 3: Proportion of Feedback Form Respondents by Self-Identified Pillar of Research

Figure 3. Proportion of respondents who completed the feedback form, by self-identified pillar of research. Pillar 1 includes biomedical research, Pillar 2 includes clinical research, Pillar 3 includes health system and services research; and, Pillar 4 includes social, cultural, environmental and population health research. Totals may not agree due to rounding (N= 513).

The web-based feedback form was also made available to the public on the CIHR website between February 13 and May 1, 2012.

Further detail on the questions asked can be found in Annex I.

d. One-Way Correspondence

A central e-mail address was made available for stakeholders to submit all questions and comments related to the proposed changes. Feedback was also submitted directly to the President's and Vice-President Research's e-mail addresses, and forwarded by CIHR administrators to the designated central e-mail address for further consideration and analysis. Letters were submitted by individuals, university departments and faculties, research hospital teams, and professional associations.

As of May 1, 2012, 206 letters and e-mails were received, some with multiple signatories (n = 45), for a total number of 1,270 correspondents [Annex II]. It was noted that the vast majority of letters were submitted by senior researchers within the biomedical community.

Approach to Synthesizing the Feedback

An attempt has been made to summarize in a qualitative manner both the structured and unstructured/free form feedback received by CIHR. To assist with the synthesis of the unstructured comments, an organizing framework was developed. All individual comments and questions received, including those within and outside the scope of the proposed changes to the New Open Suite of Programs and Peer Review Enhancements were carefully read, coded, and catalogued according to 14 pre-defined areas (Figure 4):

Figure 4. List of common feedback themes identified by design topic area.

Comments and questions were further assigned sub-topics to capture common and recurring themes. Once comments and questions were assigned a topic and sub-topic, each was further coded as either:

• Positive: Comment found to be supportive of the principles behind the design element;
• Negative: Comment not found to be supportive of the principles behind the design element; or
• Neutral: Question/Design Consideration/Suggestion.

Narrative Analysis Qualitative Reporting Scale

For each design element, commonly expressed opinions were aggregated and reported as either design strengths or design concerns. Common and unique design considerations and suggestions were also reported for each design element. The descriptors for the percentage of respondents in agreement with a specific design element are as follows:

Defining an "Equivalent Number of Individual Comments"

All narrative feedback received was assigned a weight based on the number of signatories/co-signatories. For example, if a letter was signed by 10 individuals, then the comments in that letter were each assigned a weight of 10.

3. Summary of the Research Community's Feedback

3.1 Feedback on the Proposed Architecture

This section focuses on the feedback received about the Foundation/Programmatic Research Scheme (3.1.1); the Project Scheme (3.1.2); Integrated Knowledge Translation (3.1.3); and, General Feedback on the New Open Suite of Programs (3.1.4).

Overall, several respondents supported the creation of the Foundation/Programmatic Research and Project Schemes, with particular support for the distinction between the evaluation of people and ideas, respectively. However, some respondents expressed uncertainty as to which stream would best meet their needs and noted that distinguishing between the streams adds unnecessary complexity to an already complex system.

Regarding funding ratios for the two Schemes, a few respondents disagreed with the proposed 45/55 budget split between respectively the Foundation/Programmatic Research and Project Schemes. Of these respondents, some requested that CIHR provide more information about how this "optimal" ratio was calculated.

3.1.1 Foundation/Programmatic Research Scheme

A) Structured Responses from the Feedback Form

According to the summary of responses from the Design Discussion Document Feedback Form, approximately 68% of respondents would be interested in applying to the Foundation/Programmatic Research Scheme (Figure 5).

Figure 5 and 6: Proportion of Feedback Form Respondents who characterized themselves as individuals who would apply to the proposed funding mechanisms

Figure 5. Proportion of respondents who characterized themselves as individuals who would apply to the proposed funding mechanisms. Data as of May 1, 2012 (N = 513). Totals may not agree due to rounding.

The greatest level of interest in applying to the Foundation/Programmatic Research Scheme came from:

• 73% of early career researchers from Pillar 3 (Health System and Services);
• 73% of senior researchers from Pillar 1 (Biomedical);
• 57% of early career and 56% senior researchers from Pillar 4 (Social, Cultural, Environmental and Population).

B) Analysis of Narrative Responses

Approximately 61% of respondents (n = 1,028) from all feedback venues submitted comments and/or questions about the proposed Foundation/Programmatic Research Scheme. In general, narrative responses were found to be supportive of the overall principles behind this design element, with feedback focusing on suggestions and considerations that could improve the specifics of the Foundation/Programmatic Research Scheme. Details of the discussions regarding the Foundation/Programmatic Research Scheme focused on the following themes: Grant value; Eligibility; Collaboration; Institutional commitment; the New/Early Career Investigator Stream; and, Transition.

Grant Value

Although the Design Discussion Document noted that the estimated average grant values were based on financial modelling of current CIHR funded grants, and that actual grant values would be awarded "commensurate with scientific need", some respondents interpreted this average value of grants as a cap on the amount of funding to be awarded per grant per year. These respondents commented that such a cap would be too limiting, and would not provide adequate funding.

Several respondents were concerned with the funding level for the Foundation/Programmatic Research grants, with some respondents stating they would need more than the average value of $300,000 per year and a few stating that smaller grant values would meet the needs of their program of research. Several respondents stated that CIHR should ensure that the Foundation/Programmatic Research grants cover a wide range of programs with grant values that are realistic and commensurate with the research proposed.

In addition, several respondents commented that it would be advantageous to have some flexibility in funding, with the opportunity to request additional funds if required due to new lines of inquiry or activity/program growth during a 7-year period. This was noted to be especially relevant to new/early career and mid-career investigators who could expect to see significant ramp-up over the course of their grant. Many respondents commented that having a Foundation/Programmatic Research Grant would be like having "all of their eggs in one basket," and expressed concern about the outcome should their grant not be renewed. A few suggested that a bridge grant would be critical to the success of these researchers or that renewal should occur in Year 6 of the grant, so that unsuccessful applicants still had a year left on their Foundation/Programmatic Research grant to explore other options.

Eligibility

Some respondents expressed uncertainty as to whether they would meet the eligibility criteria for the Foundation/Programmatic Research Scheme, which considers "independent researchers (new or established) with a demonstrable track record of excellence and/or impact in their field of study." Comments centered on accessibility of the grants for part-time researchers, clinician scientists, holders of single vs. multiple grants, mid-career scientists, and equally across all CIHR pillars of health research, including the social sciences and humanities. Some respondents cautioned that the Foundation/Programmatic Research grants would be more difficult to obtain by smaller labs and smaller institutions that may have more restricted access to supports, such as research infrastructure and trainees. A few also raised concern regarding support for less populous regions, especially in light of the Regional Partnerships Program.

In relation to assessing the caliber of an applicant, a few commented that researchers from different fields of health research have different metrics of productivity and success. These respondents requested more information on how appropriate metrics would be incorporated into the evaluation criteria and process.

Several respondents cautioned that the changes would create a two-tier system causing the rich to get richer. The transition from the Project Scheme to the Foundation/Programmatic Research Scheme was a concern for some respondents who commented that they would be disadvantaged in being compared to more senior scientists.

Collaboration

In the Design Discussion Document, CIHR recognized that research is increasingly being conducted by groups of researchers, and acknowledged concerns over the eligibility of applications with multiple leads. Of respondents who commented on collaboration, many supported the notion that teams of researchers should be eligible to apply for Foundation/Programmatic Research grants. Several respondents pointed out that although teams are an important mechanism, complications can arise in relation to tenure and promotion requirements for individual team members.

Considerations for researchers to collaborate outside the scope of the immediate Foundation/Programmatic Research grant team were also discussed. The Design Discussion Document states that an investigator holding a Foundation/Programmatic Research grant would be ineligible to apply for a Project grant. Several suggested that Foundation/Programmatic Research grant holders should be eligible to be co-applicants on Project grants, though a few were opposed to this approach.

A few respondents requested more information on how the proposed changes would facilitate international partnerships, and suggested CIHR align its policies with other large international funding organizations (e.g., salaries being paid from grants, as is done in the United States).

Evaluation of Foundation/Programmatic Research grants

The comments regarding the review process for the Foundation/Programmatic Research grants centered mainly on the Stage 1 evaluation. As described in the Design Discussion Document, Stage 1 would assess the caliber of an applicant based on a short summary of the program of research and on the applicant's curriculum vitae. Some respondents further elaborated that an applicant's productivity should be assessed using citation analysis and linkage maps, and that track record and research interests should be assessed using the applicant's curriculum vitae.

Institutional Commitment

Several respondents expressed concern that institutional support or commitment would be a factor in the assessment of their applications. Consequently, they cautioned that this may disadvantage smaller institutions, with some expressing concern that applicants from institutions that provide less support will be negatively impacted when applying for grants. There is some concern that institutions may not be able to provide more support than they are currently providing or that the required institutional sign-off on applications may result in favoritism between applicants. Similarly, a few respondents commented that there may be difficulties in obtaining the long-term institutional support required for the Foundation/Programmatic Research Scheme. In addition, some respondents pointed out that the requirements for protected research release time must be designed so as not to disadvantage researchers with teaching responsibilities.

From a number of meetings held with Institution administrators, CIHR learned that the requirement for additional institution support in the Foundation/Programmatic Scheme was contributing to the misconception that the proposed schemes would support a two-class system of research. To avoid this, institution administrators recommended CIHR enforce the same requirement for institutional support in both funding schemes.

New/Early Career Investigator Stream

Some respondents conveyed that the inclusion of a separate stream for new/early career investigators within the Foundation/Programmatic Research Scheme was a strength, while several commented that the proposed design element would not address challenges faced by new/early career investigators today. Many respondents expressed that CIHR's proposed definition of a new/early career researcher to be too limiting, and would like to see it revised to be more reflective of the actual early career trajectory. Many respondents requested further details as to how new/early career investigators would be evaluated in the Foundation/Programmatic Research Scheme.

Transition

A few respondents expressed concern about the transition of their current funding into a Foundation/Programmatic Research grant, stating that they were uncertain whether they would be disadvantaged if some or all of their funding for their current program came from other agencies outside of CIHR. Some said it would be difficult to decide which scheme would be best suited to their needs.

3.1.2 Project Scheme

A) Structured Responses from the Feedback Form

According to the summary of responses from the Design Discussion Document Feedback Form, approximately 71% of respondents would be interested in applying to the Project Scheme (Figure 6).

Figure 5 and 6: Proportion of Feedback Form Respondents who characterized themselves as individuals who would apply to the proposed funding mechanisms

Figure 6. Proportion of respondents who characterized themselves as individuals who would apply to the proposed funding mechanisms. Data as of May 1, 2012 (N = 513). Totals may not agree due to rounding.

The greatest level of interest in the Project Scheme came from:

• 84% of early career Pillar 1 (Biomedical) and 83% of early career Pillar 2 (Clinical) researchers; and
• 94% of mid-career and 76% of senior Pillar 3 (Health System and Services) researchers.
• 77% of senior Pillar 4 (Social, Cultural, Environmental and Population) researchers.

B) Analysis of Narrative Responses

Approximately 40% of respondents (n = 674) from all feedback venues submitted comments and/or questions about the proposed Project Scheme. Overall, narrative responses were found to be generally supportive, but focused on suggestions and considerations that could improve the specifics of the Project Scheme. Details of the comments regarding the Project Scheme focused on the following themes: Grant Value and Duration; Evaluation of Project Grants; and, Encouraging Innovation.

Grant Value and Duration

Similar to the Foundation/Programmatic Research grants, a few respondents expressed concern that the average value suggested for the new Project Grants (stated as $125,000 per year in the Design Discussion Document) would be lower than the current average in the CIHR Open Operating Grant competition (stated as $123,000 per year in 2010-11 in the Design Discussion Document). Some respondents cautioned that support for large grants and Randomized Controlled Trials would not be feasible in the Project Scheme. Of these, a few reiterated that grant values should be flexible depending on the length of the project.

Many respondents agreed with CIHR that required grant values should vary with the type of research, and that flexibility is required in order to meet the needs of all applicants. However, a few respondents expressed concerns about reviewer bias against grants that deviate from the proposed average grant value (e.g., grants that are much larger ormuch smaller than $125,000 per year).

Several respondents commented on the average duration of the Project grants, specifically stating that the majority of project grants should be closer to five years when considering the three-to-five-year range to address applicant and peer review burden and to maximize the impact of the project. Some respondents added that the five-year duration would be better suited to supporting trainees within the research project.

Evaluation of Project Grants

The comments regarding the review process for the Project grants centered mainly on the Stage 1 evaluation. As described in the Design Discussion Document, Stage 1 would assess the quality of an idea based on a short 2-3 page project proposal, independent of the track record of the applicant/team. Some respondents noted that this short application would not contain sufficient information for meaningful review, and while some stated their support for this concept, many of the respondents who commented on the Stage 1 review expressed that it would not be feasible or practical to evaluate anonymous Stage 1 applications. A few respondents suggested omitting the Stage 1 application entirely from the Project Scheme.

Encouraging Innovation

One of the objectives of the Project Scheme, as described in the Design Discussion Document, is to encourage original or innovative advances in health knowledge or knowledge translation. Of the respondents who commented on innovation, several supported CIHR's efforts to promote the funding of high-risk grants, and reiterated that the current peer-review system tends to favour conservative or safe research. On the other hand, several respondents questioned whether the proposed changes would, in fact, foster innovative grants. Some respondents requested that CIHR clarify what is meant by innovation, as the application of the term could vary across the different disciplines of health research.

A few suggested a separate stream to support innovation, where high-risk grants could be better recognized and supported without as much pressure to produce. A few respondents also suggested including an innovation score as part of the evaluation criteria.

The need for CIHR to be more responsive to emerging opportunities was identified by a few respondents, who suggested that quicker funding competition turn-around times would better support this fast-paced environment.

3.1.3 Integrated Knowledge Translation

A) Structured Responses from the Feedback Form

According to the summary of responses from the Design Discussion Document Feedback Form, approximately 34% of respondents agreed or strongly agreed when asked whether integrated knowledge translation would help address CIHR's current challenges with its Open Suite of Programs and peer review system (Figure 7).

Figure 7: Feedback Form Respondent Degree of Agreement: Integrated Knowledge Translation addresses challenges with CIHR's Open Suite of Programs and peer review system

Figure 7. Feedback Form responses showing the extent to which respondents agreed or disagreed that integrated knowledge translation would help address CIHR's current challenges with its Open Suite of Programs and peer review system. Data as of May 1, 2012 (N = 511). Totals may not agree due to rounding.

Further breakdown of structured responses (by self-identified pillar and career stage) indicated that general agreement with integrating knowledge translation to address current challenges with the Open Suite of Programs and peer review system was highest among other respondents, which include Knowledge Users. Researchers from Pillar 3 (Health Systems and Services) and Pillar 4 (Social, Cultural, Environmental and Population) also showed high levels of agreement. General disagreement with integrating knowledge translation to address current challenges with the Open Suite of Programs and peer review system was highest with mid-career and senior researchers, as well as with Pillar 1 (Biomedical) researchers (Figure 8).

Figure 8: Feedback Form Respondent Degree of Agreement: Integrated Knowledge Translation addresses challenges with CIHR's Open Suite of Programs and peer review system (by self-identified pillar and career stage)

Figure 8. Feedback Form responses showing the percentage of respondents who believe that integrated knowledge translation would help address CIHR's current challenges with its Open Suite of Programs and peer review system, by self-identified pillar and career stage. Responses classified as "in agreement" include those who responded as "agree" and "strongly agree", respondents classified as "in disagreement" include those who responded as "disagree" and "strongly disagree", and respondents classified as "neutral" include those who responded as "neither agree nor disagree", "don't know" and "(blank)". "Early career researcher" is defined as less than 5 years as an independent researcher (including self-identified graduate students and postdoctoral fellows), "Mid-career researcher" is defined as 5-10 years as an independent researcher, "Senior Researcher" is defined as more than 10 years as an independent researcher. "Other" includes respondents who identified themselves as "Knowledge User" and "Other". P1 refers to Pillar 1 (Biomedical), P2 refers to Pillar 2 (Clinical), P3 refers to Pillar 3 (Health System and Services), and P4 refers to Pillar 4 (Social, Cultural, Environmental and Population). Note that Knowledge Users represent approximately 46% of the "Other" sample. Data as of May 1, 2012 (N = 519, which includes duplicate responses included from Senior Researcher/Knowledge User and Mid-Career Researcher/Knowledge User combinations).

The strongest levels of agreement came from:

• 71% of early career and 72% of senior Pillar 3 (Health Systems and Services) researchers;
• 70% of early career Pillar 4 (Social, Cultural, Environmental and Population) researchers and 79% of "other" respondents (including Knowledge Users).

The strongest levels of disagreement came from:

• 49% of mid-career and 46% of senior Pillar 1 (Biomedical) researchers.

B) Analysis of Narrative Responses

Approximately 3% of respondents (n = 45) from all feedback venues submitted comments and/or questions about integrated knowledge translation. In analysing both the structured and unstructured responses on integrated knowledge translation, it would appear that principles behind this design element may have been misunderstood. For example, several respondents assumed that the requirement for integrated knowledge translation would be applied to all applicants, and expressed that this requirement would contribute to an erosion of investigator-driven research. Nevertheless, narrative responses were generally found to be supportive. Details of the comments regarding integrated knowledge translation focused on the following themes: Recognition of Integrated Knowledge Translation; Consolidation of Knowledge Translation Programs into the New Open Schemes; Commercialization and Technology Transfer; and, Support for Researchers.

Recognition of Integrated Knowledge Translation

Many respondents were supportive of CIHR's efforts to encourage the involvement of knowledge users in the research process, as appropriate, recognizing that inclusion requirements would vary across research types. Many respondents recognized the advantages of including knowledge users such as patients, citizens, consumers and clinicians in research projects and programs. Some respondents expressed concern about the application of this requirement, specifically in the area of biomedical research. These respondents commented that CIHR must ensure that curiosity-driven research is preserved. A few respondents commented that the focus on knowledge translation, and the inclusion of relevant knowledge users in research projects, was too great and would not yield the results that are intended.

Consolidation of Knowledge Translation (KT) Programs into the New Open Schemes

In order to simplify the application and funding processes, CIHR proposed to consolidate some of the objectives of smaller funding programs, including the knowledge translation and commercialization programs, into the new funding schemes. An equal number of respondents supported and opposed this approach. Those in support of the approach commented that consolidation of the budgets would lead to better support of investigator-initiated research, while at the same time increasing flexibility of funding for those groups who were traditionally supported through these smaller mechanisms. Those in opposition of this approach stated that these separate programs are required in order to build capacity in these areas and move the results of research into practice.

Commercialization and Technology Transfer

Some respondents commented on the inclusion of commercialization and technology transfer within the broader integrated knowledge translation concept. Of these, many recognized that industry as a knowledge user would apply most often to biomedical research and that collaboration with industry should be encouraged. Most respondents who commented on this suggested that more support for commercialization and technology transfer activities is needed, and a few respondents stated that commercialization and innovation should be the main focus of the reforms in order to better position Canada within the OECD countries.

Support for Researchers

A few respondents suggested that supports for integrated knowledge translation, such as a system to facilitate matching of researchers to knowledge users/partners, would be essential in ensuring the success of this approach. In addition, some respondents expressed that more clarity is needed in describing what level of knowledge user involvement is required. Some also commented that evaluation criteria must address the variability in knowledge user involvement across all fields and pillars of research, to ensure that the science and knowledge translation components are considered appropriately.

3.1.4 New Open Suite of Programs: General Feedback

A) Structured Responses from the Feedback Form

Reducing Barriers to Funding Excellence

According to the summary of responses from the Design Discussion Document Feedback Form, approximately 23% of respondents agreed or strongly agreed when asked whether the proposed changes would reduce barriers to funding excellence across the full spectrum of research (Figure 9).

Figure 9: Feedback Form Respondent Degree of Agreement:The proposed changes would reduce barriers to funding excellence across the full spectrum of health research

Figure 9. Feedback Form responses showing the extent to which respondents agreed or disagreed that the proposed changes would reduce barriers to funding excellence across the full spectrum of health research. Data as of May 1, 2012 (N = 513). Totals may not agree due to rounding.

Further breakdown of structured responses by self-identified pillar and career stage indicated that general agreement with this statement was highest among:

• 53% of Pillar 3 (Health System and Services) 48% of Pillar 4 (Social, Cultural, Environmental and Population) early career researchers; and
• 46% of mid-career Pillar 2 (Clinical) researchers.

B) Analysis of Narrative Responses

Approximately 36% of respondents (n = 606) from all feedback venues submitted comments and/or questions about the feasibility of the overall design, and its ability to reduce barriers to funding excellence in health research. Overall, while this section was designed to capture frequently cited concerns about the overall design of the new Open Suite of Programs, most concerns were accompanied by constructive suggestions. Details of the comments regarding the overall design focused on the following themes: Supporting the Full Spectrum of Health Research; General Eligibility to the Foundation/Programmatic Research and Project Schemes; and, CIHR's Grants and Awards Budget.

Supporting the Full Spectrum of Health Research

One of the key features of the proposed new Open Suite of Programs is the intent to increase access to funding for investigators in all areas of health research. Some respondents emphasized to CIHR the importance of equal access for all health researchers, and requested clarification on how this would be monitored and achieved in the new schemes.

Respondents from Pillars 1 (Biomedical) and 2 (Clinical) noted that the new Schemes seemed more focused on applied research, and expressed concern that curiosity-driven research would be underfunded. On the other hand, respondents from Pillars 3 (Health system and Services) and 4 (Social, Cultural, Environment and Population) indicated that the new system would increase opportunities for funding, but needed more information as to how their applications would be assessed, given the loss of the standing committee structure.

General Eligibility to the Foundation/Programmatic Research and Project Schemes

Access to funding for research from all areas of health was an important topic for several respondents. Specific comments centred on the need for improved access to funding for social sciences and humanities researchers and for clinician scientists. Respondents liked that the proposed changes would provide equal opportunity across pillars and across all areas of research (not just current "hot topics"). In addition, a few respondents suggested targeted funding to support health professionals and colleges.

Some respondents expressed concern that the proposed system would not adequately support mid-career investigators. These respondents indicated that they see the best fit for their research within the Foundation/Programmatic Research Scheme, but are uncertain if they will be successful in competition with more senior researchers.

CIHR's Grants and Awards Budget

Several respondents commented that federal support for health research and development allocated through CIHR is insufficient to maintain adequate support for excellent research in Canada. Of these, many respondents stated that any changes implemented to the system would have limited impact in the absence of additional funding. Low success rates were a concern raised by several of the respondents, who commented that the expansion of the Medical Research Council's (MRC) mandate to create CIHR was not accompanied by a proportionate increase in budget, and was insufficient to respond to the significant increase in application pressure.

Some respondents remarked that reviewer fatigue was not a result of the current peer review system but, rather, a direct result of low success rates. These respondents generally commented that burden and fatigue were most often a result of reviewing excellent submissions and re-submissions, knowing that a significant proportion of meritorious grants would not be funded.

Several respondents suggested that CIHR re-allocate a portion of the budget currently reserved for strategic funding into the Open Operating Grant competition. A few proposed re-allocating some of the budget used for training grants into the Open Operating Grant.

A few respondents suggested consolidating funding from other agencies into the CIHR Open Operating Grant competition. Specific suggestions included re-directing a portion of the additional funding received by the Canadian Foundation for Innovation (CFI), or a portion of the $7 billion annual budget allocated to encourage R&D as part of Canada's Economic Action Plan.

3.2 Feedback on the Proposed Mechanisms

This section focuses on the feedback received about the multi-phased competition process (3.2.1); application-focused review (3.2.2); structured review criteria (3.2.3); the remote (virtual) screening process (3.2.4); and, the College of Reviewers (3.2.5).

3.2.1 Multi-phased Competition Process

A) Structured Responses from the Feedback Form

According to the summary of responses from the Design Discussion Document Feedback Form, approximately 50% of respondents agreed or strongly agreed when asked whether the multi-phased competition process would help address CIHR's current challenges with its Open Suite of Programs and peer review system (Figure 10).

Figure 10: Feedback Form Respondent Degree of Agreement: The Multi-Phased Competition Process addresses challenges with CIHR's Open Suite of Programs and peer review system

Figure 10. Feedback Form responses showing the extent to which respondents agreed or disagreed that the multi-phased competition process would help address CIHR's current challenges with its Open Suite of Programs and peer review system. Data as of May 1, 2012 (N = 511). Totals may not agree due to rounding.

Further breakdown of structured responses (by self-identified pillar and career stage) indicated that general agreement with the ability of the multi-phased competition process to address current challenges with the Open Suite of Programs and peer review system was highest among early career researchers and "other" respondents, which include Knowledge Users. Researchers from Pillars 2 (Clinical) and 3 (Health Systems and Services) and 4 (Social, Cultural, Environmental and Population) also showed high levels of agreement. On the other hand, general disagreement with the multi-phased competition process' ability to address current challenges with the Open Suite of Programs and peer review system was highest with mid-career and senior researchers, as well as with Pillar 1 (Biomedical) researchers (Figure 11).

Figure 11: Feedback Form Respondent Degree of Agreement: The Multi-phased Competition Process addresses challenges with CIHR's Open Suite of Programs and peer review system (by self-identified pillar and career stage)

Figure 11. Feedback Form responses showing the percentage of respondents who believe that the multi-phased competition process would help address CIHR's current challenges with its Open Suite of Programs and peer review system, by self-identified pillar and career stage. Responses classified as "in agreement" include those who responded as "agree" and "strongly agree", respondents classified as "in disagreement" include those who responded as "disagree" and "strongly disagree", and respondents classified as "neutral" include those who responded as "neither agree nor disagree", "don't know" and "(blank)". "Early career researcher" is defined as less than 5 years as an independent researcher (including self-identified graduate students and postdoctoral fellows), "Mid-career researcher" is defined as 5-10 years as an independent researcher, "Senior Researcher" is defined as more than 10 years as an independent researcher. "Other" includes respondents who identified themselves as "Knowledge User" and "Other". P1 refers to Pillar 1 (Biomedical), P2 refers to Pillar 2 (Clinical), P3 refers to Pillar 3 (Health System and Services), and P4 refers to Pillar 4 (Social, Cultural, Environmental and Population). Note that Knowledge Users represent approximately 46% of the "Other" sample. Data as of May 1, 2012 (N = 519, which includes duplicate responses included from Senior Researcher/Knowledge User and Mid-Career Researcher/Knowledge User combinations).

The strongest levels of agreement came from:

• 83% of Pillar 2 (Clinical) and 77% of Pillar 3 (Health Systems and Services) early career researchers;
• 77% of "other" respondents (including Knowledge Users).

The strongest levels of disagreement came from:

• 53% of mid-career and 57% of senior Pillar 1 (Biomedical) researchers.

B) Analysis of Narrative Responses

Approximately 62% of respondents (n = 1,038) from all feedback venues submitted comments and/or questions about the multi-phased competition process. The structured responses found that approximately 50% of respondents were in agreement and 37% of respondents were in disagreement with the principles behind this design element. The narrative responses reflected mixed opinions about the feasibility of the multi-phased competition process, and its ability to reduce applicant and peer reviewer burden. Details of the comments regarding the multi-phased competition process focused on the following themes: Stage 1 and 2 application and review; Stage 3 review; the Adjudication and Decision Process; and, the Competition Process.

Stage 1 and 2 Application and Review

Increasing or decreasing applicant and/or peer reviewer burden were frequently discussed when considering the proposed multi-phased competition process. Of the respondents who provided comments on applicant burden, several submitted positive comments about the proposed shorter applications (particularly at Stage 1), and believed the use of defined application requirements at each stage would reduce the overall amount of time it would take to complete an application (as compared to completing a single, full application).

Of the respondents who provided comments on peer reviewer burden, most believed that the greater number of review stages, combined with more reviewers per application, would increase the overall workload and time commitment for peer reviewers. Many respondents felt that short applications would increase application pressure by enticing more researchers to submit applications to CIHR, and enabling frequent re-submissions of both competitive and non-competitive applications. A few respondents suggested implementing an application quota to manage application pressure, while some respondents proposed that CIHR work with Institutions and take advantage of Institution internal review processes to screen out non-competitive applications.

Some respondents indicated that the proposed multi-phase review process could reduce reviewer burden, and were supportive of the opportunity to quickly screen out non-competitive applications.

Stage 3 Review

Of the respondents who discussed Stage 3 Review, which involves a face-to-face review of applications that did not reach consensus at the remote review stage, most had questions regarding the size and composition of interdisciplinary committees, and how to integrate or weight the points of view of reviewers with potentially disparate backgrounds.

Several agreed with the principle of having applications close to the funding cut-off range (the "grey zone") further examined by a committee. However, many did not believe a generalist panel of experts from multiple disciplines would be able to adequately evaluate applications that fell close to the funding cut-off range (the "grey zone"). Some suggested having the same reviewers from Stage 2 populate the interdisciplinary panel to ensure sufficient understanding of the applications, and fairness in review.

Adjudication and Decision Process

Of the respondents who discussed scoring practices, many agreed with CIHR's position that scoring should only be used as a provisional working tool, and that applications should be rated as either "meritorious" or "not meritorious," and ranked against a pool of applications. Some respondents were concerned about reviewing and ranking within a very heterogeneous pool of applications, in that individually rating and ranking applications could potentially introduce bias and decrease the consistency and fairness of reviews. However, an equal number of respondents commented that bundling several applications for a single reviewer's assessment and ranking could help calibrate decisions. Several respondents were also in favour of including an opportunity for applicants to provide clarifications to reviewers, when needed, to ensure a fair review of their application.

Competition Process

Some respondents identified the need for CIHR to provide more information on the submission, decision, and re-submission process. Several commented that there should be sufficient time between the Stage 1 Review decision and Stage 2 Application deadline to develop a quality proposal, and a few suggested at least 2-3 months would be ideal.

Many of respondents who commented on the competition process were in favour of holding fewer competition cycles per year, with the caveat that transitional support (e.g., bridge funding) be available for those whose grant renewals were not successful.

3.2.2 Application-Focused Review

A) Structured Responses from the Feedback Form

According to the summary of responses from the Design Discussion Document Feedback Form, approximately 67% of respondents agreed or strongly agreed when asked whether Application-Focused Review would help address CIHR's current challenges with its Open Suite of Programs and peer review system (Figure 12).

Figure 12: Feedback Form Respondent Degree of Agreement: Application-Focused Review addresses challenges with CIHR's Open Suite of Programs and peer review system

Figure 12. Feedback Form responses showing the extent to which respondents agreed or disagreed that Application-Focused Review would help address CIHR's current challenges with its Open Suite of Programs and peer review system. Data as of May 1, 2012 (N = 512). Totals may not agree due to rounding.

Further breakdown of structured responses (by self-identified pillar and career stage) indicated that general agreement with the ability of Application-Focused Review to address current challenges with the Open Suite of Programs and peer review system was highest among early career researchers and "other" respondents, which include Knowledge Users. Researchers from Pillars 2 (Clinical) and 3 (Health Systems and Services) and 4 (Social, Cultural, Environmental and Population) also showed high levels of agreement. On the other hand, general disagreement with the ability of Application-Focused Review to address current challenges with the Open Suite of Programs and peer review system was highest with mid-career and senior researchers, and with Pillar 1 (Biomedical) researchers (Figure 13).

Figure 13: Feedback Form Respondent Degree of Agreement: Application-Focused Review addresses challenges with CIHR's Open Suite of Programs and peer review system (by self-identified pillar and career stage)

Figure 13. Feedback Form responses showing the percentage of respondents who believe that application-focused review would help address CIHR's current challenges with its Open Suite of Programs and peer review system, by self-identified pillar and career stage. Responses classified as "in agreement" include those who responded as "agree" and "strongly agree", respondents classified as "in disagreement" include those who responded as "disagree" and "strongly disagree", and respondents classified as "neutral" include those who responded as "neither agree nor disagree", "don't know" and "(blank)". "Early career researcher" is defined as less than 5 years as an independent researcher (including self-identified graduate students and postdoctoral fellows), "Mid-career researcher" is defined as 5-10 years as an independent researcher, "Senior Researcher" is defined as more than 10 years as an independent researcher. "Other" includes respondents who identified themselves as "Knowledge User" and "Other". P1 refers to Pillar 1 (Biomedical), P2 refers to Pillar 2 (Clinical), P3 refers to Pillar 3 (Health System and Services), and P4 refers to Pillar 4 (Social, Cultural, Environmental and Population). Note that Knowledge Users represent approximately 46% of the "Other" sample. Data as of May 1, 2012 (N = 519, which includes duplicate responses included from Senior Researcher/Knowledge User and Mid-Career Researcher/Knowledge User combinations).

The strongest levels of agreement came from:

• 77% of Pillars 1 (Biomedical), 78% of Pillar 2 (Clinical), 100% of Pillar 3 (Health System and Services) and 83% of Pillar 4 (Social, Cultural, Environmental and Population) early career researchers;
• 80% of senior Pillar 3 (Health Systems and Services) researchers;
• 93% mid-career and 80% senior Pillar 4 (Social, Cultural, Environmental and Population).

The strongest levels of disagreement came from:

• 24% of mid-career, and 25% of senior Pillar 1 (Biomedical) researchers;
• 29% of mid-career Pillar 3 researchers (Health Systems and Services).

B) Analysis of Narrative Responses

Approximately 33% of respondents (n = 562) from all feedback venues submitted comments and/or questions about application-focused review. The structured responses found relatively equal balance between those in agreement and those in disagreement. The narrative responses were found to be generally supportive. Details of the comments regarding application-focused review focused on the following themes: Matching of Appropriate Expertise to Individual Applications; Types of Reviewers; Reviewer Workload; and, On Discontinuing the Standing Peer Review Committee Structure.

Matching of Appropriate Expertise to Individual Applications

Overall, several respondents agreed with CIHR's position that matching the right mix of expertise to individual applications would improve the quality and fairness of reviews. This perspective was further supported by respondents working in interdisciplinary areas of health research, as well as respondents from Pillars 2 (Clinical), 3 (Health System and Services) and 4 (Social, Cultural, Environmental and Population).

Although matching was supported, some respondents cautioned that in very specialized disciplines, having an application matched to 5-8 reviewers could increase the potential for conflicts of interest. Stricter guidelines would be needed to ensure intellectual property is respected, and the integrity of review is maintained.

Several respondents questioned how CIHR intends to match applications to reviewers. A few suggested that applicants be allowed to choose their reviewers, whereas the majority of other respondents agreed that using common descriptors would help bundle pools of qualified reviewers. All respondents agreed that the use of electronic matching is not sufficient, and that the current manual matching of reviewers to applications is preferred. Some identified that Scientific Officers and Committee Chairs could adopt this role.

A few respondents expressed interest in helping CIHR identify keywords relevant to their areas of research. A number of keywords were submitted for CIHR`s consideration.

Types of Reviewers

Some respondents provided their comments and suggestions on the types of reviewers they would like to have review their applications. Of these respondents, several emphasized that applications should be assessed by expert reviewers with similar interests to the application.

The feedback revealed a general discomfort from respondents concerning the use of non-expert (generalists and lay) reviewers. Most respondents who commented on the use of non-experts agreed that giving equal weight to the points of view of non-experts may decrease the quality of reviews. However, a few respondents noted that perspectives of non-expert reviewers, such as patients and other knowledge users, may be valuable in assessing potential impact and uptake of research results. These respondents also noted that, while research proposals should incorporate the most modern scientific approaches, they should be expressed in a manner that can be evaluated by any competent reviewer.

Reviewer Workload

Several respondents remarked on the potential for increased reviewer workload as a result of the implementation of application-focused review. Of these respondents, many commented that assigning 5 to 8 reviewers per application was too much, whereas a few noted that a greater number of reviewers would be useful for calibrating review decisions. A few respondents expressed concern that finding 5-8 reviewers for a single application may be difficult considering the size and scope of Canada's research community, and suggested CIHR consider assigning no more than four reviewers per application.

On Discontinuing the Standing Peer Review Committee Structure

While several agree that matching the appropriate expertise to individual applications would improve the quality of reviews, several respondents commented that the problems with CIHR's current panels are not so severe as to warrant a complete elimination of the peer review committee structure. Discontinuing the traditional standing committee structure was seen as a threat to the reliability and accountability of the peer review process.

Of those respondents who advocated that CIHR keep its current peer review committee structure, several suggested CIHR re-focus its current set of peer review committees, with consolidation of smaller panels into larger panels with broader mandates.

3.2.3 Structured Review Criteria

A) Structured Responses from the Feedback Form

According to the summary of responses from the Design Discussion Document Feedback Form, approximately 60% of respondents agreed or strongly agreed when asked whether the structured review criteria would help address CIHR's current challenges with its Open Suite of Programs and peer review system (Figure 14).

Figure 14: Feedback Form Respondent Degree of Agreement: The use of Structured Review Criteria addresses challenges with CIHR's Open Suite of Programs and peer review system

Figure 14. Feedback Form responses showing the extent to which respondents agreed or disagreed that the structured review criteria would help address CIHR's current challenges with its Open Suite of Programs and peer review system. Data as of May 1, 2012 (N = 512). Totals may not agree due to rounding.

Further breakdown of structured responses (by self-identified pillar and career stage) indicated that agreement with the use of Structured Review Criteria to address current challenges with the Open Suite of Programs and peer review system was highest among early career researchers and "other" respondents, which include Knowledge Users. Researchers from Pillars 2 (Clinical) and 3 (Health Systems and Services) also showed high levels of agreement. On the other hand, general disagreement with use of Structured Review Criteria to address current challenges with the Open Suite of Programs and peer review system was highest with senior career researchers, and with Pillar 1 (Biomedical) researchers (Figure 15).

Figure 15: Feedback Form Respondent Degree of Agreement: The use of Structured Review Criteria addresses challenges with CIHR's Open Suite of Programs and peer review system (by self-identified pillar and career stage)

Figure 15. Feedback Form responses showing the percentage of respondents who believe that the use of structured review criteria would help address CIHR's current challenges with its Open Suite of Programs and peer review system, by self-identified pillar and career stage. Responses classified as "in agreement" include those who responded as "agree" and "strongly agree", respondents classified as "in disagreement" include those who responded as "disagree" and "strongly disagree", and respondents classified as "neutral" include those who responded as "neither agree nor disagree", "don't know" and "(blank)". "Early career researcher" is defined as less than 5 years as an independent researcher (including self-identified graduate students and postdoctoral fellows), "Mid-career researcher" is defined as 5-10 years as an independent researcher, "Senior Researcher" is defined as more than 10 years as an independent researcher. "Other" includes respondents who identified themselves as "Knowledge User" and "Other". P1 refers to Pillar 1 (Biomedical), P2 refers to Pillar 2 (Clinical), P3 refers to Pillar 3 (Health System and Services), and P4 refers to Pillar 4 (Social, Cultural, Environmental and Population). Note that Knowledge Users represent approximately 46% of the "Other" sample. Data as of May 1, 2012 (N = 519, which includes duplicate responses included from Senior Researcher/Knowledge User and Mid-Career Researcher/Knowledge User combinations).

The strongest levels of agreement came from:

• 89% of Pillar 2 (Clinical), 94% of Pillar 3 (Health System and Services), and 83% of Pillar 4 (Social, Cultural, Environmental and Population) early career researchers and;
• 88% of mid-career Pillar 3 (Health Systems and Services) researchers.

The strongest levels of disagreement came from:

• 35% of senior Pillar 1 (Biomedical) researchers;
• 29% of mid-career Pillar 1 (Biomedical) researchers.

B) Analysis of Narrative Responses

Approximately 26% of respondents (n = 439) from all feedback venues submitted comments and/or questions about the use of structured review criteria. The structured responses found that approximately 60% of respondents were in agreement and 21% of respondents were in disagreement with the principles behind this design element. The narrative responses were generally supportive of the use of structured review criteria with applicants demonstrating interest in engaging with CIHR to suggest criteria considerations and potential indicators of success. Details of the comments regarding the use of structured review criteria focused on the following themes: The Use of Structured Evaluation Criteria for All Types of Research; Application Criteria; Improving the Reliability, Consistency and Fairness of Review; and, Feedback to Applicants.

The Use of Structured Evaluation Criteria for All Types of Research

Overall, many respondents responded positively to CIHR's proposal to incorporate clear objective review criteria into the peer review process. Some applicants commented that structured review criteria would provide them with additional guidance when writing grant applications, and a few peer reviewers agreed that the use of structured evaluation criteria could improve the quality of feedback provided to applicants.

Several respondents remarked on the subjective nature of peer review. Some respondents commented that the use of structured review criteria would mean conforming to a "one size fits all" model, which would inadvertently disadvantage health researchers outside of the biomedical community (namely, researchers from the social sciences, emergent, or multidisciplinary areas of health research). However, several respondents agreed that structured review criteria should be flexible enough to accommodate different disciplinary expectations, and be applied according to the accepted standards of excellence for those disciplines. A number of evaluation frameworks and review criteria were suggested for CIHR's consideration, including:

• The Grade Scale;
• The Jadad Scale;
• PEDro Evaluation Criteria;
• The CLASP framework;
• NIH's peer review criteria for Impact, Investigator(s), Innovation, Plan and Environment.

Several respondents from all areas of health research also provided CIHR with criteria for consideration and suggested indicators for Track Record, Impact, Innovation, Productivity, Knowledge Translation, Institutional Support and the Quality of an Idea.

Application Criteria

A few respondents noted that in refreshing CIHR's application criteria, there is an opportunity to simplify CIHR's application modules. Several respondents requested more information about future application requirements for the new schemes, so that they could include the right information in their application packages.

With regard to specific suggestions, budgetary requirements were the most frequently discussed application criteria. Some respondents proposed that CIHR should not ask for a formal budget, but should instead define tiers of funding an applicant could select from. A few of these respondents suggested CIHR engage institutions to assess the cost of research across the different disciplines and regions.

Improving the Reliability, Consistency and Fairness of Peer Review

Some respondents believed the consistent use of standard evaluation criteria would improve the fairness, reliability, and consistency of review, but only if criteria were clearly understood by both reviewers and applicants. Several respondents were in agreement that clear criteria descriptors and usage guidelines would be necessary to ensure the appropriate application of review criteria for all types of applications.

Feedback to Applicants

Some respondents believed the use of structured review criteria could facilitate more meaningful interactions between reviewers and applicants, depending on the type of review criteria proposed. Of these respondents, several commented that feedback provided via structured review could inform constructive changes to grant applications.

However, some respondents expressed concern that the structured review process would become an arbitrary "box ticking" exercise. These respondents noted that the added structure would leave little room to convey feedback they believe is important, but that does not fit under specific, structured headings.

3.2.4 Remote (Virtual) Screening Process

A) Structured Responses from the Feedback Form

According to the summary of responses from the Design Discussion Document Feedback Form, approximately 48% of respondents agreed or strongly agreed when asked whether the Remote (Virtual) Screening Process would help address CIHR's current challenges with its Open Suite of Programs and peer review system (Figure 16).

Figure 16: Feedback Form Respondents' Degree of Agreement: The Remote (Virtual) Screening Process addresses challenges with CIHR's Open Suite of Programs and peer review system

Figure 16. Feedback Form responses showing the extent to which respondents agreed or disagreed that the Remote (Virtual) Screening Process would help address CIHR's current challenges with its Open Suite of Programs and peer review system. Data as of May 1, 2012 (N = 513). Totals may not agree due to rounding.

Further breakdown of structured responses (by self-identified pillar and career stage) found that general agreement with the Remote (Virtual) Screening Process' ability to address current challenges with the Open Suite of Programs and peer review system was highest among early career researchers and "other" respondents, which include Knowledge Users. Researchers from Pillars 2 (Clinical) and 3 (Health Systems and Services) also showed high levels of agreement. On the other hand, general disagreement with the Remote (Virtual) Screening Process' ability to address current challenges with the Open Suite of Programs and peer review system was highest amongst Pillar 1 (Biomedical) researchers (Figure 17).

Figure 17: Feedback Form Respondent Degree of Agreement: The Remote (Virtual) Screening Process addresses challenges with CIHR's Open Suite of Programs and peer review system (by self-identified pillar and career stage)

Figure 17. Feedback Form responses showing the percentage of respondents who believe that the remote (virtual) screening process would help address CIHR's current challenges with its Open Suite of Programs and peer review system, by self-identified pillar and career stage. Responses classified as "in agreement" include those who responded as "agree" and "strongly agree", respondents classified as "in disagreement" include those who responded as "disagree" and "strongly disagree", and respondents classified as "neutral" include those who responded as "neither agree nor disagree", "don't know" and "(blank)". "Early career researcher" is defined as less than 5 years as an independent researcher (including self-identified graduate students and postdoctoral fellows), "Mid-career researcher" is defined as 5-10 years as an independent researcher, "Senior Researcher" is defined as more than 10 years as an independent researcher. "Other" includes respondents who identified themselves as "Knowledge User" and "Other". P1 refers to Pillar 1 (Biomedical), P2 refers to Pillar 2 (Clinical), P3 refers to Pillar 3 (Health System and Services), and P4 refers to Pillar 4 (Social, Cultural, Environmental and Population). Note that Knowledge Users represent approximately 46% of the "Other" sample. Data as of May 1, 2012 (N = 519, which includes duplicate responses included from Senior Researcher/Knowledge User and Mid-Career Researcher/Knowledge User combinations).

The strongest levels of agreement came from:

• 84% of senior Pillar 3 (Health Systems and Services) researchers;
• Early career researchers from Pillar 2 (Clinical - 72%), Pillar 3 (Health System and Services – 77%) and Pillar 4 (Social, Cultural, Environmental and Population – 74%).

The strongest levels of disagreement came from:

• 49% of mid-career and 49% of senior Pillar 1 (Biomedical) researchers.

B) Analysis of Narrative Responses

Approximately 36% of respondents (n = 603) from all feedback venues submitted comments and/or questions about the remote (virtual) screening process. The structured responses found that approximately 48% of respondents were in agreement and 33% of respondents were in disagreement with the principles behind this design element. The narrative responses reflected mixed opinions about the remote (virtual) screening process; showing a balance of support for the use of internet-assisted technology to facilitate application-focused review, as well as concerns over calibration of reviews and the re-purposing of face-to-face discussions. Details of the comments regarding the remote (virtual) screening process focused on the following themes: Using Internet-assisted Technology; When to Use the Remote Screening Process; The Importance of Face-to-Face Discussions; and, the Implementation of the Remote (Virtual) Screening Process.

Using Internet-Assisted Technology

Among those that agreed with the adoption of a remote screening process, the use of Internet-assisted technology was cited as the biggest benefit. In addition to cost-savings, which most respondents proposed should be re-directed to funding more grants, some respondents expressed that the use of Internet-assisted technology could:

• Facilitate discussions between reviewers in different geographical regions (including international experts);
• Reduce biases that may arise in group discussions, and support more accurate initial assessments and less variable scores;
• Reduce travel-related burden; and
• Contribute to environmental sustainability.

Some provided suggestions describing what type of Internet-assisted technology would be most successful in maintaining meaningful interaction and discussions between reviewers. Overall, the use of instant messaging tools was strongly discouraged, whereas the use of videoconferencing tools (e.g., Skype) was preferred.

Regardless of the tool proposed, some respondents suggested there should be an oversight mechanism to maintain a level of quality and integrity to Internet-assisted discussions. Employing an engaged chairperson, or moderator, was proposed. A few respondents remarked that this would be a natural transition for the role of CIHR's Scientific Officers.

When To Use the Remote Screening Process

In general, respondents noted that the most beneficial use of a remote screening process would be to identify meritorious applications at Stage 1 of the multi-phased competition process. Most respondents emphasized that Stage 2 applications should be reviewed by a face-to-face committee of expert reviewers to ensure a well-considered, calibrated decision is made on the overall merit of the applications.

On the other hand, the retention of face-to-face discussions for meritorious applications ranked in the "grey zone" was seen as a good compromise by the few respondents who expressly preferred face-to-face discussions, but agreed with the principles behind this design element.

The Importance of Face-to-Face Discussions

Overall, the most frequently cited concerns from respondents about the remote (virtual) screening process were related to the impact that losing face-to-face discussions would have on the quality of review. Of these responses,

• Many respondents expressed discomfort with judging the merit of an application without validation/calibration from their peers;
• Several respondents remarked that their peers would not review an application to the best of their abilities if not held accountable in a face-to-face setting;
• Several respondents noted that face-to-face discussion is necessary to neutralize individual biases.

Some respondents pointed out that the remote screening process would be counter-productive and problematic, and suggested that the only reason that the proposed change was being considered was to address budgetary constraints.

Respondents also cautioned that intangible benefits to the current peer review system would be lost if CIHR adopted a remote screening process. For example, a few of respondents noted the loss of opportunities to network with peers in a face-to-face setting and set the foundation for future collaborations. A few early career respondents raised concerns that the new system would offer little opportunity to learn about peer review from more experienced peers.

Implementation of Remote (Virtual) Screening Process

A few respondents noted the lack of detail in the Design Discussion Document regarding the implementation of the remote screening process. Of these, some expressed they were not wholly convinced that this new process would reduce peer reviewer burden, and were concerned about their ability to effectively use a new web-based system.

A few respondents also expressed concerns about the logistics of the remote review process. While this mechanism would facilitate access to a greater mix of reviewers, including international experts, it was pointed out that it may be difficult to coordinate an online discussion with reviewers who live in very different time zones.

3.2.5 College of Reviewers

A) Structured Responses from the Feedback Form

According to the summary of responses from the Design Discussion Document Feedback Form, approximately 57% of respondents agreed or strongly agreed when asked whether the College of Reviewers would help address CIHR's current challenges with its Open Suite of Programs and peer review system (Figure 18).

Figure 18: Feedback Form Respondents' Degree of Agreement: The College of Reviewers addresses challenges with CIHR's Open Suite of Programs and peer review system

Figure 18. Feedback Form responses showing the extent to which respondents agreed or disagreed that the College of Reviewers would help address CIHR's current challenges with its Open Suite of Programs and peer review system. Data as of May 1, 2012 (N = 512). Totals may not agree due to rounding.

Further breakdown of structured responses (by self-identified pillar and career stage) indicated that agreement with the College of Reviewers' ability to address current challenges with the Open Suite of Programs and peer review system was highest among early career and mid-career researchers, as well as "other" respondents, which include Knowledge Users. Researchers from Pillars 3 (Health System and Services) and 4 (Social, Cultural, Environmental and Population) also showed high levels of agreement. On the other hand, disagreement with the College of Reviewers' ability to address current challenges with the Open Suite of Programs and peer review system was highest with senior researchers, and with Pillar 1 (Biomedical) researchers (Figure 19).

Figure 19: Feedback Form Respondent Degree of Agreement: The College of Reviewers addresses challenges with CIHR's Open Suite of Programs and peer review system (by self-identified pillar and career stage)

Figure 19. Feedback Form responses showing the percentage of respondents who believe that College of Reviewers would help address CIHR's current challenges with its Open Suite of Programs and peer review system, by self-identified pillar and career stage. Responses classified as "in agreement" include those who responded as "agree" and "strongly agree", respondents classified as "in disagreement" include those who responded as "disagree" and "strongly disagree", and respondents classified as "neutral" include those who responded as "neither agree nor disagree", "don't know" and "(blank)". "Early career researcher" is defined as less than 5 years as an independent researcher (including self-identified graduate students and postdoctoral fellows), "Mid-career researcher" is defined as 5-10 years as an independent researcher, "Senior Researcher" is defined as more than 10 years as an independent researcher. "Other" includes respondents who identified themselves as "Knowledge User" and "Other". P1 refers to Pillar 1 (Biomedical), P2 refers to Pillar 2 (Clinical), P3 refers to Pillar 3 (Health System and Services), and P4 refers to Pillar 4 (Social, Cultural, Environmental and Population). Note that Knowledge Users represent approximately 46% of the "Other" sample. Data as of May 1, 2012 (N = 519, which includes duplicate responses included from Senior Researcher/Knowledge User and Mid-Career Researcher/Knowledge User combinations).

The strongest levels of agreement came from:

• 93% of mid-career Pillar 4 (Social, Cultural, Environmental and Population) researchers;
• 82% of early career Pillar 3 (Health Systems and Services) researchers;

The strongest levels of disagreement came from:

• 37% of senior Pillar 1 (Biomedical) researchers.

B) Analysis of Narrative Responses

Approximately 61% of respondents (n = 1,021) from all feedback venues submitted comments and/or questions about the College of Reviewers. The structured responses found that approximately 57% of respondents were in agreement and 22% of respondents were in disagreement with the principles behind this design element. The narrative responses were found to be generally supportive, and focused on suggestions and considerations that could improve the College of Reviewers design. Details of the comments regarding the College of Reviewers focused on the following themes: Recruitment of qualified reviewers; The Use of Incentives; Peer Reviewer Training and Evaluation; and the Implementation of the College of Reviewers.

Recruitment of Qualified Reviewers

Many respondents agreed with CIHR's position that a larger pool of reviewers is needed to manage peer reviewer burden. There is general agreement from respondents that the College of Reviewers is a good way to recruit and manage a broad base of expert reviewers from a wide range of health research fields (including the social sciences and humanities). Members of the College of Reviewers should include a mix of senior investigators, junior investigators (including postdoctoral fellows), clinicians, regional representatives, knowledge users, and members of the community. Transparency in the recruitment process is essential, and a few respondents cautioned against creating an elitist club of reviewers from a very narrow range of disciplines.

A few respondents expressed concern that CIHR would not be able to recruit a sufficient number of qualified (expert) reviewers across the spectrum of health research to support the current scope of CIHR's programs and initiatives. A few suggested that CIHR recruit international experts to increase the breadth of experts available to review a grant, and decrease overall peer reviewer burden. A more common suggestion from several respondents was the mandatory enrollment of current CIHR grant holders to the College of Reviewers, with repercussions for refusing the invitation (e.g., restricted eligibility for future funding from CIHR). However, a few respondents cautioned that forced participation in the College may be seen as a disincentive, and could reduce the quality of reviews.

The Use of Incentives

Although many respondents agreed that CIHR should actively recruit more peer reviewers, some proposed that CIHR should make use of incentives to attract qualified reviewers, and maintain excellence in reviewing. Of the comments received about the use of incentives, most respondents agreed that incentives should include some form of compensation to reviewers, such as:

• Increased duration or value of CIHR research grants, commensurate with participation and performance;
• Application deadline extensions;
• Honoraria (particularly for international reviewers).

Peer Reviewer Training and Evaluation

Some respondents were supportive of CIHR's proposal to train reviewers to improve the reliability, consistency, and fairness of reviews. A few respondents commented on the variability of reviews, the need to have a common understanding of the process, and how to write appropriate critiques. Of these, some suggested that training modules include explicit examples of excellent and sub-standard peer review reports.

Beyond training, a few respondents further proposed that the quality and integrity of the College be maintained through regular performance evaluations of its reviewers. An analysis of scoring patterns could be used to either screen out poor reviewers, or identify opportunities for further reviewer training.

Implementation of the College of Reviewers

A few respondents noted that while this design element was laudable, implementing the College of Reviewers may be a challenge. A few respondents commented that there will be opportunities to share peer review resources once the College of Reviewers has been implemented. Respondents from voluntary health organizations expressed that the College should not only benefit CIHR, but also the broader Canadian research community. Similarly, a few researchers agreed that College could also be used in the peer review of applications to strategic funding opportunities.

3.3 Transition

This section focuses on the feedback received related to involving the research community in the design discussions, validating the proposed design, the timing of implementation, transitioning to the new funding schemes, and monitoring and evaluating the outcomes of implementation.

Approximately 50% of respondents (n = 835) from all feedback venues submitted comments and/or questions about transitioning to the new Open Suite of Programs and peer review process. Overall, narrative responses focused on describing how CIHR should proceed in order to gain buy-in from the research community and ensure a smooth transition. Details of the comments regarding transition focused on the following themes: Involvement of the research community; Validating the Proposed Design; Timing of Implementation; Transitioning to New Funding Schemes; and, Monitoring and Evaluating the Outcomes of Implementation.

Involvement of the Research Community

Of the comments received regarding the transition, many were related to engagement of the community. Some respondents commented that CIHR could do more to engage the research community and would welcome the opportunity to be more involved in the design of the new competitions. Some expressed concern that the changes were moving forward too quickly to allow for meaningful engagement and consultation with stakeholders and many remarked that the proposed timelines for the implementation were too ambitious and that more evidence was needed to ensure that the changes would have the intended effect when implemented. A few respondents commented that the changes would have significant impact on other funders of health research in Canada, and recommended that these players be more involved in shaping the changes. A few respondents also noted that there are now opportunities to harmonize with other funding organizations to maximize the amount of funding available to sustain the health research enterprise.

Validating the Proposed Design

Most of the respondents in this area indicated that further evidence is needed to ensure the changes will have the intended outcome when implemented. Many respondents suggested that the changes should be implemented gradually after careful consideration of the results of pilot studies for the proposed changes. Some respondents proposed that the modeling of the proposed changes should be presented to the community through an ongoing engagement process regarding the changes. A few respondents cited specific concerns where they would like to see more modeling. These include evidence that the new system will continue to support the same number of nominated principal investigators as in the current system; evidence that funding for curiosity-driven research will not be compromised by the perceived increased focus on solution-driven research; and evidence that new- and mid-career investigators will have at least an equal opportunity in the new system.

Timing of Implementation

Some respondents noted that implementation of changes of this magnitude will be a great challenge for CIHR and the community. Most respondents who commented on the implementation emphasized that any changes should be phased-in, allowing for impact analysis at each stage to ensure that the changes have the intended outcome. Some suggested that changes to the peer review system should be tested and implemented in advance of any changes to the architecture, as a strong peer review system is critical to the overall success of the reforms.

Transition to New Funding Schemes

Some respondents identified potential challenges regarding the transition from the current competitions to the new schemes. A few respondents pointed out that it may be difficult to bundle a nominated principal investigator's existing grants into a single Foundation/Programmatic Research grant if the investigator belongs to multiple research teams with very different research goals. A few also commented that more information regarding policies for this roll-up is needed, particularly in the context of early renewals and the implications/risks of applying for a Foundation/Programmatic Research grant for their current funding. Detailed information on the transition plan will be required in order for researchers to make informed decisions about when and where to submit future grant applications.

A few respondents anticipate that application pressure will be significantly higher than normal on both the last competition of the current system and the first competition of the new system, and emphasized the need to mitigate this.

It was noted by some respondents that the community will go through an adjustment period where significant training and guidance will be required for applicants, university administrators, and peer reviewers. A few respondents suggested that the changes may be more difficult for researchers at smaller institutions who may not have as many supports in place.

Monitoring and Evaluating the Outcomes of Implementation

Some of the respondents commented on the need for ongoing monitoring and evaluation of the changes. Of those, most agreed that CIHR must develop a systematic evaluation plan for each aspect of the reforms as well as the effect on the health research enterprise as a whole. Some of the respondents suggested that the main focus of such an evaluation should be on the success of the three-stage competition process, while a few respondents expressed that CIHR should focus on monitoring the impact of the changes, specifically for new/early career investigators.

4. Next Steps

CIHR thanks the research community, partners, and other CIHR stakeholders for taking the time to consider, and provide feedback on, the proposed changes to the new Open Suite of Programs and peer review processes. The feedback has identified mixed views with respect to the specific details about the funding schemes and how peer review will be conducted. There is a lot of work left to do and decisions to be made in order to develop a competitive and sustainable system that supports the creation and translation of health research across all health fields. As such, the next two CIHR Open Operating Grants Program (OOGP) competitions will be launched according to usual (current) processes.

In the coming months, CIHR will work with a variety of stakeholders including institutions, the University Delegates, the Institute Advisory Boards, and other advisors to further develop and refine the design. We recognize the research community's increasing interest in learning more about the Foundation/Programmatic Research and Project Schemes. Engaging with the research community continues to be a priority for CIHR. We will be updating institutions and partners about the progress made in developing and refining the design through presentations and discussions at upcoming meetings.

CIHR views the transition to the new Open Suite of Programs as a multi-year process. Given the scope of the proposed changes, CIHR acknowledges that course corrections may be required along the way. CIHR is committed to piloting and rigorously evaluating various aspects of the reforms, and will be investigating opportunities to study the feasibility and functionality of the proposed design elements as part of its implementation plan. Proceeding in an organized fashion will allow CIHR to minimize disruption, and provide opportunities to adjust the system, as required.

Keeping the research community and other stakeholders involved and informed as we move forward with the development and implementation of the reforms is important to CIHR. Our work with the University Delegates and others will continue to ensure researcher concerns are considered at every step of the process. Updates will also be posted to CIHR's website and disseminated through e-Alerts throughout the process, with the publication of the design details and implementation plan anticipated for the fall 2012.

As the major federal funder of health research in Canada, CIHR must support the creation and translation of health research across all domains. This support must be sustainable over the long term and must maintain Canada's competitiveness in today's knowledge-based economy.

The goal of the proposed designs is to ensure that CIHR's funding and peer review systems are capable of identifying and supporting research excellence across the entirety of its mandate. The feedback provided by Canada's health research community will prove invaluable as CIHR further refines the design elements.

Ultimately, the modernization of CIHR's funding and peer review systems will strengthen Canada's health research enterprise, and improve health outcomes for Canadians.

Annex I: Feedback on the Design Discussion Document Survey

Part A: Basic Information

1. Would you classify yourself as a:
   Please select all that apply:
   1. Early career researcher (<5 years as an independent researcher)
   2. Mid-career researcher (5-10 years as an independent researcher)
   3. Senior researcher (>10 years as an independent researcher)
   4. Knowledge User
   5. Other (specify):
2. Which research position(s) do you currently hold at this time?
   Please select all that apply:
   1. Professor
   2. Assistant Professor
   3. Associate Professor
   4. Researcher
   5. Research Assistant or similar position
   6. Clinician
   7. Intern
   8. Other (specify):
3. Which of the following is your primary research domain (pillar):
   1. Biomedical
   2. Clinical
   3. Health systems/services
   4. Social, cultural, environmental and population health
4. Have you peer reviewed for CIHR in the past 5 years?
   1. Yes
   2. No

5. Have you applied for an Open Operating Grant in the past 3 years?

   1. Yes
   2. No

   If Yes, please answer Questions #6, 7 and 8:

6. On average, how long does it take you to prepare a full Open Operating Grants application package (including attachments) for electronic submission to ResearchNet (in hours)?

7. Do you have an internal deadline at your institution that precedes the CIHR deadline for applications?
   1. Yes
   2. No
8. Do you have an internal peer review process at your institution?
   1. Yes
   2. No

Part B: Design Discussion Document

1. The Design Discussion Document adequately describes the challenges with our current Open Suite of Programs and peer review system.
   1. Strongly Agree
   2. Agree
   3. Disagree
   4. Strongly Disagree
   5. Don't Know
2. Having read the Design Discussion Document, the distinction between the Foundation/Programmatic Research Scheme and the Project Scheme is clear.
   1. Strongly Agree
   2. Agree
   3. Disagree
   4. Strongly Disagree
   5. Don't Know
3. Having read the Design Discussion Document, I would characterize myself as someone who would apply to:
   1. The Foundation/Programmatic Research Scheme
   2. The Project Scheme
   3. Both
   4. Neither
4. Having read the Design Discussion Document, I believe the proposed changes would reduce barriers to funding excellence across the full spectrum of health research.
   1. Strongly Agree
   2. Agree
   3. Disagree
   4. Strongly Disagree
   5. There are no barriers
5. As described in the Design Discussion Document, to what extent do you agree or disagree that the following design elements would help address CIHR's current challenges with its Open Suite of Programs and peer review system?

   	Strongly Agree	Agree	Neither Agree nor Disagree	Disagree	Strongly Disagree	Don't know
   Multi-phased competition process This design element is intended to screen the number of applicants that complete full applications and reduce the length of time required to review applications at each stage
   Application-focused review This design element is intended to match applications to reviewers to ensure the appropriate expertise is assigned to each application
   Integrated Knowledge Translation This design element is intended to recognize the importance of knowledge users, and would support collaborative, applied research
   Structured Review Criteria This design element is intended to provide clearly defined review criteria and relevant application information to support fair, reliable and consistent peer review evaluations
   Remote (virtual) screening process This design element is intended to utilize internet-assisted technology to support matching for application-focused review
   College of Reviewers This design element is intended to facilitate access to appropriate expertise, and provide the framework for mechanisms to recruit, train and reward reviewers

6. What are the strengths of the design that is being considered?

7. What are the gaps in this design that CIHR should address to ensure a successful implementation?

8. What challenges do you anticipate as a researcher/peer reviewer in adopting these changes?

9. Other Comments:

Annex II: Correspondence Received from Researchers, Institution Administrators and other stakeholders by Institution or Health Research-related Organizations

From February 8 – May 1, 2012

• Association of Faculties of Medicine of Canada
• Athabasca University
• B-Temia
• Bloorview Research Institute
• Bruyère Research Institute
• Canadian Arthritis Network
• Canadian Association of Gastroenterology
• Canadian Association of Schools of Nursing
• Canadian Association of University Research Administrators (CAURA)
• Canadian Centres for Research in Health Professions Education
• Canadian Dental Hygienists Association
• Canadian Federation for the Humanities and Social Sciences
• Canadian Foundation for Dental Hygiene Research and Education
• Canadian Foundation for Dietetic Research
• Canadian Foundation for Research on Incontinence
• Canadian Physiotherapy Association
• Canadian Society of Endocrinology and Metabolism
• Canadian Society for Molecular Biosciences
• Carleton University
• Centre for Addiction and Mental Health
• Child and Family Research Institute
• Children's Hospital of Eastern Ontario (CHEO) Research Institute
• CHUQ Research Centre
• Columbia University
• Concordia University
• Cornell University
• Council of Academic Hospitals of Ontario (CAHO)
• Crohn's and Colitis Foundation of Canada
• Dalhousie University
• Dystonia Medical Research Foundation Canada
• Fragile X Research Foundation of Canada
• Hospital for Sick Children (SickKids Hospital)
• Indigenous Peoples' Health Research Centre
• lnstitut de recherches cliniques de Montréal
• Institute for Clinical Evaluative Sciences
• Jewish General Hospital
• Lawson Health Research Institute
• London Health Sciences Centre
• McGill University
• McMaster University
• Memorial University of Newfoundland
• Michael Smith Genome Sciences Centre
• MitoCanada
• Mount Allison University
• Mount Sinai Hospital - The Samuel Lunenfeld Research Institute
• New Brunswick Health Research Foundation
• Northeastern University: Centre for Drug Discovery
• Nova Scotia Health Research Foundation
• Ontario Cancer Institute
• Ottawa Hospital Research Institute
• Pediatric Emergency Research Canada
• Queen's University
• Ryerson University
• Simon Fraser University
• St. Francis Xavier University
• St. Joseph's Health Care London
• St. Mary's University
• Sunnybrook Health Sciences Centre
• Toronto Rehabilitation Institute
• Toronto Western Research Institute
• Université de Laval
• Université de Moncton
• Université de Montréal
• Université de Sherbrooke
• Université du Québec à Trois-Rivières
• University of Alberta
• University of British Columbia
• University of Calgary
• University of Guelph
• University of Manitoba
• University of New Brunswick
• University of Ottawa
• University of Prince Edward Island
• University of Saskatchewan
• University of Toronto
• University of Waterloo
• Western University
• Wilfrid Laurier University
• Wilson Centre for Research in Health Professions Education
• York University

Date modified:

2012-08-14