Midterm Evaluation of the Pandemic Preparedness Strategic Research Initiative
The aim of this midterm evaluation was to assess the overall design and implementation of the Pandemic Preparedness Strategic Research Initiative, in order to inform future development and identify potential areas for improvement. At the halfway point of this five-year initiative, and within the context of its overall goal of improving Canada's ability to prevent and/or respond to an influenza pandemic, a number of early conclusions can be drawn. These are presented in terms of the extent to which the PPSRI is producing its desired outputs and early outcomes, as captured by responses to the evaluation issues and questions.
4.1 How effective was the PPSRI's priority setting?
Findings: The evaluation findings suggest that in most areas, an effective organizational model was developed and implemented, facilitating the identification and implementation of strategic pandemic research priorities. The mechanisms for identifying research priorities were inclusive and effective, and have resulted in a sound set of priorities that are widely endorsed. Other priorities were nonetheless identified, along with concern about spreading limited resources too thinly across theme areas that are too broadly defined.
The PPSRI's identified priorities have been faithfully translated into funding opportunities that capitalize on existing programs and structures within and outside of CIHR. This has resulted in the introduction of timely and attractive funding opportunities to the research community.
The processes for identifying priorities have not created widespread perceptions of conflict of interest, but such perceptions exist in a small proportion of the potential pandemic research community.
Suggestions for improvement: It may be opportune to consider reassessment of the pandemic and influenza strategic research priorities taking into consideration the strengths of what has been funded so far, remaining gaps and emerging issues1.
4.2 How effectively has the PPSRI built national and international partnerships?
Findings: The PPSRI has identified and created effective linkages with the Public Health Agency of Canada, the Canadian Food Inspection Agency, and the Health Research Foundation. These have led to partnered funding opportunities that have satisfied the interests of all partners and have leveraged a significant amount of additional funds, more than doubling the amounts available through CIHR for pandemic preparedness research.
Linkages with international research efforts in pandemic preparedness have not been as successful to date, not because of lack of effort or will. At the institutional level, the evaluation results reflect the need to take into consideration during program design and implementation the considerable complexity and time lags associated with international partnerships. At the level of funded research, less than 30% of researchers credit the PPSRI with strengthening their international collaboration. Partnerships at the level of researchers have tended to concentrate in countries with existing research capacity. Students involved in about one-third of the grants have been integrated into international collaborations.
Suggestions for improvement: The findings suggest that the PPSRI will need to continue efforts to establish linkages with international agencies and strengthen the partnership with IDRC. The group of PPSRI researchers who are involved in international collaborations could be solicited to help identify pathways to increased international research linkages throughout the PPSRI.
4.3 How appropriate was the PPSRI's design?
Findings: The PPSRI's program design has created a platform for the achievement of expected short term results, although some of these remain to be fully achieved. A first result is a national pandemic preparedness research agenda that is much more coordinated than it would have been in the absence of the PPSRI, as evidenced by the findings that: a) partners had not worked together before in this area; and b) III worked effectively towards buy-in and adaptation to ensure ongoing partner engagement. This national agenda covers all four of CIHR's theme areas (biomedical, clinical, health systems and services, and population and public health) in its design (but not necessarily its implementation). It has likely reduced duplication among Canadian funding initiatives available to Canadian pandemic researchers and has combined resources from the multiple sources effectively.
Suggestions for improvement: Reasons for lower then expected uptake of some program components could be explored, as they may reflect design issues. Otherwise, the program design to date has largely been successful and there is no evidence to suggest it should not be pursued along the same lines.
4.4 Have the desired funding opportunity outputs been produced?
Findings: The PPSRI has successfully launched about 30 targeted funding opportunities, reflecting strategic research priorities. These were launched in a timely manner despite significant time pressures. The challenges in mounting a large, multi-faceted targeted initiative within a five-year funding window were considerable, and while most of these challenges were successfully faced, one important consequence was reduced duration of some of the grants offered, compared to the original intent.
In most cases, the funding opportunities reached and attracted applications from the relevant research community. There is, however, some lack of awareness of the program and some evidence that communications about it have not been maximally effective.
Researchers' responses to the suite of funding opportunities offered through the PPSRI have resulted in a large body of funded research. The relatively high funding rate for applications of meritorious scientific quality may confirm the Task Group's identification of the need for capacity building. As well, the lack of applications in some areas that were identified as key knowledge gaps led in turn to a lack of targeted research. This indicates that alternate strategies may need to be developed to achieve the advances in knowledge that were judged to be necessary for an effective pandemic response. This is especially true in the area of public health measures.
The evaluation results confirm that research capacity development is occurring, through several avenues. It can be estimated that the PPSRI is currently engaging between 150 and 200 trainees in pandemic preparedness research, or about 2.5 trainees per grant. The PPSRI is also contributing to the integration of new collaborators and reorientation of research foci.
Suggestions for improvement: More aggressive capacity building and targeted promotion of pandemic public health research, including long term training approaches and/or application development support, may be needed in specific areas. The intended embedding of training within all PPSRI components could perhaps be somewhat intensified, as about 15% of grants involve no trainees. Communications about the program could also be examined with a view to increasing awareness and drawing researchers' attention to funding opportunities and their specificities.
4.5 How successful are the PPSRI's networking and KT activities?
Findings: The PPSRI's networking activities such as application development workshops, and in particular, the first annual meeting of researchers and end-users, have been very successful. These are seen as an innovation with immediate and potential benefits to researchers, trainees and research users.
It is too soon to tell whether the PPSRI – as a targeted strategic initiative where, in principle, all funded research would contribute directly or indirectly, immediately or eventually, to pandemic preparedness – is maximally facilitating knowledge translation towards the AI/PI Strategy goals of reducing illness deaths and societal disruption from influenza pandemic. However, a significant proportion of funded research projects, about three-quarters, have knowledge translation plans in place, and end-users are currently involved in about two-thirds (66%) of funded projects. This is seen by III as clearly an improvement over existing, non-targeted programs, a result of it having been a program requirement in some of the PPSRI funding opportunities and strongly encouraged in others. Events such as the first annual meeting of researchers and end-users are also contributing to knowledge translation potential. There has nonetheless been very limited uptake of the funding tools made available through the PPSRI to facilitate knowledge translation.
Suggestions for improvement: If possible, the PPSRI networking activities should be continued and expanded. 2 Some needs assessment may be helpful to better understand the barriers and facilitators3 to engaging in effective KT and end-user integration, especially among vaccine and virus researchers. As well, the development and implementation of KT skills development opportunities such as workshops4, for both researchers and end-users, may be considered. More direct promotion and marketing of KT and KT capacity development opportunities to funded researchers could help ensure that uptake of the PPSRI's KT funding opportunities will strengthen. These efforts should prioritize those researchers whose funding will end in 2009. Some financial incentive for KT integration may also be considered, for example by stipulating that a fixed proportion of grant funds be used for KT.
4.6 Overall conclusion
Overall, the design, delivery and initial outputs of the PPSRI are ensuring that the overall goals for the PPSRI, of improving Canada's pandemic preparedness and of increasing pandemic preparedness research capacity, can be achieved.
Areas identified for additional support or perhaps alternative strategies to ensure that they can maximally contribute to the achievement of objectives include: development of a coordinated international research agenda; stimulating pandemic preparedness research with a public health focus; and, facilitating broader engagement of the influenza research community with more elements of the PPSRI's KT strategy. Capacity development through the engagement of trainees and other strategies could also be reinforced, as some evidence suggests that higher application pressure would be desirable. Communications about the PPSRI could also ensure that the research community is aware of the initiative and all its specificities.
The evaluation findings show that the PPSRI has been especially successful in developing solid and productive partnerships with national agencies, developing consensus on research priorities and implementing tools to address those priorities. It has also been shown to be successful in creating platforms to foster networking and eventual collaboration that engage researchers, trainees and potential research users. Further, it has resulted in significant resource leveraging for pandemic preparedness research and the funding of excellent research. These strengths provide a solid foundation for continued success as the PPSRI moves forward.
[ next section ]
- The PPSRI priority setting process included all of the process elements identified as essential for successful healthcare priority setting by Sibbald et al (2009): (stakeholder engagement; use of explicit process; information management; and consideration of values and context) except revision or appeal mechanisms.
- Partnerships between end-users and researchers are key to ensuring research impacts on policy and practice. These are most effective when they involve interpersonal strategies other than only written information. Kalucy et al, 2009. Exploring the impact of primary health care research; Jacobson N et al. 2003. “Development of a framework for knowledge translation: understanding user context.” Journal of Health Services Research & Policy; 8(2): 94-99.
- This should include organizational barriers as well as professional and personal ones: Jacobsen, Butterill & Goering, 2004.
- Existing frameworks and tools may be adapted to the pandemic preparedness context: e.g. A Guide for Developing Health Research Knowledge Translation Plans; Knowledge Transfer Research Template Plan
- Date modified: