Expert Review Team Report for Institute of Health Services and Policy Research

Table of contents

• Summary
• Section 1 – Institute mandate
• Section 2 – Status of this area of research in Canada
• Section 3 – Transformative Impacts of the Institute
• Section 4 – Outcomes
• Section 5 – Achieving the Institute mandate
• Section 6 – ERT observations and recommendations
• Appendix 1 - Expert Review Team
• Appendix 2 - Key Informants

Summary

The Institute of Health Services and Policy Research (IHSPR) has an extensive mandate - it is responsible both for developing health service and policy research within its Institute (institute role) and for stimulating research respecting health systems and services across CIHR as a whole (theme role). It must have a strong focus on translation and ensuring research is relevant and accessible to policy agencies.

There were considerable challenges facing the IHSPR at its establishment including limited capacity in health service and policy research (albeit with existing pockets of excellence), no prior history of funding through Medical Research Council and few established relationships between CIHR and policy agencies. In addition, the Institute was faced with the need to develop relationships with all of the provincial governments responsible for the delivery of health care.

The Institute has taken a strategic approach to achieving its mandate – for the first few years it focused on building capacity broadly in the health service and policy research community, increasing the community's capacity to conduct excellent research and forging partnerships to create more opportunities for researchers. More recently, it has developed knowledge translation (KT) strategies and refined its focus to a smaller number of priority areas. There appeared to be considerable support for the priorities among the researchers and stakeholders, particularly for the work in primary care.

Given the initial challenges, it will inevitably take some time before the impact of the Institute's work can be adequately assessed. Evaluation of the impact of the IHSPR will also require collection of more sophisticated information about the quality of research and its influence on policy and practice than was available to the ERT (see section 6).

However, the IHSPR has put in place a platform that seems likely to result in important changes in the future. It has:

• Invested in capacity development resulting in a 7 fold increase in the number of health service and policy researchers
• Developed research infrastructure of value to many researchers including investments in improving the use of secondary data sources
• Developed a sense of community among health service and policy researchers
• Established innovative approaches to funding vehicles ( particularly the team grants) and to KT (through the Evidence on Tap initiatives)
• Leveraged additional funds for health services and policy research estimated at $2 leveraged funds for every CIHR dollar invested. In total grant expenditures related to health services and policy research have increased from 5.5 M in 2000/01 to $97.5M in 2009/10.
• Had some impact on CIHR overall including developing several partnerships with other institutes and the adoption of IHSPR innovations as CIHR wide initiatives

The ERT team was impressed by the value placed by researchers and stakeholders on the Scientific Director and staff of the Institute.

A key measure of the impact of the Institute's capacity building is the proportion of open grants allocated to health service and policy research. This rose from 1% in 2000/01 to 6% in 2009/10 and is a significant achievement.

However, there are opportunities to strengthen the work of the Institute going forward. The ERT recommends that:

1. The IHSPR with the support of CIHR as a whole place more emphasis on monitoring the quality and quantity of health service and policy research, the impact of this research on policy and practice and the evaluation of its own initiatives and programs.
2. The IHSPR continue to refine its strategy to ensure the most effective portfolio of programs. This should include consideration of the number of current initiatives and the most appropriate balance of research focused and KT activities.
3. CIHR work with the IHSPR to develop additional support for the Institute's theme role, including additional funding.
4. As IHSPR moves beyond broad capacity building, there is a greater focus on the quality and policy and practice relevance of the research outputs. Particular attention should be paid to areas where there is an opportunity for both international leadership and relevance to Canadian policy and practice including:
   1. Further development of the work to make secondary data more available to researchers. Canada has excellent resources and an existing skill base; these data are of fundamental importance to informing health services and policy
   2. The development of approaches to integrate rigorous research into the evaluation of government policies and programs. Canada has a unique opportunity to exploit the natural experiments arising from variations across the provinces and an existing skill base in implementation research, including economics.

1. Institute mandate

The mandate of the IHSPR is to:

'champion and support excellent health services and policy research and knowledge translation to identify, understand and address health system needs and challenges and to contribute to health system accessibility, responsiveness, effectiveness, efficiency and sustainability.'

We note that this is a very ambitious mandate, and that one would not expect immediate or short-term impacts on health system accessibility, responsiveness, effectiveness, efficiency and sustainability in Canada as a result of the activities of this Institute.

2. Status of this area of research in Canada

Health services and policy research is an emerging field internationally and lacks the status of more established research fields. The mandate of the CIHR has a special focus on health services and policy research, as it is required to ensure translation of knowledge from research into 'more effective health services and products and a strengthened health care system'. Accordingly the CIHR Act defines Research respecting health systems and health services as one of its four themes.

Significant progress has been made over the past ten years in building the status of health services and policy research in Canada, although more work remains to be done. There are several indicators of the status of research in this area.

In terms of international standing of Canadian health service and policy research, the ERT noted that Canada is internationally regarded as having research strengths in health services and policy research including in systematic reviews (through the Cochrane Collaboration), the use of secondary data sources, knowledge transfer and implementation science.

Although there has been considerable investment in building capacity in health services and policy research, it was difficult to judge the extent to which this had resulted in an increase in scientific excellence and international standing. The bibliometric data provided by CIHR included analysis of only one area, access to appropriate care, and was unable to distinguish whether papers were the outcome of CIHR funding; this limited analysis provided some evidence of international standing, indicating that Canada was ranked fourth in the world based on average of relative citations and that some of the researchers working in this area were evidently highly productive and had excellent citation rates. However, the overall standing of Canadian health services and policy research could not be judged from the bibliographic information (see also section 6).

In terms of the status of the research among policy makers and practitioners, the IHSPR report provided several examples of the impact of particular pieces of research on policy including those focused on rising cancer drug costs and geographical variation in access to coverage, the quality of care provided in long term care facilities, improving patient safety in Canada and the effectiveness of public report cards for improving the quality of cardiac care. The policy stakeholders perceived that health services and policy research had the potential to be very useful in their work and identified specific pieces of research that had been useful. However, they also felt that more research of direct relevance to policy makers should be supported.

In terms of the status of health services and policy research within the CIHR, growing value is being placed on this research although there is room for further improvement. In recent years, a greater number of funded researchers are identifying an affiliation with IHSPR and a greater proportion of competitive funds are going to this area. However, this theme continues to receive the smallest amount of funding of all four themes. There has been relatively limited success in partnering with other institutes, particularly those outside the cross cutting themes, although some interesting partnerships are beginning to emerge such as that with the Institute of Genetics to support work on genetics and health services and policy research.

3. Transformative Impacts of the Institute

The IHSPR has a broad mandate and is beginning from a relatively low base in terms of both capacity and partnerships; it will therefore take some time before the transformative impact of the Institute can be adequately assessed. However, it is evident that the IHSPR has put in place a platform that seems likely to result in transformative impacts in the future including:

• Capacity development - people: IHSPR identified the need to build capacity across the field generally as a first priority. Accordingly, it invested a considerable proportion of its funds into capacity development and through partnerships leveraged additional monies. There is evidence of a considerable increase in the number of researchers in the field – over the past 10 years, the number of researchers reporting a primary affiliation with health service and policy research has increased from 200 to 1600, an 8 fold increase. Although these data are not without problems, it seems clear that there has been a significant increase in health services and policy research capacity. IHSPR has now begun to focus its capacity development in specific areas (e.g. primary care) to build depth and this might be expected to build new areas of strength in the future.
• Capacity development – infrastructure: IHSPR also identified the development of research infrastructure as a priority. This approach has the potential to improve the strength and quality of health services and policy research. Two initiatives have the potential to be particularly valuable: (i) the establishment of the Drug Safety and Effectiveness Network which provides infrastructure for drug safety and effectiveness research and (ii) efforts to improve access to secondary data for research. Canada has excellent secondary data and existing research strengths; the Institute has correctly identified this as an area where international leadership could be provided. Other investments include the support of the Cochrane Collaboration and a funding stream for methods and conceptual development.
• Built a community of health service and policy researchers: There was evidence that the Institute had been instrumental in building a stronger sense of community amongst its research constituency. It had established opportunities for collaboration across provinces, contributed to the establishment of a journal, raised the profile of health service and policy researchers and established a national conference. The ERT noted that there were between 800 and 900 participants at recent conferences.
• Increased funds for health services and policy research: The IHSPR has clearly been effective in leveraging additional monies; it estimates that for every CIHR dollar expended, an additional $2 is raised from external sources, tripling the funds available. For example, CIHR contributed funds of $5.6 million through Partnerships for Health System Improvement (PHSI) program which resulted in an additional $30 million investment from partners. Similarly, the investment by IHSPR of $3 million in primary health care research resulted in a flagship strategic initiative across CIHR which with other partners will result in investment of $50million over the coming years.
• Developed innovative approaches: The ERT was impressed by the innovative approaches developed by the IHSPR in both research funding tools and in KT. The establishment of team approaches such as the Interdisciplinary Capacity Enhancement Team and New Emerging Team grants have the potential to develop a more national approach and to facilitate between province comparisons. The redesign of the PHSI scheme led by the Institute has clearly been effective in building policy research partnerships and has now been adopted across the CIHR. The Evidence on Tap and Best Brains programs are innovative developments by the Institute that have now also been taken on board by CIHR more generally.
• Built partnerships with policy agencies: IHSPR has developed new partnerships and engagement with policy agencies including those at the provincial level. This has leveraged additional funding but also the opportunity to increase the impact of health services and policy research. It was evident from the consultations that the policy stakeholders perceived value in the work of the Institute and in some cases regarded the work as changing the landscape. There was a high level of engagement with the Institute with some developed views about what could be done better going forward (see section 6).
• Impact on the CIHR more broadly: The impact on the CIHR overall is still emerging. IHSPR has developed some innovations that have been taken up more broadly and the effective advocacy for a flagship program in primary care is important. There is however still more to be done in increasing health service and policy research across the institutes.

4. Outcomes

Strategies: IHSPR has been very active in implementing a large number of varied programs designed to build capacity and increase the impact of research. Capacity building strategies include establishment of team grants, researcher support (through the Capacity for Applied Developmental Research and Evaluation (CADRE) and Applied Chairs programs) and Strategic Training Initiatives in Health Research (STIHR). Other programs seek to develop research infrastructure or stimulate research in areas of priority – examples include the Drug Safety and Effectiveness Network and the new primary care initiative. The KT programs under the umbrella of Evidence on Tap include Best Brains, Café Scientifiques and Expedited Knowledge Synthesis.

Outputs: There is also some evidence of outputs from this work. In terms of capacity building, over a ten year period there was an 8 fold increase in the number of researchers identifying a primary affiliation with IHSPR. The CADRE program alone supported 83 post docs, 12 mid-career chairs, five regional training centres and 13 career reorientation awards and in total supported more than 1250 researchers. The 33 STIHRs over the past ten years have attracted 2,400 trainees. The high numbers participating in the conferences indicates the growing development of a community of health service and policy researchers. It would be helpful in the future to collect data on next destinations and ultimate job profiles of these recipients of training funds, both to assess the value of these training programs and also their contribution to policy, practice, and academia.

There is evidence that the KT activities are valued; over 300 policy makers have participated in the ten Best Brains Exchanges.

Outcomes: The ERT found it difficult to judge the outcomes from the work of the IHSPR. It is still early in the development of the Institute and the data provided were insufficient for this purpose. However:

1. The increase in the share of funds from the CIHR open competition from 1 -6% indicate that the work of the IHSPR is contributing to the development of significant capacity in health services and policy research and
2. A number of examples of research impacting on policy and practice were cited in the IHSPR report and during the interviews.

5. Achieving the Institute mandate

The Institute's mandate is extensive and encompasses both (i) championing and supporting excellent health services and policy research and (ii) knowledge translation to identify, understand and address health system needs and challenges and to contribute to health system accessibility, responsiveness, effectiveness, efficiency and sustainability. The Institute has both institute and theme responsibilities.

It is too early to fully evaluate the extent to which the Institute is delivering on its mandate. The Institute has:

• Made good progress in developing the platforms to enable excellent health services and policy research in the future through increased workforce capacity, improved infrastructure, the development of a sense of community and improved links to policy agencies as described in section 3.
• Has established KT strategies although it is too early to tell whether these have impacted on the health system
• Has made some headway in increasing Research respecting health systems and health services (theme 3) across CIHR but will need more support to increase engagement

The ERT also noted that the Institute has continued to modify its approach to delivering on its mandate from broad programs in the first five years to an increasingly targeted approach focused on a smaller number of selected priority areas – access to care, pharmaceutical policy, community based primary care, and use of secondary data. This is an appropriate approach and likely to make best use of the limited funds available.

An adequate assessment of the extent to which the broad and complex mandate of the IHSPR is being achieved will likely require purpose built evaluation strategies. The Institute has recognised this need and should be encouraged to develop stringent approaches including independent assessment and rigorous methodologies. Over time, these should include assessment of the impact on health systems and health (see also section 6).

6. ERT observations and recommendations

Additional observations

In addition to the comments above, the ERT noted that:

Measuring impact

• For future reviews, it would be helpful if CIHR provided more numeric information about publications and grants and in a more digested form. For example, it would be useful to understand citations and publications across the field as a whole, changes over time and the relative impact of research funded by CIHR. Likewise, the information about grants could have been usefully grouped by types of funding over time and to reflect both the CIHR contribution and leveraged funds. We are unable to assess value for money since we did not have high-level information on the amount investment in the various schemes. The selection of bibliographic data also warrants attention – the data presented included papers published before the establishment of CIHR, do not distinguish between CIHR funded and other funded research, are only for a selected topic area, and some of which we would not have classified within that subject area ( for example 'Effect of rosiglitazone on the frequency of diabetes in patients with impaired glucose tolerance or impaired fasting glucose: a randomised controlled trial' does not appear to the ERT to be a good example of health services and policy research on access to appropriate care).
• The IHSPR is to be commended for its work in building partnerships and leveraging additional funds for health services and policy research. The Institute has envisaged its role as a catalyst, using the relatively small amounts of funding strategically. However, this also made it difficult to evaluate the extent to which improvements in health services and policy research were attributable to the Institute and to assess what would not have happened in its absence. While it was clear that both researchers and policy makers felt that the Institute's role had often been critical, it would be valuable for the IHSPR to consider how to measure its impact in more detail going forward, especially given that one would anticipate a time lag between publication of research, knowledge translation, and uptake of any new policies or practices.
• The IHSPR is also to be commended for recognising the importance of evaluation of its initiatives. However, it should be encouraged to develop stringent approaches to evaluating its impact including independent assessment and rigorous methodologies. The impact of the IHSPR's work on health (as opposed to policy and practice) will be particularly challenging to assess and there is considerable international discussion currently about how best to measure the impact of research on policy and practice. The Institute should consider developing measures that go beyond case histories over time. Any evaluations undertaken by the Institute should be made available to any future ERTs.

Theme and Institute
The IHSPR is required both to undertake the work of a traditional institute within CIHR and to lead the development of the Research respecting health systems and health services (theme 3) across CIHR. While a good start has been made in integrating health services and policy research within other institutes, much more could be done. CIHR could consider providing additional strategic funds to IHSPR to assist it in achieving its dual roles. Additional strategies to impact on other institutes should be considered – for example, it might be of value to include someone with health service or policy research skills on the Institute Advisory Boards of the other institutes to provide a prompt and expertise for considering these research opportunities, and to ensure representation of health service or policy research skills on open grant funding committees.

Complex and shifting environment
The ERT noted that the IHSPR is working in a complex environment with many agencies and some overlap in roles and responsibilities. Some of the stakeholders saw that Health Canada should seek to better coordinate agencies in this area. It is apparent that the changed role of the Canadian Health Services Research Foundation and the closure of the social science funding agency will have some implications for IHSPR. Likewise some opportunities to engage more closely with policy agencies may be created through the renegotiation of the Health Accords.

IHSPR appeared aware of the challenges described above, and in our view needs high-level support from the CIHR corporately if it is adequately to meet them.

Focus and balance of activities
The IHSPR has clearly worked very hard to establish strategies and programs to address its broad remit. While a large portfolio of programs may be inevitable at the outset, there is a danger that a small staff (four plus a half time Scientific Director) may be spread too thinly. There is evidence that the Institute is aware of this challenge and has begun to refine its focus.

However, some of the stakeholders expressed a concern that the IHSPR may be too thinly spread and should increase its focus on its core activity of building strong and relevant health services research. These stakeholders felt that the Institute could do more to ensure that the research is as relevant to policy makers as possible; for example, it would be of considerable value to stimulate research that makes the evaluation of government policies and programs more rigorous.

However, other stakeholders felt that the Evidence on Tap programs were very valuable and the Scientific Director reported that these programs have demonstrated the potential value of health services and policy research and resulted in partnerships and additional funding.

Given the limited resources, it may be of value to undertake a strategic review of the IHSPR portfolio to ensure that the best balance is achieved.

Competitive funding and peer review
A key outcome for the IHSPR is to increase the amount of competitive funding from CIHR open grant schemes for health services and policy research. This will be determined by the volume of applications and the numbers of review committees. Careful attention will be required during the upcoming collapse of the number of committees to ensure that the capacity of health service and policy researchers to successfully compete for the open funding is not diminished.

Staff
The ERT was impressed by the very positive comments about the staff of the IHSPR during the interviews. The previous scientific director was clearly respected and trusted and had excellent skills innovation, facilitation and partnership development, and there was considerable goodwill towards the incoming director. The staff was regarded as able and hard working. However we note that the activities and performance of the Institute (as with the other institutes) is heavily dependent on the skills, vision and energies of a single part-time research leader. (We note for example that all the Best Brain sessions had been facilitated by the outgoing scientific director). This dependence on one person may pose a risk to the organisation, both because of the excessive demands that may be put on that one person and because of the changes in strategy and direction that may occur in the handover from one scientific director to another. The excessive demands are particularly relevant to the two institutes which have a dual mandate of responsibility for their Institute and for representing their pillars across the other 12 institutes and the funding committees. A related issue is what the career incentives are for scientific directors, and whether research leaders might be deterred from becoming or remaining scientific directors because of the perceived lack of benefit for their research careers.

Areas for future focus
The areas nominated by the IHSPR as priorities for the future are clearly important. Particular attention should be paid to areas where there is an opportunity for both international leadership and relevance to Canadian policy and practice including:

• Further development of the work to make secondary data more available to researchers. Canada has excellent resources and an existing skill base; these data are of fundamental importance to informing health services and policy
• The development of approaches to integrate rigorous research into the evaluation of government policies and programs. Canada has a unique opportunity to exploit the natural experiments arising from variations across the provinces and an existing skill base in implementation research.

Recommendations

The ERT recommends that:

1. The IHSPR with the support of CIHR as a whole place more emphasis on monitoring the quality and quantity of health service and policy research, the impact of this research on policy and practice and the evaluation of its own initiatives and programs.
2. The IHSPR continue to refine its strategy to ensure the most effective portfolio of programs. This should include consideration of the number of current initiatives and the most appropriate balance of research focused and KT activities.
3. CIHR work with the IHSPR to develop additional support for the Institute's theme, including additional funding.
4. As IHSPR moves beyond broad capacity building, there is a greater focus on the quality and policy and practice relevance of the research outputs. Particular attention should be paid to areas where there is an opportunity for both international leadership and relevance to Canadian policy and practice including:
   1. Further development of the work to make secondary data more available to researchers. Canada has excellent resources and an existing skill base; these data are of fundamental importance to informing health services and policy
   2. The development of approaches to integrate rigorous research into the evaluation of government policies and programs. Canada has a unique opportunity to exploit the natural experiments arising from variations across the provinces and an existing skill base in implementation research, including economics.

Appendix 1 - Expert Review Team

Chair - Professor Sally Redman
CEO Sax Institute
Sydney, NSW Australia

Expert Reviewer – Professor Sally Macintyre
Professor, Division of Community Based Sciences, Faculty of Medicine, University of Glasgow
Honorary Director MRC/CSO Social & Public Health Sciences Unit, UK

International Review Panel – Dr. Chris Murray
Director, Institute for Health Metrics and Evaluation
Professor of Global Health, University of Washington
Seattle WA USA

Appendix 2 - Key Informants

Session 1 – Review of Institute

1. Dr. Colleen Flood, IHSPR Scientific Director (former)
   
2. Dr. Robyn Tamblyn, IHSPR Scientific Director (current)
   Professor, Departments of Medicine, and Epidemiology and Biostatistics
   Faculty of Medicine
   McGill University
   
3. Jean Louis Denis, Chair – Institute Advisory Board
   Director, Institut de recherche en santé publique de l'Université de Montréal
   Professor, Department of Health Administration
   Université de Montréal
   
4. Dr. Anne Sales
   Associate Professor, Faculty of Nursing
   University of Alberta

Session 2 – Consultation with researchers

1. Dr. Pat Martens
   Director, Manitoba Centre for Health Policy
   Professor, Faculty of Medicine
   University of Manitoba
   
2. Dr. Paula Goering
   Director, Health Systems Research and Consulting Unit
   Centre for Addiction and Mental Health
   Professor, Department of Psychiatry
   University of Toronto
   
3. Dr. Bill Hogg
   Director, CT Lamont Primary Health Care Research Center, Élisabeth Bruyère Research Institute
   Professor and Director of Research, Department of Family Medicine
   University of Ottawa

Session 3 – Roundtable with stakeholders

1. Ms. Pauline Rousseau
   Executive Director, Strategic Planning Branch
   Saskatchewan Health
   
2. Ms. Lillian Bayne
   President, Lillian Bayne & Associates
   Former President, Canadian Association for Health Services and Policy Research, Former Assistant Deputy Minister of Health in British Columbia
   
3. Ms. Alison Paprica
   Acting Director, Health System Planning and Research Branch
   Ontario Ministry of Health and Long-Term Care
   
4. Mr. Dave Clements
   Director, Corporate Planning and Accountability
   Canadian Institute for Health Information
   
5. Dr. Ruth Wilson
   Consulting Director, Health Policy, College of Family Physicians of Canada
   Professor, Department of Family Medicine
   Queen's University

Date modified:

2011-08-02