Evaluation of the CIHR Clinical Trials Network in HIV/AIDS Program

Acknowledgements

This report was authored by Heather Mustoe.

The evaluation was carried out with significant contribution from the Evaluation Working Group: Aslam Anis, David Cox, Suzete Dos Santos, Jennifer Gunning, Danielle Halloran, Natalie Henrich, Michelle Jones, Shari Margolese, Robert McLean, Heather Mustoe, Jim Pankovich, Joe Pater, Kevin Pendergraft, Jacquie Sas, Martin Schechter, Joel Singer, and Joanne Tucker. The evaluation design was completed by David Peckham and Heather Maysenhoelder.

Thanks to CHARAC members and the Subcommittee on Performance Measurement for their valuable comments on previous drafts and to other stakeholders for their participation.

Canadian Institutes of Health Research

160 Elgin Street, 9th Floor Address Locator 4809A Ottawa, Ontario K1A 0W9 Canada

Cover photo courtesy of the CIHR Canadian HIV Trials Network.

Table of Contents

Executive Summary

Background

The CIHR Canadian HIV Trials Network (CTN) was established in 1990 as a key component of the Federal Initiative to Address HIV/AIDS in Canada (FI). The CTN is a national network of clinical investigators, physicians, nurses, people living with HIV/AIDS, pharmaceutical manufacturers and others who facilitate HIV clinical trials of the highest scientific and ethical standards. Initially, funding for the CTN was managed by Health Canada, but was transferred to CIHR for administration in 2001. The CIHR Clinical Trials Network in HIV/AIDS Program (Program) is the mechanism that CIHR utilizes to administer the funding and outline the requirements for a national HIV/AIDS clinical trials network. The Network provides an infrastructure and coordinates services intended to assist HIV/AIDS clinical trials. Since 2001 and the transfer of the funding to CIHR, the CTN has been awarded two renewal grants, each with a five-year funding term. The current CTN grant expires in March 2014. The Institute of Infection and Immunity, through the CIHR HIV/AIDS Research Initiative plans to develop a funding opportunity (FO) for spring 2013 for the next funding period.

CIHR’s goals in funding the Program have been to position Canada as a global leader in HIV/AIDS clinical research; and to have in Canada a network and strong research infrastructure that enables Canadian researchers to collaboratively and efficiently conduct excellent, ethically sound HIV/AIDS clinical trials across Canada and internationally.

Evaluation Purpose

This evaluation of the Program has two objectives. First, it will summarize the current funding term (2008-2012), informing CIHR on the performance, delivery and relevance of the program. The evaluation will offer recommendations for program improvement that should be considered in the program’s subsequent FO, expected to be completed in the spring of 2013. Secondly, this evaluation will also help inform the evaluation of the FI through which the Program receives its funding, currently ongoing with an anticipated completion of 2014. Issues relating to the appropriateness or level of funding received by the program are outside the scope of this evaluation as this is determined as part of the wider CIHR HIV/AIDS Research Initiative.

Key Findings

The CTN is generally viewed as important in supporting the development and conduct of innovative and important research.

When considering the relevance and continued need for the Program we find that:

The following evaluation findings can inform the development of the FO:

Conclusions

The CIHR Clinical Trials Network in HIV/AIDS Program has made significant contributions to supporting a research infrastructure within Canada, building research capacity and creating knowledge in the field of HIV/AIDs clinical trials. The program has been effectively designed and delivered although some areas for improvement are noted and should be considered when drafting the upcoming funding opportunity.

Recommendations

The following recommendations are made to further enhance the success of the Program.

  1. As a part of its renewed funding opportunity, CIHR should review the future strategic focus of a clinical trials network in HIV and look for areas of alignment within other CIHR programs.
  2. CIHR Clinical Trials Network in HIV/AIDS Program should encourage the CTN to address the identified concerns surrounding governance and leadership in a transparent and inclusive manner.
  3. The subsequent funding term for the CIHR Clinical Trials Network in HIV/AIDS Program should continue to enhance roles for community in the CTN.
  4. The CIHR Clinical Trials Network in HIV/AIDS Program should reconsider the purpose of engagement with relevant partners and stakeholders.
  5. CIHR should encourage the CTN to pursue partnerships leading to additional leveraged funds and maximum impact of the network.
  6. To enhance efficiency and effectiveness, CIHR should streamline the reporting structure and enhance the mechanism for oversight.

Management Response

Recommendation Response (Agree or Disagree) Management Action Plan Responsibility Timeline
1. As a part of its renewed funding opportunity, CIHR should review the future strategic focus of a clinical trials network in HIV and look for areas of alignment within other CIHR programs. Agree The CIHR HIV/AIDS Research Initiative will consult with its advisory boards, partners in the Federal Initiative, international experts and CIHR senior management regarding the strategic scope and focus of the next iteration of the Program. The CTN will be required to define its strategic focus for the next term, within the scope outlined in the funding opportunity. Scientific Director, CIHR Institute of Infection and Immunity HIV Initiative consultation: January – March 2013

Definition of focus by the CTN: October 2013
2. CIHR Clinical Trials Network in HIV/AIDS Program should encourage the CTN to address the identified concerns surrounding governance and leadership in a transparent and inclusive manner. Agree The CTN will be required to explicitly describe the process followed to determine CTN leadership for the next phase of the network and provide rationale for the proposed governance structure. Scientific Director, CIHR Institute of Infection and Immunity Completion of the Funding Opportunity - June 2013
3. The subsequent funding term for the CIHR Clinical Trials Network in HIV/AIDS Program should continue to enhance roles for community in the CTN. Agree The CIHR HIV/AIDS Research Initiative will require an ongoing and valuable role for community in the CTN in the 2013 funding opportunity for the Program. Scientific Director, CIHR Institute of Infection and Immunity Completion of the Funding Opportunity - June 2013
4. The CIHR Clinical Trials Network in HIV/AIDS Program should reconsider the purpose of such engagement with relevant partners and stakeholders. Agree During the development of the 2013 funding opportunity, the CIHR HIV/AIDS Research Initiative will consult regarding value and established best practices in partnerships for research networks. The 2013 funding opportunity will clarify purpose and expectations for CTN partnerships for the next funding cycle. Scientific Director, CIHR Institute of Infection and Immunity Completion of the Funding Opportunity - June 2013
5. CIHR should encourage the CTN to pursue partnerships leading to additional leveraged funds and maximum impact of the network Agree The CTN will be encouraged to further leverage the core network funding provided by the Program in order to further enhance the impact of the network. The 2013 funding opportunity will outline expectations for leveraging, which will be monitored through future performance reporting. Scientific Director, CIHR Institute of Infection and Immunity Completion of the Funding Opportunity in June 2013; and as per reporting requirements.
6. To enhance efficiency and effectiveness, CIHR should streamline the reporting structure and enhance the mechanism for oversight. Agree CIHR will utilize best practices for efficient and effective reporting, oversight and engagement of the CTN in ongoing strategic direction setting of the HIV Initiative the next iteration of the program. Requirements for reporting and oversight will be outlined in the funding agreement Associate Vice President, Research and Knowledge Translation, CIHR Completion of the Funding Opportunity - June 2013

Background

CIHR Clinical Trials Network in HIV/AIDS Program

The CIHR Clinical Trials Network in HIV/AIDS Program (Program) is the mechanism that CIHR utilizes to administer the funding and outline the requirements for the CIHR Canadian HIV Trials Network (CTN or Network). Specifically the CIHR Canadian HIV Trials Network is committed to:

CIHR’s objectives in funding the CIHR Clinical Trials Network in HIV/AIDS Program are:

Overall the CIHR Canadian CTN on HIV/AIDS is expected to further strengthen clinical research in HIV/AIDS by providing funding for the enhancement, operation and maintenance of infrastructure for a national HIV clinical trials network. This targeted investment is intended to create a collaborative and productive environment that conducts high-quality, ethically sound HIV/AIDS clinical trials in Canada and internationally

Evaluation Purpose

The current CTN grant expires in March 2013. The Institute of Infection and Immunity, through the CIHR HIV/AIDS Research Initiative, is in the process of establishing an expert advisory group to guide the development of the spring 2013 directed grant funding opportunity (FO). The purpose of this evaluation is to provide CIHR with the information needed for enhancing the CIHR Clinical Trials Network in HIV/AIDS Program for its next funding term. It is expected that the results of this evaluation will also help inform the ongoing evaluation of the Federal Initiative to Address HIV/AIDS in Canada (FI), through which the CTN is funded. This evaluation will not assess the level or appropriateness of the funding received by the CTN. These questions will be addressed in the evaluation of the FI.

This evaluation used five lines of evidence to corroborate findings: bibliometric analysis, survey and interview data from investigators, postdoctoral students and community members, a network scan and document review.

Key Findings

Global Leadership

A major objective of the CIHR Clinical Trials Network in HIV/AIDS Program is to position Canada as a global leader in HIV/AIDS clinical research. While “global leadership” connotes measurement along several dimensions, the key CTN stakeholder groups involved in this evaluation identified six: academic leadership, knowledge translation leadership, international visibility, international collaboration, partner engagement and service provision. The first three dimensions will be addressed in the following sections; collaboration and partner engagement will be described in under Evaluation Question 3 and service provision will be discussed in detail under Evaluation Question 4.

Academic Leadership

Bibliometric analysisFootnote 1, conducted by the Observatoire de Sciences et des Technologies (OST), contextualizes the academic production of the CTN within Canada and the international community. This analysis, presented on the following page in Table 1, shows that Canada was the 5th largest producer of publications in the field of HIV/AIDS from 2008-2011, moving up in rank from 6th for the period of 2001-2007. Filtering further, it is observed that Canada was the 6th largest producer of clinical trial papers in the field of HIV/AIDS worldwide from 2008-2011 as shown in Table 2.

Overall, the number of papers attributed to the CTN within the field of HIV/AIDS or specifically relating to HIV/AIDS clinical trials accounts for 14% and 30% respectively of Canada’s overall publications in these areas. Given that the CTN has had 22 years of continuous funding, these proportions may seem small. However, a large amount of HIV/AIDS related research is supported by the pharmaceutical industry, and these studies are not included in CTN publications. Similarly, other HIV/AIDS related research supported by non-government sources often does not involve CTN members. In addition, individual Canadian investigators can participate in international trials, can conduct single-site clinical research at their home site, and can participate in industry studies that are non-CTN trials. Publications arising from these activities would not appear on the CTN list.

Table 1: Annual number of papers in the field of HIV/AIDS, the top 10 leading countries and the world (2008-2011)
Rank Country 2008-2011
CTN 165
1 United States 11,881
2 United Kingdom 2,365
3 France 1,662
4 South Africa 1,495
5 Canada 1,181
6 Italy 1,175
7 Spain 1,168
8 Brazil 914
9 Germany 875
10 Australia 795
Table 2: Annual number of clinical trial papers in the field of HIV/AIDS, the top 10 leading countries the world (2008-2011)
Rank Country 2008-2011
CTN 55
1 United States 1,730
2 United Kingdom 471
3 France 310
4 South Africa 258
5 Spain 249
6 Canada 183
7 Italy 173
8 Switzerland 129
9 Netherlands 128
10 Germany 126

Source: Data compiled by the Obeservatoire de sciences et des technologies (OST) from Web of Science (Thomson Reuters - BDBC) database, September 2012.

Another metric to assess the productivity of the CTN is to calculate the ratio of total number of trials supported over the funding terms to the number of publications attributed to the CTN over the same time period. It is recognized that this is an imperfect measure in that trials currently supported will still yield publications but does provide an indication as to the number of publications per trial. The Progress Reports present figures on the number of trials supported by the CTN on a fiscal year term, as presented in Table 3, where “supported” includes trials at all stages: pending, enrolling, ongoing, continuing and completing finishing results. The average number of clinical trials supported in a given year is 20. Since the same study may be ongoing for a number of years, removing duplicates, it is observed that the CTN has supported 35 unique trials over that time period. Data from Table 1 shows that 165 publications were produced from 2008-2011, indicating that on average, the CTN has produced about 4.7 publications per clinical trial supported.

Table 3: Trials and publications supported by CTN and participants enrolled, 2008-2012
2008-09 2009-10 2010-11 2011-12 Total # of Trials
# of trials supported by CTNFootnote 2 21 25 15 19 80
# of unique trials from 2008-2009 35

Source: CIHR Canadian HIV Trials Network Progress Reports (2008-2012)

A comparison of productivity was made to the Kirby Institute’s Therapeutic and Vaccine Research Program (TVRP) of Australia, one of the organizations included in the network scan and the most similar organization to the CTN. The TRVP aims to produce “one primary publication for each trial and as many additional publications as possible representing formal sub-studies or ancillary analyses”. A review of their publications shows that the TVRP averaged 3.7 publications per trial from 2008 to 2011. Using the TVRP as a benchmark, the CTN is producing well.

However, the number of papers alone tells nothing of the quality of the publications. Assessing the scientific impact of publications within research is important when considering quality. Two indicators of scientific impact are Average Relative Citations (ARC) and Average Relative Impact Factor (ARIF)Footnote 3:

These results are presented in Figure 1.

Figure 1: Average ARIF and ARC of publications in the field of HIV/AIDS for CTN and top 10 leading countries (2008-2011)

Source: Data compiled by OST from Web of Science (Thomson Reuters - BDBC) database, September 2012.

Figure 1: long description

In efforts to corroborate findings, a survey collected information from investigators (N=21), postdoctoral students (N=17) and past and present Community Advisory Committee (CAC) members (N=10). The majority of respondents from all surveys felt that the CTN contributes to translating the knowledge from research into real world applications, as seen in Figure 2.

Figure 2: Extent to which the CTN contributes to creating new knowledge, all survey groups (N=67)

Figure 2: long description

It can be concluded that while the CTN produced a relatively small number of papers in the field of HIV/AIDS related research compared to national production, they were of the highest quality. Thus the CTN has made a valuable contribution to the field and is considered an academic leader.

Knowledge Translation Leadership

The CTN is expected to facilitate the translation of knowledge into improvements in the health of Canadians living with or at risk of HIV/AIDS, health policy and/or the Canadian health care system. The CTN has facilitated knowledge translation to users in a variety of ways including but not limited to peer review publications in high quality journals, national and international conference presentations and poster sessions, community publications, as well as less traditional methods such as social media via Twitter, Facebook and Youtube.

Beyond these traditional modes of knowledge translation, the CTN continues to create partnerships with agencies that support knowledge transfer, the CIHR Centre for REACH (Research Evidence into Action for Community Health) in HIV/AIDS, and knowledge exchange broker Canadian AIDS Treatment Information Exchange (CATIE). For example, CTN studies are referenced in CATIE's Treatment Updates. These updates, akin to news briefs, are sent out to physicians, treatment information officers, people living with HIV and other interested stakeholders. While not codified treatment guidelines or a peer-reviewed journal, CATIE news is an important component of knowledge transfer and exchange that may not have been captured in the evaluation regarding citations of CTN studies.

CTN clinical leaders frequently engage the community and speak at public events such as World AIDS Day events; and the CTN shares information through clinical trials workshops and knowledge exchange campaigns. For example, print copies of the updated Clinical Trials: What you need to know handbook were disseminated to sites and community organizations across Canada and CATIE continues to act as a supplier of the handbook. Electronic versions in both official languages have been posted as an e-book on the CTN website and are available for PDF download, and these electronic documents have also been shared by various groups over social media, including a blog post on positivelite.com.

The CTN also connects with the community through its knowledge brokering service - writing plain language summaries that capture the broad use of language, information and differences (such as Informed Consent) for trial participants to ensure comprehension and encourage participation. These summaries are provided for all studies and posted online. Informed consents are approved by the Community Advisory Committee to ensure inclusion of the valuable community perspective.

The CTN has become a leader in using social media to translate HIV clinical research. As of June 2012, over 530 organizations and individuals were following the CTN on Twitter. The CTN has posted over 1,900 tweets relevant to HIV and co-infection research and the community (1,700 of those in the past year). The CTN has also twice moderated the weekly Health Care Social Media Canada (#HCSMCA) Twitter chat. In June 2011, the CTN launched a Facebook page and has 119 individuals and 128 organizations “liking” and following the CTN. In September 2011, CTN launched a YouTube channel to post videos; the channel now has over 2,400 views with 26 videos posted (videos include interviews with principal investigators of ongoing and new studies, postdoctoral fellows, members of partner organizations and community members). The CTN is also offering guidance to studies and trials that are engaging in social media, which is an innovative growth area for the CTN and its studies.

The CTN has significant involvement nationally on the development and implementation of standard operating procedures, training & education, and other clinical research standardization efforts as part of their involvement with the Network of NetworksFootnote 5. In addition to normalizing clinical trials, four recent examples of the CTNs efforts to translate knowledge into practice and/or better treatment for the HIV/AIDS community come from:

  1. Dr. Mona Loutfy and Ms. Shari Margolese of the CTN Community Advisory Committee who together, with a multi-stakeholder development team, led the development of the Canadian National HIV Pregnancy Planning Guidelines (NHPPG). The guidelines address fertility issues among HIV-positive individuals and couples and are a valuable reference for healthcare providers.
  2. With advisory support from the CTN, Does HIV Look Like Me? International and Vancouver Coastal Health (VCH) trained dentists, certified dental assistants and hygienists to offer and perform a rapid HIV test known as the Point of Care (POC) test in dental clinics located within high-risk populations. The project, a feasibility pilot that took place over eight months in 2011, was to determine if testing could be offered in dental clinics. Though the pilot has ended, the three sites continue to offer HIV screening to their patients as part of their regular care. As a result of the relationships that through this project, all students in their final year of dentistry at the University of British Columbia (the only school of dentistry in the province) attend HIV-specific workshops. The pilot continues to inform strategic decision-making regarding the possible rollout of a similar practice on a larger scale, identifying challenges and opportunities to doing so.
  3. Investigators from the Co-infections & Concurrent Diseases (CCD) Core are in the process of developing national treatment guidelines for HIV-HCV co-infected individuals, set to be released early in 2013.
  4. CTN members and staff were part of the group of international experts convened by the International AIDS Society (IAS) to develop a scientific strategy for research towards an HIV cure. Several priorities for basic, translational and clinical research were identified. [Nature Reviews Immunology, 2012]

All survey groups (N=67) believed that the CTN has created new knowledge to some degree as shown in Figure 3. The majority of postdoctoral and investigator survey respondents (N=48) agreed that CTN trials are innovative (81%) and are high impact breakthrough research (62%).

Figure 3: Extent to which the CTN contributes to translating the knowledge from research into real world applications, N=67

Figure 3: long description

Over 87% of these same respondents also agreed at some level with the following four statements:

Interviewees, however, expressed mixed feelings regarding the CTN’s success in knowledge translation. Several suggested the CTN has successfully communicated research findings and related information, while others indicated that the CTN should evaluate their communications strategy and seek ways to improve it, for example through their website or bulletins. Some commented on the CTN’s lack of contribution to clinical trials policies and procedures, practice guidelines and drug recommendations.

It is concluded that overall the CTN has facilitated the translation of knowledge into improvements in the health of Canadians living with or at risk of HIV/AIDS, health policy and/or the Canadian health care system. The CTN could be more effective in the knowledge translation activities by improving their communications strategy.

International Visibility

Ensuring the CTN has a strong global visibility was identified as important in positioning Canada as a global leader. The vast majority (93%) of survey respondents (N=67) agreed that the CTN is positioning Canada as a global leader in HIV/AIDS clinical research.

Figure 4: The CTN is positioned as a global leader

Figure 4: long description

The majority (75%) of investigator and postdoctoral survey respondents (N=48) agreed that CTN trials have resulted in major achievements that are referred to by international experts. Attending international conferences is another way the CTN ensures international visibility, as is having CTN investigators in prominent roles in international HIV/AIDS organizations. The CTN’s success in promoting itself internationally is expressed in the following quote:

The network has developed individuals who have been world leaders on the research front and the policy front. Canada has had two individuals, both involved with the CTN, who have been president of the IAS. I don’t think any other country has done that...All along we’ve had people playing important roles in international groups in the UK and South Africa. Our researchers occasionally get grants from the NIH. Anyone getting funding from outside Canada is a reflection of international relevance and the research we’re doing. (CTN, I31, p.9).

The CTN is thought to be well positioned as a global leader by a number of survey and interview respondents because they work respectfully and collaboratively with various communities of people infected and affected by HIV. This is understood to be a unique concept of the Program, one that strengthens the abilities of the Network. The CTN works in partnership with local clinics and the community domestically and abroad with a focus on equitable access to trials and treatment and enrollment in CTN or joint trials.

Over the past three years the international program has grown from 0 to including 5 countries: Uganda, Russia, China, Colombia and Botswana. While the CTN has a formal strategy in place for increasing the CTN’s work with other countries, researchers indicated through interviews and survey responses that they are unaware of it or how it is applied. Many connections are made on an ad hoc basis through personal relationships and informal networking. Engaging the international strategy could help ensure that the CTN can do relevant clinical trials, as international partners often have access to the populations required to do such studies. The CTN should also consider the purpose of making international connections. For instance, there was no evidence collected from interviewees or survey respondents pertaining to linkages with other clinical trial networks. Making connections within networks with similar mandates in other countries could be valuable to share ideas, best practices, and strategies.

Overall, the CIHR Clinical Trials Network in HIV/AIDS Program has assisted in Canada’s presence in the world forum on HIV/AIDS clinical research. Supported by the Program, and through the CTN’s efforts conducting and disseminating high quality research at home and abroad, the CTN has successfully advanced knowledge, translated findings to users and is seen as a world leader.

Training & Mentoring

The Program aims to provide a superior research training and mentorship environment to support capacity building. It does so by prioritizing learning opportunities for students as well as community members and researchers.

The Postdoctoral Fellowship Award Program

Of the CTN’s $4.56 M annual budget, $200K is allocated to the Postdoctoral Fellowship Award Program (PFAP), which provides support to clinical scientists within Canada and abroad. The CTN has been very successful in leveraging funds for this program and over the past 5 years has leveraged an additional $1M from external sources, doubling the funding available to support postdoctoral fellows.

Table 4 shows the number of Canadian and international fellowships awarded since 2008.

Table 4: Number of Canadian and International Fellowships Awarded, 2008-2013
2008-09 2009-10 2010-11 2011-12 2012-13
No. of Canadian postdoctoral fellowships 6 7 7 8 8
No. of international postdoctoral fellowships - - 1 2 1

Source: CIHR Canadian HIV Trials Network Progress Reports (2008-2012)

Data on career trajectory was available for individuals who have received a CTN Postdoctoral Fellowship Award from 1992 to 2012. Over 100 Fellowships were awarded to 59 individuals. Figure 5 shows present positions of awardees and indicates that 42/59 individuals who received a CTN Postdoctoral Fellowship Awardees since 1992 are still in academia, fellowships or student positions. It should be noted that there is some double counting in Figure 5 as some individuals are currently physicians as well as professors or assistant professors. In such cases they are counted in the physician and academia category.

Figure 5: Present Positions CTN Postdoctoral Fellowship Awardees From 1992 to 2012, N=59

Figure 5: long description

Figure 6: CTN Postdoctoral Fellowship Awardees from 1992 to 2012 (N=59) Who Have a Record with CIHR

Figure 6: long description

As noted in a recent report that investigated training models in HIV/AIDS:

“Based on roles and titles, virtually all [CTN Postdoctoral Fellowship Awardees] seem to have pursued a career with a core focus on HIV and/ or clinical research. It is clear that, at least for this kind of specialized research training, that the CTN model is highly successful in preparing its fellows for, and keeping them active in, careers involving HIV/ AIDS research and clinical practice.” (Campbell and Bisby, 2013)

The training provided by the CTN Postdoctoral Fellowship Program has led to several former postdoctoral fellows receiving CIHR investigator awards. Figure 6 shows that almost 1/3 of individuals who have received a CTN Postdoctoral Fellowship Award since 1992 have currently submitted an application to CIHR, 15 are or have been a principal investigator and 21 are or have been a co-investigator. Furthermore, 3/4 of the current cores are led or co-led by former trainees.

All respondents to the postdoctoral survey (N=17) expressed satisfaction with the training and mentoring provided by the CTN, with 10 being “very satisfied” and 7 being “satisfied” and they all agreed that the training is effective. Respondents to the investigator survey (N=31) gave similar results: 70% stated satisfaction with the training provided and 77% agreed that the training was effective.

Training and Mentoring for Members

As a compliment and extension of the Postdoctoral Awards Program the CTN continues to build capacity by providing informal mentoring opportunities to new investigators, contributing to their career development and providing collaboration opportunities across the Network. The seed money the Network has available to fund new investigators has provided them with the ability to conduct research as principal investigators alongside senior investigators. These development activities have contributed to younger investigators taking on more senior roles. For example, Dr. Ari Bitnun (postdoctoral fellow, 1999-2001) joined the SRC as a full member at the May 2010 meeting and in January 2011, Dr. Catherine Worthington, a social scientist new to the CTN, was appointed the PVP Core co-lead, along with Dr. John Gill. Dr. Worthington is the first social scientist to hold a Core Lead position at the CTN, signaling a shift and response to the clinical research landscape.

The CTN has pursued other training opportunities intended to strengthen the skills of their members and facilitate high quality clinical trials. For example, through an agreement with CAAN and the Canadian Medical Association (CMA), the CTN is facilitating bilateral understanding between the research community and community members. This tri-lateral partnership has resulted in the creation of an online module hosted on the CMA website, targeting practicing physicians in Canada in rural and isolated regions, to educate them on how to deliver culturally appropriate care for Aboriginal peoples infected with HIV/AIDS. This went live in April 2012, was expanded to all health practitioners in December and continues to be available to all CMA members. In another example, through the Networks of Networks, the CTN has been able to provide its network members an online Good Clinical Practices module, which can contribute to their continuing medical education accreditation.

Training & Mentorship Opportunities for the Community

Most stakeholder groups interviewed acknowledged the opportunities for training and mentoring of community members provided through the CTN as significant. Community members described their learning through involvement with the CTN as occurring through CTN-facilitated workshops for community-based members, attending CTN meetings, and being involved in research projects. However, community members strongly expressed that learning is in fact mutual between them and investigators, as the following quote demonstrates:

We’ve actually really been pushing reciprocal training as well to the core leaders because we feel very strongly that this needs to be an iterative process and that the researchers need as much capacity building around community issues as we need around research, in some cases more, because we’ve been around research for years. Some of us have PhDs. (Community Member, I19, p. 3)

Having committees set up to address specific populations, such as the Aboriginal Working Group, was described as a form of training and mentorship for clinical trials to community members but also for educating researchers about the needs of the community. Through attending CTN annual meetings and giving presentations, community members sought to inform investigators about the needs and concerns of people infected and affected by HIV in the relation to clinical trials.

Community members feel training and mentorship could be extended both for themselves and for investigators. One community member suggested that the CTN should provide training and mentoring opportunities for the community outside of large centers. This would promote involvement and contribution from the community at large versus those who live in larger centres and have access to developmental opportunities.

Overall, the training provided by the CTN is successful, rewarding and important. The PFAP is attractive to young scientists because fellows can advance into members and leaders within the Network. The bilateral training that occurs between researchers and community members is unique and enriches the work facilitated by the network.

Collaboration

Collaboration in the CTN takes place largely in the following two ways: (1) among researchers, including postdoctoral fellows, and (2) between researchers and community/partner organizations.

Almost all respondents in the groups surveyed (N=67) agreed that the CTN has created or enhanced a collaborative environment. Almost 80% of investigators and postdoctoral survey respondents were satisfied with the degree of collaboration created and/or enhanced by the CTN. They also agree that participation in the CTN aids or enhances the creation of teams. A Community Member described the CTN staff’s ability to speak the languages of many different groups, including: scientists, researchers, community-based groups, peoples living with HIV or AIDS, and government as a strength that allows the CTN to bring people together that might not otherwise be connected in addressing HIV and AIDS.

All groups surveyed (N=67) were asked the extent to which they agreed certain groups had influence within the Network. The results, presented in Figure 7, show that survey respondents perceive that partners, community members and affiliated investigators do have an influential role in the CTN.

Figure 7: Perceived role of influence of partners, community members and affiliated investigators

Figure 7: long description

Collaboration among Researchers

Most stakeholder groups interviewed find the CTN plays a significant role in fostering a collaborative environment among researchers. This is done primary through meetings, namely, the CTN’s semi-annual conventions, now consisting of one annual meeting and one investigator workshop, and those held periodically within the Cores. These meetings provide the opportunity for researchers to share ideas and potentially form partnerships to carry out specific projects.

Interviewees find the CTN facilitates inter-provincial dialogue thereby allowing researchers to collaborate across Canada. The CTN’s infrastructure and expertise has allowed for greater collaboration with other researchers and organizations. As a Core Researcher described:

...by having the infrastructure and having the ability to meet each other and knowing people’s areas of expertise, if I’m developing a project I know who in the country might have an interest...and who might be able to support me or review something or help me in the development of something. [The CTN] has brought people together with this interest. (Core Research Lead I4, p. 3)

The opportunity to talk to researchers from other disciplines, a feature that enriches research, was generally viewed as a strength arising from collaboration with the CTN by interviewees and survey respondents. Postdoctoral and investigator survey respondents (N=48) indicated that participating in the CTN has broadened their research interests and/or prospects (80% agreed).

As the CTN works to bring relevant people together it stands to reason that inter-institutional collaboration would be high within the CTN. This was measured through bibliometric data that considered the percentage of publications that included at least two institutional addresses as inter-institutional collaborations for the top 10 countries and the CTN producing publications in the field of HIV/AIDS. This analysis found that inter-institutional collaboration rates are highest for the CTN, where for all years from 2001 to 2011 the inter-institutional collaboration rate was above 85%. By comparison, the average annual inter-institutional collaboration rate was 48% for the top 10 countries over the same time period.

Figure 8: Inter-institutional collaboration rate of publications in the field of HIV/AIDS for CTN, top 10 countries

Figure 8: long description

Source: Data compiled by the OST from Web of Science (Thomson Reuters - BDBC), September 2012.

Collaboration with Trainees

Collaboration with trainees has been an access point to other organizations. Collaboration between junior and senior investigators occurs formally through the CTN Postdoctoral Fellowship Awards Program, and informally throughout the CTN as previously discussed in section 1.4. It was observed that co-sponsoring trainees in different organizations through the CTN Postdoctoral Fellowship Awards Program has led to greater collaboration within those organizations. For instance, international postdoctoral students are helping the CTN build relationships within international institutions.

Collaboration with Community Members

A major and unique strength of the CTN is its inclusion of the community in its activities with the desire to be sensitive to the community’s specific needs. The CTN staff has focused recently on forming linkages with community-based groups, including CAAN, All Nations Hope AIDS Network in Regina (ANHAN), and the African and Caribbean Council on HIV/AIDS in Ontario (ACCHO), to form relationships through which dialogue can happen between the researchers and community-based people, as well as government agencies.

Some interviewees from across most of the stakeholder groups described the CAC as a means through which the CTN fosters collaboration between researchers and community. The CAC not only introduces investigators and community-based members to one another but was also seen as providing a venue for the community to contribute to the network in an active and influential way. For example, all CTN trial protocols are reviewed by the CAC, ensuring that community members have a formal mechanism through which to provide input and ensure community involvement. A Postdoctoral Fellow described the importance of the CAC as follows:

We have had very little training in the language we use and the way we approach things. It’s great to be able to just chat to people from the Community Advisory Committee, and talk to them about how we’re doing things, the way we’re thinking of doing things.. It’s been really helpful and showed me a different way of looking at the way I write things. Before the meetings I would have had no idea to turn to. And to be honest I probably wouldn’t have even thought about it, which is terrible. (Post-doctoral Fellow, I13, p. 4)

The creation of the Prevention and Vulnerable Populations (PVP) Core has played a particularly important role in fostering collaboration with the community. The Aboriginal Working Group housed within the PVP was highlighted as a means by which members of the Aboriginal community play an influential role in ensuring CTN-supported research is ethically appropriate. A Community Member described her participation in the Aboriginal Working Group as important for incorporating into CTN-supported research Aboriginal medicines, philosophy, and ways of thinking in terms of adjusting to the HIV/AIDS they are facing.

Collaboration through Partnerships with Other Organizations

Collaboration has also been fostered through partnerships made with other HIV/AIDS organizations. Examples of such collaborations have involved either CTN investigators’ participation in multi-site trials, which include Canadian sites, or building capacity and supporting clinical trial infrastructure development in other countries such as Russia, China and several African countries. It was suggested that while participating in international clinical trials the CTN, by setting standards for international research, is demonstrating a leadership role globally.

Review of the CTN Progress Reports shows that partnership activity has increased from 37 distinct partners in May 2009 to 44 in May 2012.Footnote 6 The most recent Progress Report (2011-2012) stated partnerships included 2 sponsors in (addition to CIHR), 30 research partners and or fellowship sponsors (18 of which are international entities), 3 project based partners (2 of which are American based pharmaceutical companies) and 9 national partners. Community Members who work for such organizations described becoming more involved in the CTN due to the efforts of CTN management to engage more explicitly with the community. For example, the CTN and CAAN have developed a formal relationship with a Memorandum of Understanding (MOU). Through the MOU, a commitment of respect to each other’s mandates is described, as well as how they will seek out opportunities to work together, review proposals, and engage with the community. This type of relationship building is considered important because it helps the CTN raise their national profile and credibility.

Challenges in Fostering Collaboration

Despite its successes, there remain important challenges for the CTN in fostering collaboration. A few Core Research Leads said it is difficult for the CTN to compete with pharmaceutical companies which have more resources for collaboration with clinics and research centres in trials. Private scientific groups are less likely to participate in a project and when they do it is difficult to sustain their participation. One Core Research Lead perceives the CTN is privileging the participation of larger clinics that can contribute larger sample sizes over smaller clinics that do not have the same level of infrastructure.

As is common in all competitive research funding environments, competition among researchers can be a barrier to collaboration in the CTN. There has been concern among researchers that when a good idea is shared in a CTN venue but not accepted, it risks being adapted and presented by another researcher a few years later. Furthermore, collaboration reduces the possibility of being first author on publications, a desirable distinction.

A few interviewees find collaboration among researchers in the network weak. A Core Research Lead identified the main problem as a lack of joint ownership among collaborators throughout the research process. This respondent suggested a need to better involve research collaborators from the development of the project through to its execution and to share leadership.

Collaboration with community members was also noted as an area for improvement. The CTN needs to give greater attention to promoting the participation in research of affected communities, and specifically Aboriginal people, in ways they are comfortable with. A few community members would like to see a more proactive and influential role carved out for community-based members in the CTN that goes beyond advising. While all CAC members were invited and attended the fall investigator workshop in Toronto in October 2012, one community member interviewed stated the need for community members to be more involved in workshops where investigators develop ideas for protocols. Respondents suggested the CIHR Clinical Trials Network in HIV/AIDS Program play a greater role in the promotion of Aboriginal scientific viewpoints in research.

Implementation

This section will make careful distinctions between the CIHR Clinical Trials Network in HIV/AIDS Program and the Network, which is supported by CIHR funding though the Program. Program level issues will include whether the program has been effectively delivered, CIHR oversight and how the program aligns with CIHR’s mandate. Network level issues evaluated include the organization of the CTN, CTN governance structure, processes and decision-making.

Effective Delivery

The CIHR Canadian HIV Trials Network is committed to:

When AIDS was first detected in Canada in the early 1980s there were no treatments available for infected individuals. Now CATIE, Canada’s source for HIV and hepatitis C information, lists 29 possible drugs as well as complementary therapies on their website. Research supported by the CTN has contributed to the development of these drugs and therapies. While the disease is still spreading within Canada, the rate of infection has been decreasing as seen in Figure 9 below.

Figure 9: Positive HIV tests and AIDS diagnoses by year in Canada
Year 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
AIDS diagnoses, adult male 439 352 348 300 261 296 256 257 238 181
AIDS diagnoses, adult female 58 71 63 78 61 79 69 50 72 42
AIDS diagnoses, total, all ages 521 445 434 399 343 395 344 336 336 258 221 151
Positive HIV test reports, adult male 1,533 1,603 1,793 1,816 1,825 1,818 1,802 1,798 1,919 1,759
Positive HIV test reports, adult female 482 540 614 623 648 619 692 602 668 609
Positive HIV test reports, total, all ages 2,092 2,216 2,459 2,461 2,519 2,478 2,535 2,440 2,616 2,399 2,311 2,221

Source: PHAC, 2010

The CTN has pursued “scientific excellence and ethical integrity” by contributing to the body of HIV/AIDS research nationally and abroad, producing very high quality papers as measured by ARC and ARIF, maintaining a high level of collaboration with other institutions as measured by affiliated institutions of authors, building capacity through its training efforts and conducting outreach and partnership activities with community members and private industry. In addition to building capacity, conducting important research and contributing to a wide array of treatments in an area where there had been none, the CTN is visible on the world stage in HIV/AIDS research and is viewed as a global leader in this area.

Investigators interviewed benefited from being involved with the CTN program in the following ways (as stated in interviews): having participated in the CTN Postdoctoral Fellowship Program; making critical connections; infrastructure support that ensured the recruitment and retention of patients; received funding to hire research staff and acquire other research support; and the opportunity to participate in trials due to the CTN’s facilitation of the project. These quotes exemplify the effectiveness of the Network:

I think my career has been improved by it. It has provided me with access to people involved in the field that have helped me develop my own research skills. There certainly have been resources to enable me to conduct some of the trials I have done over the years with nursing supports and statistical supports and that sort of thing. So definitely in my career it has been advantageous. (Core Research Lead, I4, pp. 3-4)

CTN has done a great job of bringing all clinician scientist, social scientists and community organizations together under one umbrella organization with a mission to support and enrich HIV research...CTN has set up an excellent pan Canadian network of researchers and collaborators . It is a powerful organization... (Investigator Survey Respondent).

Overall, the majority of survey respondents from all groups (N=68) are either “very satisfied” (43%) or “satisfied” (41%) with the CTN. Thus the program has met its objectives and has been delivered effectively.

Network Structure

In 2008 the Network was reorganized from a regional structure to a core structure, allowing CTN investigators to intensify their efforts in four core research areas:

  1. Clinical Management Science (CMS)
  2. Co-infections & Concurrent Diseases (CCD)
  3. Prevention & Vulnerable Populations (PVP)
  4. Vaccines & Immunotherapies (VIT)

The Cores are intended to function as a catalyst of scientific activity in their area; ideas initiated and discussed at the Core Research team level become more fully developed concepts and then, with facilitation by the National Centre, move through development to final protocols for grant funding and implementation. Core teams are to focus the expertise of clinical investigators and nurses, trial coordinators, CTN support staff, and members of the HIV community, on generating trial protocols that address the most urgent clinical questions of the day.

Generally, the move from a regional structure to a core structure has been viewed positively. The role PVP plays in fostering collaboration is deemed particularly important in addressing the needs of affected populations in Canada. A similar comment was made of the Co-infections & Concurrent Diseases Core, suggesting the Core structure of the CTN more broadly has been successful in selecting research priorities that are relevant.

The CTN National Centre is located in Vancouver, BC. The Network itself currently includes 35 satellite centres across CanadaFootnote 7. The distribution of satellite centres is uneven: 37% are located in Ontario, 29% in Quebec and 17% in British Columbia. There are no satellite centres in Prince Edward Island, Nunavut or the North West Territories and only one centre SaskatchewanFootnote 8, where the rate of positive HIV tests was more than double the national average of 8.2 per 100,000 [PHAC, 2010].

Strategic Alignment and Direction

Strategic alignment between CTN and CIHR was identified by interviewees as underdeveloped. While the CTN shares information with the CIHR HIV/AIDS Research Advisory Committee (CHARAC and there are regular interactions between CTN and CIHR staff, the network does not have a direct and consistent role in informing the direction of CIHR HIV/AIDS Research Initiative strategy. A more formal relationship between the CHARAC and the CTN could allow CIHR to better learn from the CTN and use the network’s knowledge for leverage in future strategic planning and identification of priorities. The respondent also drew attention to CIHR’s Strategy for Patient-Oriented Research (SPOR) as another place for alignment, mutual learning for the network and synergy.

Many survey and interview respondents indicated that the Program should reconsider its strategic direction. In rethinking its strategic direction, the CIHR is also encouraged to evaluate the needs of the HIV/AIDS community and vulnerable populations and determine how to best meet those needs. The CTN was established at a time when there were no drugs to treat HIV and trials were urgently needed to advance knowledge, build research capacity and improve access. This is no longer the case today and CIHR should reevaluate its relevance domestically and internationally and define the objectives and scope of future funding accordingly.

CTN Governance and Decision-Making

The CTN is ultimately governed by two co-directors Martin Schechter and Aslam Anis. At the top of the committee structure is the Executive Management Committee (EMC) and the Scientific Steering Committee (SSC). They are informed by:

All protocols are reviewed by the SRC, unless they have already been peer-reviewed by another national peer-review panel such as those at CIHR or the National Health Institutes in the United States, and the CAC. Consent must be given by the CAC for protocols to proceed. The CAC also reviews all Informed Consent Forms.

Some interview and survey respondents identified the committee structure as performing well; interview and survey responses regarding the governance structure of the CTN were quite mixed. For example, the investigator survey asked respondents (N=31) to discuss their opinions on the structure of the CTN and how it operates. Of these responses, 11 were completely positive claiming the governance structure was appropriate, adequate, organized and functioning efficiently versus 13 responses that were negative discussing issues of transparency, top heavy control, and lack of investigator input into strategic direction (7 respondents provided no opinion). Similarly some interviewees commented on the lack of clarity with respect to how the CTN’s governance structure is organized and how the CTN central team functions.

Decision-making with regard to setting research priorities, disbursing funds within the CTN, and staffing was identified by survey and interview respondents as an area for improvement and greater transparency. There is a need to define work processes including a clear outline of who is responsible for making which decisions and how they will be made. There are risks associated with identifying research directions regardless of the level of decision making. For instance if Core Leaders act as principal decision-makers, it can be a struggle to bring innovative ideas up from the grassroots. In addition, Core Leads tend to be specialists within the research area of their Core and thus have a vested interest in which research is pursued. Alternatively CTN staff can play a stronger role in priority setting which would promote innovation and directed action in specific areas of research focus. Finding the appropriate balance of input into such decisions is challenging but may warrant further investigation by the Network.

Interviewee comments also addressed performance of the CTN’s management positions suggesting there is a lack of subject area expertise in CTN management. CTN staff acknowledges this and cited the conundrum of having expertise exist on EMC, the Committee responsible for making funding decisions, who can participate in those decisions without conflict. Currently all non-national centre staff who are members of the EMC are investigators who typically participate in at least one trial supported by the Network, making their participation in certain funding decisions a conflict of interest.

A few interviewee respondents acknowledged stagnancy in the most senior leadership of the network and feel there is a need to change the CTN’s management and leadership periodically in the interest of supporting emerging leaders and bringing new ideas and approaches to the network. Respondents also mentioned a lack of presence of the Co-directors outside of the National Centre (at conferences and so on).

Governance and resources are thought by a few to be too centralized at the CTN’s centre in Vancouver, given the size of Canada. The Network is meant to allow research centres across the country more access to a centralized and coordinated infrastructure. Investment in centralized services has thus resulted in less support to individual centres, some of which already have the capacity and service infrastructure also provided by the CTN. Some researchers do not think that the data from their clinical trials should be managed by the CTN given that many research centres have the capacity to do data management. Furthermore, some interviewees stated the location of the National Centre was problematic, preventing collaboration. It was said that the CTN’s communication with members weakens the further east of Vancouver they are. Conversely, it is also recognized that there are efficiency gains with having the Centre remain at St. Paul’s Hospital (UBC) partly because St. Paul’s provides additional financial and/or in-kind support, and contains an experienced staff committed to the CTN.

Services provided through the CTN

The CTN’s role was described across most of the stakeholder groups as being the provision of services and infrastructure that enable CTN investigators to conduct trials. These services and infrastructure include: research personnel, funding for meetings, CTN website, CTN newsletters, public relations support, and databases. In addition, the CTN plays a key role in networking, both in Canada and internationally, through its relationships with clinics and research centres. This feature facilitates the recruitment of participants for trials conducted in Canada, connects investigators with other HIV/AIDS organizations and supports the types of international collaborations described above.

Services provided through the CTN to researchers include study planning, study implementation, liaison and communications and funding support. These services are expected to increase efficiency as researchers need not start over with every trial. Investigators surveyed (N=31) answered questions regarding their use and satisfaction with the service offered; these results are presented in Figure 10. It should be noted that about 20-25% of respondents to each questions were neutral – neither satisfied nor dissatisfied with the services provided.

Figure 10 - Investigator and postdoctoral survey respondents satisfaction and use of CTN services, N=31

Figure 10: long description

Overall, the services are used across career stages but new investigators, categorized as having 0-5 years of experience, tend to use the Study Planning and Funding Support services more than investigators with more experience. Investigator survey respondents with more than 18 years of experience (N=6) were the only population that expressed dissatisfaction with services provided. One comment in particular worth noting is this:

“The critical elements of clinical trials are still left to the PI - these are study question, concept, design, proposal and protocol development, funding and CTN review processes, CRF development (content, not the forms), etc. These activities are mostly unsupported...Write-up remains the job of PI / authors for the most part.” (Investigator Survey Response)

This is interesting because the respondent is an experienced investigator who believes that these essential services which are offered by the network are “unsupported”.

Core Research Leads interviewed find the statistical support, the proposal writing and support with ethics important in conducting research projects. Support in the development of clinical trials provided by CTN staff was identified as helpful to CTN investigators, as clinicians may not have the time or expertise to pull proposals together in a way that makes them competitive and fundable. Interestingly, study planning and implementation services were the least utilized by survey respondents yet are intended to be available to streamline the processes of clinical trial protocols and execution. This may be because large centres such as the University of Toronto or McGill University do not take advantage of specific CTN services because they have in-house staff and infrastructure that most other centres in Canada do not have.

Protocol Review

Trials funded by the CTN are reviewed by two committees: the SRC, based on scientific merit and the CAC, based on community relevance. Respondents indicated that in efforts to obtain their funding as quickly as possible they often apply to both the CTN and CIHR simultaneously. While the presence of the CAC is specifically appreciated and considered critical for ensuring the research is relevant to communities affected, obtaining approval from two committees is thought to be onerous.

Interview and survey respondents made suggestions for streamlining the process include assigning project coordinators and/statisticians to each of the Cores so that they develop an advanced understanding of the language linked to the particular Core’s activities, or developing an expedited process whereby an investigator pitches an idea, submits a short application and gets scientific review more quickly. Combining the Community Advisory Committee and the Scientific Review Committee was also offered as a way to make the review process more efficient and therefore less costly. One respondent stated that despite a lengthy review process, it is sped up by the CTN’s support.

Enhance Opportunities for Community Members Participation

Despite the CTN’s efforts to involve the community, some community members expressed dissatisfaction through survey and interview comments. CAC members would like more developed mutually-agreed upon processes of governance. They also want more input into study designs to ensure the community’s understandings are incorporated and ultimately lead to greater research success. By involving community members throughout the study design, rather than at the end, could expedite the protocol review process (CAC reviews each trial protocol).

Two barriers to community participation were noted: First, individuals interested in being a CAC member are interviewed to determine their level of knowledge and so non-scientist community members are excluded from participating. There is a need to support non-researchers to broaden the pool of people who can contribute to the CAC. Second, community members participate on a volunteer basis. While their travel costs are paid and they are given a small stipend, many forego their salary to participate.

Efforts to honor and respect the engagement of various affected communities were acknowledged yet interview and survey respondents find there is still considerable work required to ensure cultural competency. Suggestions for improvement include providing CTN staff and members further assistance in developing understanding of the cultures of the communities they are working with, incorporating the knowledge of communities into work practices and research, and ensuring when discussions are being had that relate to communities that they are fully involved in decision-making. The resounding message in community members’ feedback is that the CTN could be better organized to deepen the engagement of communities infected and affected by HIV/AIDS. If this further engagement from the community is desirable, then the Program should determine how to better include community members to ensure active participation.

Efficient Delivery

The CTN has had recurring funding since 1990. This continuity is considered important in that it has allowed the CTN to develop considerable strengths and ensure the coordination of clinical trials research in Canada. These advantages are deemed by the respondents to be more important than perceived disadvantages such as developing a sense of entitlement, stagnancy in senior leadership and management, and a lack of innovation. A community member also supported ensuring stability for the CTN, given that its mandate involves significant relationship-building. Specifically, the CTN risks losing the people they have engaged if they cannot provide them with some degree of future security. Further, they argued that planning and carrying out the international research that characterizes a global leader is difficult if an organization is not certain of its future

The CIHR HIV/AIDS Research Initiative is responsible for investing funding CIHR receives through the Federal Initiative to Address HIV/AIDS in Canada (FI) and the Canadian HIV Vaccine Initiative. The overall annual budget of the FI is ~$73M per annum across four federal departments (CIHR, Health Canada, Public Health Agency of Canada and Correctional Services Canada). CIHR receives is ~$23M per annum to support strategic HIV/AIDS grants and awards under the FI. A portion of this funding, $4.55 M per annum, is for the CIHR Canadian HIV Trials Network. FI funding is on-going with no scheduled end date [PHAC, 2012]. The CIHR HIV/AIDS Research Initiative budget for 2012-2013 is broken down as follows:

Table 5: CIHR HIV/AIDS Research Initiative Budget
CIHR HIV AIDS Research Initiative 2012-2013 Budget
Federal Initiative to Address HIV/AIDS (FI) in Canada:
Biomedical/Clinical Research $7.53 million
Health Services/Population Health Research $5.68 million
CIHR Canadian HIV Trials Network $4.55 million
Community-based Research $3.12 million
Canadian HIV Vaccine Initiative $1.84 million
Total $22.72 million

All money is disbursed through the National Centre located at Saint Paul’s Hospital in Vancouver, BC. The CTN’s annual expenditures are depicted in Table 6. Table 6 indicates that more $2.4M of the CTNs directed grant is for salaries of CTN staff. While this seems high, the CTN employs 28 FTEs within the National Centre and another 8 FTEs within the Cores remotely. The average salary with benefits is about $65K. These figures are comparable to other networks.

Table 6: Breakdown of CTN Annual Expenditures
CIHR Annual Funding In Millions of Dollars
National Centre & Investigator support staff 1.9
Unit funding for CT and cohorts, pilot study 1.1 (1, 2, 3)
Core staff across Canada 0.5(3)
Investigator meetings (semi-annual) 0.3
NCC operations 0.3
Postdoc fellowships 0.2
Core specific meeting and travel 0.1
NCC travel and meetings 0.1
Total 4.5

Notes:

  1. includes $150k of pilot study funding in each of 2010, 2011, 2012.
  2. Unit funding supports data collection and patient reimbursement costs at the participating sites.
  3. About 35% of the CTN budget gets paid out in cash as support to unit funding, pilot studies and Core team staff support.

Value for investment

A few interview respondents said the CTN’s value is significant given the size of CIHR’s investment. They find the accomplishments of the CTN to be quite good, given the resources invested in the CTN program are substantially less than those invested in similar networks in other countries, such as the United States.

CIHR oversight

The sole-source RFA process is supported by CIHR Staff in the interest of efficiency and retaining the existing value. CIHR Staff Members interviewed feel CIHR’s oversight of the CTN is appropriate, given the annual external review process and the substantial funding investment. Throughout the current five-year term of the grant, the CTN has submitted substantial annual progress reports which are reviewed by a four-member international review panel. However it was suggested by some that the Program move from annual progress reports to bi-annual reports to CIHR as they are onerous and require significant resources. The CTN would also prefer their reporting to CIHR to involve more feedback and discussion with the peer reviewers, giving the Network an opportunity to answer questions before reviewer comments are formally documented.

Other interviewees feel greater oversight by CIHR is required, with respect to reporting. They raised concerns about the accuracy of information provided in the Network’s Annual Report and Progress Reports in terms of claiming responsibility for projects and publications that the network may have been only peripherally involved in, if at all.

Funding structure

A common theme from interview and survey respondents was the timeliness of funding. In order to maintain competitive in clinical trials, funding must be obtained quickly but the protocol review process was highlighted as far too lengthy and inadequate in ensuring trials are timely and competitive. It generally takes a minimum of 9 months to achieve funding for trials from CIHR, which hinders the competitive edge Canada may have, as well as their ability to partner on studies that will move faster. Direct funding for the trials allocated to the CTN for disbursement could allow the CTN to fund trials more efficiently.

Pilot study funding is an important feature of the CTN program for testing research ideas out on a small-scale to determine whether they are worth pursuing on a larger scale. These funds are very limited, making them difficult to access. The CTN staff would like funding for trial conception and operation. They also want flexibility in how their funds may be used and suggest this money come from CIHR’s existing funds earmarked for catalyst grants or meeting planning grants. Thus they would prefer to use their internal review processes to put out calls and manage the granting of funds.

Some interviewees suggested the Program consider the value added by more engagement with smaller centers and allocate funds accordingly. Currently satellite centers are concentrated within 3 provinces: Ontario, Quebec and British Columbia. Engagement with the smaller centers is an important aspect of clinical trials across the country, as well as accessing vulnerable populations and knowledge translation. Research tells us that institutions that participate in clinical trials have better health outcomes and health delivery [Majumdar, et al, 2008].

Leveraging funds

Investigator and postdoctoral survey respondents (N=48) indicated their level of agreement with the following statement: “My involvement with the CTN contributed to the successful leveraging of other funding or in-kind support for my research.” Just over half (58%) agreed with this statement, 10% disagreed and 16% said they either did not know or the statement was not applicable to them. Similar results were found when the same group indicated their level of agreement to: “Participation in the CTN has contributed to me receiving research funding”; 60% agreed, 15% disagreed and another 10% said they either did not know or the statement was not applicable to them.

As previously stated the Network has been quite successful at leveraging funds for the Postdoctoral Awards Program. However, the CTN has not been effective at leveraging funds to support its other activities.

Relevance

The following section will examine how the CIHR Clinical Trials Network in HIV/AIDS Program aligns with federal strategic directions, roles and responsibilities.

Canada has maintained a national HIV/AIDS strategy for more than 20 years

Canada has had a national HIV/AIDS strategy since 1990 in order to direct its response to the epidemic. In 2005, the Federal Government announced an enhanced and refocused strategy, the Federal Initiative to Address HIV/AIDS in Canada, striving for a fully integrated Government of Canada strategy and the direct involvement of people living with and at risk of HIV/AIDS. The government of Canada’s commitment to HIV/AIDS related research is apparent in its support of global initiatives, multilateral and bilateral programs, and partnerships with Canadians. Between 2005-2006 and 2011-2012, this support totaled more than $1.12 billion [Canadian International Development Agency, 2012]. In July 2010, the Government of Canada renewed its commitment to implement the Canadian HIV Vaccine Initiative (CHVI) in collaboration with the Bill and Melinda Gates Foundation with a combined contribution of $111 million.

In all disciplines of health research, clinical research plays a vital role in the health research continuum and the translation of new knowledge into improved health products, services and policy; acting as a bridge between basic science and patient care. Randomized controlled trials in particular are essential for proving the safety and efficacy of new prevention strategies or therapeutics. CIHR and the Federal Initiative have identified that maintaining and enhancing the infrastructure for HIV/AIDS clinical trials is a key component of a comprehensive and strong research response to the epidemic. The Jenkins Report [Jenkins et al, 2012] and the Government of Canada’s Economic Action Plan [Flaherty, 2012], which in part aims to support innovators and world class research, reinforces the need for clinical trials as they have the potential to lead to economic opportunities. However, it was noted that when the CTN was created 20 years ago, there were no available drugs or treatments. The environment affecting those infected and affected by HIV/AIDS is vastly different today and some interviewees questioned whether there was still a need to focus on clinical trials or whether the focus should be placed in a broader mandate more relevant to the emerging needs of the affected population s in Canada and abroad.

HIV/AIDS remains a priority and health concern for Canada

HIV/AIDS continues to be a health concern among Canadians and worldwide and related research continues to be funded under the Federal Initiative. The World Health Organization estimates that there were approximately 34.2 million people living with HIV in 2011 [World Health Organization (WHO), 2012], despite the fact that the number of new HIV infections has decreased 19% over the last decade [WHO, 2011]. Since reporting began in 1985, a cumulative total of 72,226 positive HIV test reports have been reported to the Public Health Agency of Canada through December 31st, 2010.

CTN fosters relevant research

One of the cornerstones of the CTN is to involve community members in all activities in the program. Community members were involved in the development of the previous RFA, are involved in trial development and the CTN strives to foster an environment in which various stakeholders (including researchers, community members and partners) have an active and influential role in network activities including prioritization of trials, trial protocols and application of research findings. The relationships fostered between the HIV/AIDS research community and community-based stakeholders are providing an important venue for community to become engaged in clinical research and shape that research. By listening to community members, the CTN is recognized as being in the process of better understanding the needs of those infected and affected by HIV/AIDS. A few community respondents expressed that the CTN was unique in engaging in these processes in the area of HIV/AIDS clinical trials and research. Through relationships with organizations, such as CAAN, the CTN is considered to be taking steps to address the needs of specific groups, such as Aboriginal communities, who have been identified as a population of highest priority for the CTN to address. The CTN is also thought to play an important role in knowledge transfer and communication between communities and researchers about clinical trials. This community awareness-building was expressed as valuable by community members and researchers alike.

A survey sent to community members (N=19) includes responses from members of HIV related societies as well as infected Canadians or Canadians related to an infected individual. All but one said that the CTN contributed to a great extent or moderate extent to improving health for Canadians. All community respondents said that the CTN fosters the development and conducts research relevant to the needs of the research community and all agreed to some extent that the CTN fosters development and conduct of research relevant to the needs of vulnerable populations in Canada.

Resources not offered elsewhere

Respondents from across almost all stakeholder groups interviewed expressed that the CTN plays a critical or unique role in ensuring clinical trials in HIV/AIDS are carried out in Canada. The network does so by providing funding for research-related costs (e.g., nurses’ salaries), research services (e.g., grant application assistance, data management), coordination and networking, and training which helps investigators better carry out trials. While investigators may have the capacity to do many of these activities, they do not necessarily have the time or infrastructure established to do so. Given that drug companies are not funding trials to the same degree they once were [Cameron Institute, 2011], a few respondents indicated the CTN offers resources that investigators would not be able to get anywhere else. Further, the research facilitated by the CTN is deemed significant and is considered unique because it is driven by the curiosity of researchers, not the profit-motives of pharmaceutical companies. A Core Research Lead described the CTN as offering the opportunity to work with other investigators who are conducting research not for economic gain but because they feel it is important, which facilitates collaboration.

CTN helps create an infrastructure for necessary collaboration

While there are other mechanisms available to provide necessary infrastructure, such as the National Centre’s of Excellence Program and similar albeit smaller scale initiatives, none of these approaches could have provided the sustained duration, level and type of support needed to deal with the type of epidemic that HIV has turned out to be. From its inception, the epidemic has evolved in ways that were unknown (e.g. development of resistance, changes in the relative role of transmission mechanisms that fueled the epidemic at different time periods, role of community advocacy and education, etc). The CTN has filled an important mandate envisioned by the Federal Initiative by providing an infrastructure and a means of collaborating for both clinician researchers and the community to come together and refocus and change direction as necessary, from the early focus on anti-retroviral trials, to the focus on salvage therapies to studying vulnerabilities and using observational cohorts to study system level impacts. Furthermore, in order to get sufficient participant enrolment, especially given Canada’s small population, clinical trials require collaboration between many research centres. Because of its strong infrastructure, the CTN is identified as meeting the need to bring together these actors in a way that ensures consistency and quality across sites. The CTN provides a structure which allows investigators to meet on a regular basis, exchange ideas and work together, without which respondents feel the collaboration required to complete trials and produce stronger findings would not necessarily happen.

CTN aligns CIHR’s mandate and priorities

CIHR’s vision is to position Canada as a world leader in the creation and use of health research knowledge that benefits Canadians and the global community. The Institute of Infection and Immunity, the lead institute for the CIHR HIV/AIDS Research Initiative, has identified HIV/AIDS as one of the Institute’s strategic priorities in its first strategic plan and remains one of five research priorities identified in its current strategic plan, 2007- 2012. Thus the CTN fits well within CIHR’s identified priorities.

Future relevance

In their interviews, key informants described the ways the context of HIV/AIDS has changed dramatically since the CTN’s inception. HIV/AIDS used to lead to death for most of those infected. There was a sense of urgency to conduct research resulting in better treatment. Today, treatment is widely available in Canada. There are far fewer new therapies coming to the market and they are largely led by pharmaceutical companies. Given this change in context, the responses pointed to a challenge for the CTN in adjusting to changing needs. The CTN should evaluate its strategic direction to remain relevant to the populations most in need and at risk. The CTN has made efforts to further collaborate with Aboriginal people and other vulnerable population although interviewees thought more work to engage vulnerable populations was a particularly important area requiring more attention by the CTN if it is to foster the development and conduct of research relevant to the specific needs of those communities. These respondents would like to see the CTN direct more funds towards addressing the populations most at risk.

The CTN has broadened its scope to other forms of research beyond clinical trials and there is debate on whether this is appropriate. Almost all respondents who addressed this debate feel the CTN should continue to move into supporting a broader range of research methods including basic science, social science, behaviourial science, Aboriginal science/Indigenous methodologies, and community-based research. It was suggested broadening its scope beyond clinical trials into these other forms of research would require the CTN to direct more of its funding to these areas. There were many suggestions for which areas should be of research focus, or in some cases further focus, in order for the CTN to foster the development and conduct of research relevant to the needs of the research community and affected populations in Canada. These areas were described as better addressing the current context of HIV/AIDS and, therefore, of most importance. These include:

Program Improvements

CIHR conducted a network scan of six research networks located in Canada, the United States, and Australia and across four research areas. The scan was designed with the purpose of learning more about the general operation of other networks in Canada and worldwide, with particular emphasis on best practices, lessons learned, and challenges. The participating networks were not chosen based on similarity or dissimilarity to the CTN; they were specifically selected by the working group based on the wide variance in funding amounts, funding sources, and organisational structure of each network. It was anticipated that the CTN could consult the information collected on these diverse networks to gain insight on alternative mechanisms of funding, delivery, and operations. Basic information on each of the networks is presented in the following table.

Table 7: Descriptive Information for Organizations Included in Network Scan
Research Area Network Name Acronym Country Total FundingFootnote 9
Cancer NCIC Clinical Trials Group NCIC CTG Canada $5.117 million + $50 million from industry
Southwest Oncology Group SWOG United States $39.47 million
HIV/Aids Kirby institute Therapeutic and Vaccine Research Program TVRP Australia 6.23 million
AIDS Clinical Trials Group – National Institute of Allergy and Infectious Diseases (NIAID) ACTG-NAIAID United States 24.67 million
Critical Care The Canadian Critical Care Trials Group CCCTG Canada $40,000 (all through membership fees, $400/member)
Multidisciplinary Canadian Network and Centre for Trials Internationally CANNeCTIN Canada $2 million + additional funds from industry

Of the networks considered, the Kirby Institute Therapeutic and Vaccine Research Program, (TRVP) located in Australia, is most comparable both in size and scope to the CTN. With $6M in annual funding, the TVRP conducts a range of clinical trials designed to assess the effectiveness of new HIV therapies, new treatment strategies or candidate vaccines (therapeutic and prophylactic). TVRP staff members provide leadership through the program’s role as an International Co-ordinating Centre for the INSIGHT network, a major international collaboration for the conduct of large clinical endpoint strategic trials in HIV disease which has more than 300 sites in 30 countries. The Kirby Institute is directly affiliated with the Faculty of Medicine at the University of New South Wales (NSW), and receives funding through the Australian Government Department of Health and Ageing. Infrastructure funding also comes from awards through the NSW Department of Health. Projects are funded largely through the competitive grant applications or collaborations with the pharmaceutical industry. The Kirby Institute TRVP differs from the CTN in that, while it provides support to individuals linked directly to the network’s projects, it lacks the CTN’s formal training structure.

Best Practices Identified in Network Scan

The network scan yielded the following insights that were considered best practices by our contacts at the respective organizations. These discernments can be considered best practices within the organizations scanned:

In most cases, CTN does these things and therefore these findings support the activities and design of the Program and Network. With respect to specialized staff the CTN has recognized a gap in some of their current capacities and has hired a new manager of data services to start in January 201. The CTN expects to staff a regulatory expert by March 2013.

Funding Improvements

Some funding improvements have been mentioned in Section 2.2. In addition to those, the network scan returned the following insights regarding effective and efficient funding practices:

Conclusions & Recommendations

The CTN has supported the development and conduct of innovative and important research. From 2008 to 2011, almost one third of Canadian HIV/AIDS related clinical trial papers were supported by the CTN and publications attributed to the CTN are of very high quality as measured by ARC and ARIF scores. At a national level Canada ranked 5th of the top ten productive countries in HIV/AIDS related research and achieved higher citation impact scores than the work average. Canada’s specialization in HIV/AIDS related research was above the world average from 2008-2011.

This program has been effectively designed and delivered as reflected by the level of satisfaction observed in survey and interview data. Researchers and postdoctoral students report a high level of satisfaction with communication and collaboration within the network, the training received, and the Networks contribution to and translation of research. Most researchers have used the services provided by the CTN but it should be explored why some members are not using the services available and ways to improve satisfaction for those that have used these services.

Network activities to engage and educate researchers, community members and partners, along with the PFAP, have helped to build capacity in the area of HIV clinical trials. In fact the CTN is acclaimed for their partnerships with communities affected by HIV/AIDS in Canada, and increasingly, internationally. Tension was noted in finding the right balance of community involvement to maximize the benefits attained and this important relationship should be considered in future program design and delivery.

The CTN has been successful in leveraging funds for the PFAP, doubling the funds available by adding another $1 million dollars over the previous funding term. The CTN is encouraged to utilize the successful model of leveraging for the PFAP to increase leveraging for other aspects of the network functions. This will raise the profile of the CTN and provide access to new opportunities.

The Network has improved their engagement with partners and stakeholders over the funding term. Over the last three years the CTN has increased the number of international partners from zero partners to five. The Program should encourage a better understanding of the purpose for engagement domestically and abroad. Engagement with relevant partners and stakeholders should not only be about research projects but also open a forum for sharing best practices, strategies and innovative ways to deliver a successful network. Further engagement with partners should include but is not limited to industry, provinces, other networks and centres of expertise in HIV/AIDs.

To date the Network has been required to submit substantial annual progress reports which have been reviewed by an international review committee. This has been demanding on CTN staff resources and had implications for burden on CIHR and reviewers as well. There was not enough evidence collected through this evaluation to support a specific oversight mechanism for the future. Thus the CIHR should consider revising reporting requirements in the future, based on purpose, value and burden for CIHR and the Network. Information discovered through this evaluation should be used as benchmark data for future performance and future reporting should include stronger outcome measures. Furthermore, the reporting structure utilized to date has not facilitated multi-directional communication between the Network, review committee and advisory committees, at times leading to confounding recommendations regarding strategic directions for HIV research. CIHR should look for ways to improve and clarify mechanisms for oversight and feedback into ongoing strategic development, looking for ways to increase the efficiency and accountability.

The next funding opportunity should consider the strategic focus of the Program. The Program has been responsive to the needs of the affected and vulnerable communities, which have changed dramatically over the 22 years since the CTN was created, and must continue to do so. This review should focus on the emerging needs of infected and affected populations in Canada, and internationally. The Program may consider expanding the mandate of the CTN to include more disciplines such as population health and behavioural research, and research areas such as Hepatitis C. Additionally, the CTN is encouraged to investigate opportunities for synergy in these new areas via other programs supported by CIHR, such as SPOR. The review process should involve consultation with advisory boards and CIHR senior management on the development of the FO.

Recommendations

  1. As a part of its renewed funding opportunity, CIHR should review the future relevance of a clinical trials network in HIV and look for areas of alignment within other CIHR programs.
  2. CIHR Clinical Trials Network in HIV/AIDS Program should encourage the CTN to address the identified concerns surrounding governance and leadership in a transparent and inclusive manner.
  3. The subsequent funding term for the CIHR Clinical Trials Network in HIV/AIDS Program should continue to enhance roles for community in the CTN.
  4. The CIHR Clinical Trials Network in HIV/AIDS Program should pursue more meaningful engagement with all stakeholders and reconsider the purpose of such engagement.
  5. CIHR should encourage the CTN to pursue partnerships leading to additional leveraged funds.
  6. To enhance efficiency and effectiveness, CIHR should streamline the reporting structure and enhance the mechanism for oversight.

Methodology and Data

Data Collection and Analysis Activities

Consistent with Treasury Board guidance and recognized best practice in evaluation (e.g., McDavid & Hawthorn, 2006; Wholey, Hatry & Newcomer, 2004), a range of methods were used to triangulate the program evaluation findings. The approach of using multiple methodologies, involving both quantitative and qualitative evidence, is designed to ensure that the evaluation findings are robust and credible so that conclusions can be drawn about the performance of the program and how it can be even more effective in the future. The following five methods and sources for data collection were applied:

  1. Document review and site visit
    Documentation reviewed included annual progress reports and peer review feedback, CTN internal documents, external oversight committee comments, the CTN website, and newsletters. Two members of the CTN evaluation team visited the CTN’s National Centre, located in Vancouver, BC with the intent to formally meet with CTN staff, review documentation, perform key informant interviews with critical stakeholders in-person, and to observe CTN processes and facilities in its authentic context.
  2. Surveys
    This evaluation used 3 different online surveys: one was sent to all active CTN affiliated investigators from 2009 (active was defined as participating in CTN studies or attending CTN semi-annual meetings) (n=30/80), all postdoctoral students awarded funding through the Postdoctoral Fellowship Program from 2008 (n=17/21), and past and current members of the Community Advisory Committee (CAC, n=19/34). The survey provided quantitative and qualitative data to address a range of evaluation issues and questions.
  3. Key informant interviews
    Semi-structured in-depth interviews were conducted to gather information on key stakeholders’ perceptions of the performance of the CTN and optimizing the impact of investments in the future. In order to capture a diverse and balanced view of performance, interviews were conducted with discrete sample groups including Network staff, researchers, trainees, community members, partners, national players outside of the CTN, international experts, and representatives from CIHR. Interviews were conducted by CIHR staff as well as an external contractor; a total of 32 individuals were interviewed.
  4. Bibliometric analysis
    Bibliometric analysis was contracted out to the Obeservatoire des sciences et des technologies (OST). The bibliometric analysis examined the production and scientific impact of CTN researchers publishing in the field of HIV/AIDS and HIV/AIDS clinical trials from 2001-2011 with the intent to contextualize academic production of the CTN within Canada and among the international community.
    The bibliometric methodology was formed in a collaborative process within the Evaluation Working Group and with subject area experts at the Observatoire des sciences et des technologie (OST). An initial list of 521 publications authored by CTN researchers from 2001 to 2011 was provided to the OST and was validated using headings as found in the catalogue of U.S. National Library of Medicine’s Medical Subject Headings (MeSH)Footnote 10, which rely on a controlled vocabulary to assign a medical domain to each paper indexed in the PubMed databaseFootnote 11. These MeSH terms were used to characterize CTN’s production and also to select a relevant comparative dataset. The CTN dataset was then created, based on a list of 447 publications authored by CTN and extracted from HubMed (which is a mirror database of PubMed) by a representative from the Network.
  5. Network Scan
    A scan of comparable networks, as identified by the CIHR CTN Evaluation Working Group, was conducted. The network scan included six research networks from Canada, the United States, and Australia. The networks, selected by the Working Group, spanned four research areas: a) cancer (NCIC CTG, SWOG); b) HIV/AIDS (The Kirby Institute, ACTG NIAID); c) critical care (CCCTG); and d) multidisciplinary (CANNeCTIN). Profiles on each participating network were developed using information available on their respective websites; the profiles were then sent to the networks to be completed (where information was not available online) and validated (where the profile was entirely completed using online information). Following the networks’ review, CIHR spoke with representatives of the networks via phone during 15-30 minute informal interviews, in an effort to gain context and obtain more in depth information on information provided in the network profiles.

A Working Group, consisting of CTN staff, CIHR program officers, CIHR evaluation team members, a community member and an external expert was essential in conducting this evaluation. The working group was relied upon for providing relevant documentation, identifying interviewees and networks to be included in the network scan, providing a list of CTN researchers and publications, and assisted in the development of the list of relevant MeSH terms used for the bibliometric analysis. The evaluation questions were created through consultation with the Working Group and they also assisted in the development and refinement of this report.

Data Limitations

Data limitations encountered in this study are outlined below, with the strategies used to mitigate them.

Bibliometric analysis: Bibliometric analysis in general has been criticized on the grounds that estimates of publication quality based on citations can be misleading and that citation practices differ across disciplines and sometimes between sub-fields in the same discipline (Ismail et al., 2009). To mitigate this, measures of other outputs are also used in this evaluation to assess knowledge creation as a result of the program.

The bibliometric analysis used in this evaluation is based on data for publications produced by CTN supported studies and researchers. While this method is commonly accepted (e.g. Campbell et al, 2010), an outright attribution between CTN support and publication bibliometric data cannot be made.

Limited program baseline data: This evaluation relied heavily on data collected at a single point in time using mainly self-reported data through interviews and surveys. In order to mitigate this limitation, triangulation of data was ensured where possible, thus corroborating findings through multiple lines of evidence.

Sampling issues: Self-selection bias is an inherent limitation of the online surveys and would likely have occurred in the investigator and CAC surveys (the postdoctoral survey was of the entire population and had a good response rate (81%). Responses to the investigator survey were distributed across the researcher population; responses to the CAC survey were weighted towards current CAC members.

Methodological limitations: The survey sent to all active CTN affiliated investigators from 2009 yielded the lowest response rate of 38% and so care was taken not to generalize the population based on survey responses. Similarly interview findings are not generalizable to the full population of each group that was sampled.

Lack of counterfactual evidence: Since the CTN has existed since 1990, counterfactual data was not available to suggest what the research environment may have looked like in absence of the CTN for the time period studied in the evaluation. Furthermore, collecting data from non-network stakeholders is difficult, as CIHR does not necessarily have contact information for such individuals.

Cost effectiveness analysis: A recognized limitation of this evaluation is that a cost-effectiveness study was not included in the methodology to assess the Program. Other aspects of efficiency and economy are included, in accordance with Treasury Board core issues; these provide evidence on program delivery and leveraging of partnership funding.

References

BC Centre for Excellence in HIV/IADS Primary Care Guidelines Panel. Primary Care Guidelines for the Management of HIV/AIDS in British Columbia, March 2011.

CIHR. Three-Year Implementation Plan and Progress Report [ PDF (2.8 MB) ]. Ottawa, ON. Cat No MR21-153/2011E-PDF, 978-1-100-19363-2.

Cameron Institute. Canadian Clinical Trial Summit - Starting the Conversation. Canada’s Research-Based Pharmaceutical Companies and Canadian Institutes of Health Research, September 15, 2011.

Campbell M and M Bisby. Funding HIV/ AIDS research training in Canada: What do we know about outcomes? Working draft of report being prepared for the Canadian Association of HIV Research, expected release date February 2013.

CIHR. Canadian Institutes of Health Research HIV/AIDS Research Initiative Strategic Plan 2008-2013, 2008 ISBN 978-1-100-10585-7.

CIHR. Quarterly Financial Report for the Quarter Ended June 30, 2011.

Canadian International Development Agency. HIV/AIDS: A major roadblock to development, 2012.

Flaherty, James M. Jobs, Growth and Long-Term Prosperity: Canada’s Economic Action Plan. Cat. No.: F1-23/3-2012E ISBN: 978-0-660-20189-4, 2012.

Jenkins, Dahlby, Gupta, Leroux, Naylor, Robinson. Innovation Canada: A Call to Action, Review of Federal Support to Research and Development – Expert Panel Report. Cat. No. Iu4-149/2011E-PDF ISBN 978-1-100-19384-7 60957, 2011.

McDavid, J.C and Hawthornes, L. R. L. (2006). Program Evaluation and Performance Measurement: An Introduction to Practice. Thousand Oaks, CA: Sage Publications.

Majumdar, S.R. Roe, M.T. Peterson, E.D, Chen, A.Y, Gibler, W.B., Armstrong, P.W 2008. Better Outcomes for Patients Treated at Hospitals That Participate in Clinical Trials. Arch Intern Med. 168(6):657-662.

Nature Reviews Immunology 12, 607-614 (August 2012) doi:10.1038/nri3262

Public Health Agency of Canada. At a Glance - HIV and AIDS in Canada: Surveillance Report to December 31st, 2010.

Public Health Agency of Canada. HIV and AIDS in Canada. Surveillance Report to December 31, 2009. Surveillance and Risk Assessment Division, Centre for Communicable Diseases and Infection Control, Public Health Agency of Canada, 2010. – 2010.

Public Health Agency of Canada. Federal Initiative to Address HIV/AIDS in Canada.

Statistics Canada, 2011. “Spending on Research Development”, The Daily, 2011.

Wholey, J.S. Hatrey, H.P. & Newcomer, K. E. (2004). Hadbook of Practical Program Evaluation. San Francisco, CA: Jossey-Bass.

World Health Organization. HIV/AIDS Fact Sheet No 360. October 2013.

World Health Organization. Global health sector strategy on HIV/AIDS 2011-2015 [ PDF (414.68 KB) - external link ]. France: World Health Organization, 2011 ISBN 978 92 4 150165 1 (NLM classification: WC 503.6).

Public Health Agency of Canada. At a Glance - HIV and AIDS in Canada: Surveillance Report to December 31st, 2010.

Appendix A: Matrix of Evaluation Questions & Indicators

Evaluation Questions Indicators Methods Sources
Performance
1. To what extent has the CIHR Clinical Trials Network in HIV/AIDS program positioned Canada as a global leader in HIV/AIDS clinical research? Knowledge Advanced
  • Scientific impact of CIHR Canadian HIV Trials Network researchers’ publications (identified by researchers as supported by the Network) compared with other Canadian researchers and the international community publishing in the field over time
  • Bibliometric Analysis
  • CTN supported research (as identified by the researchers)
  • Canadian and international researchers in the field
  • Comparison with other HIV trial networks and looking at the network over time
  • # of trials and # of trial participants
  • # of publications
  • Scientific accomplishments
  • Network scan
  • Other clinical trial networks
  • Document review
  • Annual Progress Reports and Peer Review feedback (2008-2012)
  • Other CTN documents (website, newsletters, etc.)
  • Extent to which researchers report that their CIHR Canadian HIV Trials Network research resulted in major achievements that are referred to by international experts
  • Examples of important and innovative work by CIHR Canadian HIV Trials Network researchers, trainees or community members – that resulted from or strongly benefited from CTN involvement
  • High impact breakthrough research, as identified by the researchers themselves
  • Survey
  • CTN researchers
  • CTN trainees
  • CTN community members
  • Key informant interviews
  • CTN staff, researchers, trainees, community members, partners, national players outside of the CTN, international experts, representatives from CIHR, International Oversight Committee
Knowledge Translation
  • Comparison with other HIV trial networks (when possible) and looking at the network over time
  • Knowledge Translation and dissemination activities (e.g., information sessions, guidelines, webinars, patents, new vaccines/drugs, etc.)
  • Network scan
  • Other clinical trial networks
  • Document review
  • Annual Progress Reports and Peer Review feedback (2008-2012)
  • Other CTN documents (website, newsletters, etc.)
  • Extent to which the translation of CIHR Canadian HIV Trials Network research findings has led to improvements in the health of Canadians living with or at risk of HIV/AIDS, health policy and/or the Canadian health care system (include examples)
  • Survey
  • CTN researchers
  • CTN trainees
  • CTN community members
  • Key informant interviews
  • CTN staff, researchers, trainees, community members, partners, national players outside of the CTN, international experts, representatives from CIHR, International Oversight Committee
Image
  • Extent to which individuals view the CIHR Canadian HIV Trials Network as a global leader in HIV/AIDS clinical research
  • Extent to which the CIHR Canadian HIV Trials Network has supported individuals as global leaders in HIV/AIDS clinical research
  • Extent to which the CIHR Canadian HIV Trials Network members are seen as individual global leaders in HIV/AIDS clinical research
  • Survey
  • CTN researchers
  • CTN trainees
  • CTN community members
  • Key informant interviews
  • CTN staff, researchers, trainees, community members, partners, national players outside of the CTN, international experts, representatives from CIHR, International Oversight Committee
2. To what extent has the CTN enabled effective training and mentorship?
  • Value of receiving training and mentoring in a HIV trial network
  • Extent to which researchers, trainees, and community members report CIHR Canadian HIV Trials Network training and mentoring is effective
  • Extent to which researchers and trainees report satisfaction with CIHR Canadian HIV Trials Network training and mentoring
  • Extent to which researchers, trainees, and community members attribute their career advancement to the CIHR Canadian HIV Trials Network training and mentoring
  • Perceptions regarding the extent to which the CIHR Canadian HIV Trials Network prioritizes new researchers
  • Number of trainees that subsequently pursued research careers in HIV/AIDS related fields
  • Key informant interviews
  • CTN staff, researchers, trainees, community members, partners, national players outside of the CTN, international experts, representatives from CIHR, International Oversight Committee
  • Survey
  • CTN researchers
  • CTN trainees
  • CTN community members
  • Number and type of training opportunities (by researcher, trainee, community member)
  • Number of post-doctoral fellowships
  • Number of new researchers
  • Document review
  • Annual Progress Reports and Peer Review feedback (2008-2012)
  • Other CTN documents (website, newsletters, etc.)
3. To what extent has the CTN fostered a collaborative environment in which researchers, community stakeholders and partners have an active and influential role in the network?
  • Perceptions as to whether the CIHR Canadian HIV Trials Network has created or enhanced a collaborative environment
  • Perceptions as to whether researchers, community members and partners have an active and influential role in the CIHR Canadian HIV Trials Network
  • Perceptions regarding the value of researchers, community stakeholders and partners having an active and influential role in the network
  • Key informant interviews
  • CTN staff, researchers, trainees, community members, partners, national players outside of the CTN, international experts, representatives from CIHR, International Oversight Committee
  • Survey
  • CTN researchers
  • CTN trainees
  • CTN community members
  • Extent to which the CIHR Canadian HIV Trials Network has created or enhanced a collaborative environment:
    • Number and type of network members (researchers, nurses, community, etc.
    • Teams created or enhanced through the CIHR Canadian HIV Trials Network
    • Researchers, trainees and community members who reported engagement in effective collaboration
    • Researchers, trainees and community members who reported satisfaction with the degree of collaboration
  • Survey
  • CTN researchers
  • CTN trainees
  • CTN community members
  • Number and type of network members (researchers, nurses, community, etc.)
  • Number of and type of research teams
  • Document review
  • Annual Progress Reports and Peer Review feedback (2008-2012)
  • Other CTN documents (website, newsletters, etc.)
Implementation
4. To what extent has the CTN and the CIHR Clinical Trials Network in HIV/AIDS program been efficiently and effectively delivered?
  • Perceptions regarding the appropriateness of the current leadership, governance and management structures
  • Identified strengths and challenges of CIHR Canadian HIV Trials Network implementation
  • Perception of the extent to which partnerships maximize the potential impact of the CIHR Canadian HIV Trials Network
  • Identified ways in which involvement with the CIHR Canadian HIV Trials Network contributed to researchers success in leveraging other funding and in-kind support
  • Perceptions regarding barriers to leverage additional funding and in-kind support
  • Perceptions regarding value of partnerships
  • Perceptions regarding barriers to establishing and maintaining effective partnerships
  • Appropriateness of CIHR’s degree of involvement/oversight in/of the network
  • Survey
  • CTN researchers
  • CTN trainees
  • CTN community members
  • Key informant interviews
  • CTN staff, researchers, trainees, community members, partners, national players outside of the CTN, international experts, representatives from CIHR, International Oversight Committee
  • Comparison of cost with other networks
  • Total funding for network ($) (core + leveraged from elsewhere)
  • Cost of administration
  • Comparing governance arrangement for the CTN with similar HIV clinical trials networks
  • Comparison of CTN documents, including Annual Progress Reports with network objectives
  • Network scan
  • Other clinical trial networks
  • Document review
  • Annual Progress Reports and Peer Review feedback (2008-2012)
  • Other CTN documents (website, newsletters, etc.)
  • Partnerships:
    • # of partners by type (Canadian, International, sector)
    • Financial and in-kind support received
    • Duration of involvement
  • Funds leveraged ($) and source (organization)
  • Cost for CIHR to run the program
    • Direct costs (e.g., staff time, program delivery, travel for competition, etc.)
    • Indirect costs (e.g., IT, utilities, rent, equipment, office supplies, etc.)
  • Survey
  • CTN researchers
  • CTN trainees
  • CTN community members
  • Document review
  • Annual Progress Reports and Peer Review feedback (2008-2011)
  • Other CTN documents (website, newsletters, etc.)
  • CIHR financial documents
Relevance
5. Is there a continued need for the CIHR Clinical Trials Network in HIV/AIDS program, and how does this align to government priorities and federal roles and responsibilities?
  • Alignment of CIHR Clinical Trials Network in HIV/AIDS program with government policies, priorities and federal roles and responsibilities
  • Alignment of CIHR Clinical Trials Network in HIV/AIDS program and CTN activities with CIHR’s mandate and priorities
  • Alignment with federal initiative to address HIV/AIDS in Canada
  • Document review
  • Annual Progress Reports and Peer Review feedback (2008-2012)
  • Other CTN documents (website, newsletters, etc.)
  • Documents such as HIV AIDS research Initiative 2008-2013, Strategic Plan, Jenkins Report, S&T Strategy, etc.)
  • Perceptions regarding the benefit to participants, as a result of participating the CIHR Canadian HIV Trials Network research
  • Perception as to whether the CIHR Canadian HIV Trials Network fosters the development and conduct of research relevant to the needs of diverse communities, including vulnerable communities in Canada
  • Survey
  • CTN researchers
  • CTN community members
  • Key informant interviews
  • CTN staff, researchers, trainees, community members, partners, national players outside of the CTN, international experts, representatives from CIHR, International Oversight Committee
Program Improvements
6. What alternative approaches to delivering the program can be identified through lessons learned/best practices and how could these be used to support improvements for this program and others?
  • Suggestions for alternative approaches to improve the network
  • Suggestions for alternative approaches to improve the funding program
  • Key informant interviews
  • CTN leaders and staff, researchers, trainees, community members, partners, national players outside of the CTN, international experts, representatives from CIHR, International Oversight Committee
  • Survey
  • CTN researchers
  • CTN trainees
  • CTN community members
  • Comparison with what other HIV clinical trials networks are doing
  • Lessons learned identified by other networks
  • Best practices implemented within other networks
  • Network scan
  • Other clinical trial networks

Footnotes

Footnote 1

Bibliometrics analysis is used to describe publication patterns within a field of research. This information can be used to evaluate the influence/performance and to provide comparisons.

Return to footnote 1 referrer

Footnote 2

These figures exclude pilot study trials and proof of concept trials, which the CTN began supporting in 2010-2011. Since 2010 the CTN has supported 12 unique pilot study or proof of concept trials. Comparing publications to trials and pilot study and proof of concept trials indicates the CTN yield about 3.5 publications per study.

Return to footnote 2 referrer

Footnote 3

The bibliometric analysis calculated an ARC score for the top 10 countries publishing in HIV/AIDS related research and then normalized to 1.0. An ARC value above 1.0 for a country means that, on average, the country's publications in that field are cited more often than the world average. ARIF, also normalized to 1.0, provides a measure of the scientific impact of the journals in which the papers are published.

Return to footnote 3 referrer

Footnote 4

ARC could not be calculated for CTN clinical trial related papers as the body of papers was too small to produce a significant figure.

Return to footnote 4 referrer

Footnote 5

Network of Networks (N2) is non-profit alliance of Canadian research networks and organizations working to enhance national clinical research capability and capacity. It brings together trialists and clinical research professionals from across the country, N2 provides a common platform for sharing best practices, resources and research-related content to ensure efficient and high-quality research, integrity of clinical practices and accountability.

Return to footnote 5 referrer

Footnote 6

Some double counting exists; in any given year presented up to 3 project based partners are also research partners and or fellowship sponsors.

Return to footnote 6 referrer

Footnote 7

For locations, visit The CIHR Canadian HIV Trials Network (CTN) Website.

Return to footnote 7 referrer

Footnote 8

The CTN has been working with Regina General Hospital and plans to open a new satellite in Saskatchewan in 2013.

Return to footnote 8 referrer

Footnote 9

Self-reported annual budgets, as off November 2012. Foreign currencies have been converted into Canadian dollars using Bank of Canada exchange rates as of Jan 20, 2013.

Return to footnote 9 referrer

Footnote 10

U.S. National Library of Medicine’s Medical Subject Headings (MeSH).

Return to footnote 10 referrer

Footnote 11

PubMed database.

Return to footnote 11 referrer

Date modified: