Initial Implementation Working Group Final Report with Recommendations (2007): Appendix F[ Return to table of contents ]
Appendix F: Final Report
Data Collection and Analysis Phase
2006 Evaluation of CIHR Document
Best Practices for Protecting Privacy in Health Research
June 28, 2007
Senior Policy Advisor, Ethics Office
Canadian Institutes of Health Research
Ottawa ON K2A 1P9
Table of Contents
- Summary Report
- Document-Recipients: Interview Findings
- Peer Review Chairs: Interview Findings
- Peer Review Chairs: Text Responses
- Ethics Community Stakeholders: Interview Findings
- Ethics Community Stakeholders: Text Responses
- REB Representatives: Interview Findings
- REB Representatives: Text Responses
- Focus Groups Report
- Web Survey Report: Findings
- Web Survey Report: Text Responses
List of Tables
Table 1 presents the numbers of persons identified to be interviewed for each of the data sets and the numbers of persons actually interviewed in each group. In addition to the five sets of persons individually interviewed, data was also collected by means of focus groups and a web survey. These latter two are not included in Table 1; for neither group was there an estimate of the "Number in Full List".
Following Table 1 is a description of each of these seven groups.
|Category of Respondent||Number In
|Recipients of the PBPs document||8||7|
|CIHR peer review Chairs||32||27|
|Ethics community stakeholders||21||19|
|Research ethics board members||50||46|
Recipients of the PBPs document
CIHR provided a list of eight persons who had requested from CIHR copies of the Best Practices for Protecting Privacy in Health Research document and who had accepted to be interviewed. One person, due to personal circumstances, withdrew leaving a total of seven persons who were interviewed.6
CIHR Peer Review Committee Chairs
Thirty-four Peer Review Committee Chairs were sent e-mail messages by CIHR in order to provide prior notice of the disclosure of their names to a contractor for purposes of a telephone survey. Two replied that they would not be able to participate. The remaining 32 were contacted by e-mail and by telephone. Of these 32, 27 were interviewed by telephone. The remaining five interviewees either did not respond to the invitations to participate or notified the caller that they were not available for an interview.
Ethics community stakeholders
A list of 21 ethics community stakeholders was provided by CIHR. Ethics community stakeholders are defined by CIHR as "research funding bodies, research ethics bodies with an education mandate, Aboriginal research and ethics bodies, and federal and provincial data custodians." Two of the persons on the list were also members of the CIHR Privacy Advisory Committee that had developed the Privacy Best Practices document September 2005. Interviews were completed with 19 of the 21 persons.
Research ethics board members
A sample of 50 Research Ethics Boards (REBs) was drawn using random selection from the National Council on Ethics in Human Research (NCEHR) list of Canadian REBs. Two experienced interviewers conducted all of the interviews which were predominantly with the Chair of an REB. They reached representatives of 46 of the REBs. Early in the interview process, the interviewers realized that a large proportion of respondents were either unaware of the document, or had little knowledge of the PBP document. Of the 46, 21 were in this category. In some cases, the respondent had decided, based on a description or a quick scan of the document, that it was not required for the operations of their REB. Therefore, with the agreement of the Project Authority, a variant of the interview schedule was prepared and used for this subset of respondents; it concentrated on REB operations and concerns regarding privacy without reference to the CIHR Document. The other 25 respondents were asked the full set of questions.
Three focus group sessions sought views on the CIHR Best Practices for Protecting Privacy in Health Research document (PBPs). One focus group was with members of a single REB. The other two brought together members of REBs from different regions and institutions.
Twenty persons volunteered to answer the questions posed by a web carried survey hosted at the CIHR web site over a period of about five months.
Of the seven data sources, two represent a defined population. The labels for the other five groups, in spite of being labels for identifiable groups, should not be understood to mean that the individuals within a group represent the population suggested by the group label. The two groups for which samples from known populations were obtained are the Chairs of CIHR Review Committees and the representatives of Canadian REBs.
The implications of this observation are:
- The views of the Chairs of CIHR Review Committees and the views of the representatives of Canadian REBs may be generalized to the respective populations from which each was drawn.
- It is also the case that the views of the Chairs of CIHR Review Committees may be compared to the views of the representatives of Canadian REBs.
- For the other five groups, which cannot be shown to represent defined populations, it would be inappropriate either to generalise from the views of a data sample to an intended population or to make comparisons across these five data samples.
The inability to make comparisons across all of the populations for which such comparisons were wanted is unfortunate. A comparison across groups of respondents is what the IIWG (Privacy Best Practices Initial Implementation Working Group) wished to achieve. When both of two conditions - proper sampling from a known population and an adequate response rate (80% response rate, or higher) - are met, formal comparison across data samples is feasible. If both of the two conditions are not met, then such formal comparison cannot be made and would not be written into a formal evaluation report. Given that the formal conditions7 for sampling from the intended populations were not achieved for five of the seven groups, formal comparison across the seven data sets is not possible even when the response rate from a designated list of respondents has been achieved.
The reality that some of the data samples do not represent a true sample from a defined population should not be construed as a suggestion that their views are not of value. Their views have been most informative and insightful and are amenable to informal comparison. In the absence of formal comparisons, informal comparisons can still be made. Such informal comparisons (dependent on the adequacy of the judgement that the data sample represents a population) are frequently made and the Appendices to this report allow for such informal comparisons.
- Those who have received a copy of the CIHR Best Practices for Protecting Privacy in Health Research document (to be referred to on this report either as PBP or as the Document) typically did so in the period between September 2005 and September 2006; a number received their copy earlier and very few after September 2006.
- The persons interviewed had most frequently become aware of the document between September 2005 and September 2006 or earlier. The earlier date was especially the case for persons in offices with responsibility for privacy practices.
- About half of all persons interviewed had participated in some form of training on the document.
- A commonly voiced problem is to find ways to get the information in the Document to researchers, most of whom are unlikely to review a long online document. Suggestions by which to accomplish this included a short handbook version, an executive summary version and a well indexed Document that allowed for more rapid finding of the item of interest at a given moment.
- Younger researchers tend to have been exposed already to varying degrees to ethics and privacy requirements. Many respondents find that their greater challenge is to convince the veterans, or just engage their attention, as they are the ones who might view the focus on privacy as an additional requirement, as an obstacle.
- Although many respondents (although not all) were aware that the PBPs document exists, many also claimed little knowledge of its contents, on the grounds that they did not have a need for it. This was particularly the case for CIHR Peer Review Committee (PRC) Chairs and Research Ethics Board (REB) representatives.
- Since many respondents were not using the document, it is not surprising to find that they also feel that the document is not making a contribution to improving practices. Some respondents made remarks to the effect that since becoming aware (very recent in many cases), they found some of the information in the document to be potentially very useful.
- The manner in which the Document is used is a function of the role played. REB members did not use the Document directly; their uses are typically in an educational role (other researchers, medical students) and are largely outside the REB context,
- Those who use the Document report it to be highly readable, well-formatted, comprehensive and very useful. Some who use it less (or not at all) criticize its length. There was very little criticism of style and format, except for searching difficulties in the '.pdf' version.
- The Document is most frequently used as a reference both for own use and as a reference document to which people new to the field (including students) may be referred. At least some of those uses are in the context of policy development for a jurisdiction (political or institutional).
- Peer Review Committees do not frequently refer to the CIHR PBPs document. There are a number of reasons including: infrequent meetings; high turnover of members; low level of Document knowledge by committee members; and the presumption that others (the REBs in particular) will take responsibility for the proper privacy procedures.
- More than half of the respondents have used the PBPs document for study and for reference purposes. There has also been some use made in teaching and in developing guidelines or practices. Those who had not used it explained that it was not pertinent to their day-to-day duties.
- One reason many researchers, including Peer Review Chairs, are not well versed in the privacy issue is that they do not see a need. They view the issue as being well handled elsewhere (REBs, Provincial Privacy legislation and the Tri-Council Policy Statement).
- When used as guidance by a REB, it was most frequently used as reference for the conditions for setting up long-term research databases, for data sharing within the research team or institution, for determining if consent is required and for secondary uses of data. The Document served to heighten awareness of key privacy issues; foster changes to research design and process; engender more consistent application of privacy protection and improved understanding of key privacy issues among Research Ethics Boards.
- The PBP Document is found useful for those tasked with reviewing proposals for privacy issues within Canada. Many felt it could be made even more useful with a more comprehensive, graphical approach to the universe of various requirements. A few signalled a need for more inclusion of the diversity of academic disciplines; for example, researchers in some humanities areas (e.g. English) and other disciplines (e.g. fashion design) do not realize that there are ethical and privacy dimensions to their research involving human subjects.
- Some REBs, especially smaller ones with lower volumes of cases, think that they may be overly cautious in applying privacy legislation, because they are not aware of a lot of variations and nuances that have become routine in locations with high volumes of health research, and therefore more familiarity on the part of the lawyers and others sitting on the REB. It was also noted that federal and provincial privacy laws, being general in scope, are not self-evident with respect to how they apply to health research specifically. REBs rely on lawyer members for specific guidance.
- It was recognized that the privacy issues related to the banking, storage and use of biological materials (blood and other human biological materials such as blood type, DNA code and the presence or absence of disease), are beyond the scope of this Document. Several on the persons interviewed indicated they would like to have guidance on the privacy issues associated with the storage and use of such materials.
- Many researchers are looking for a knowledge base to serve their need to ensure privacy in an appropriate fashion. The full Document may be an excellent starting point for creating such a knowledge base - that is, a dynamic, on line electronic document rather than a static paper document. Resources could include, for example: discussion forums; a facility for contributing; storing and making available cases; and training modules.
- Privacy issues are becoming increasingly complex for studies which are multi-jurisdictional in nature. The internationalization of research and the broadening of health research into disciplines other than bio-medical increase the complexity.
- Social science researchers, in particular, pointed to a need for specific guidelines and best practices for Aboriginal health research. It was known that CIHR is now working with this topic and seeks to develop Aboriginal-specific health research guidelines to ensure adequate protections for Aboriginal research participants.
- For a significant number of Document users, patient consent was a very important issue. Respondents wanted more assistance to determine when consent was required, and when "opting out" was acceptable. 8
- It was recognised that the Document does deal with physical and technological security (electronic databases) although inadequately in the views of some. Given that researchers, especially social scientists, also collect audio and video information, and that commercial electronic data security has recently be shown to often be weak, unreliable, it was suggested that CIHR may wish to have a more comprehensive look at the whole area of personal information stored in all media.9
- Two issues emerged for social science researchers in particular with respect to paediatric and education-institution research: (1) the age of consent is seen to be a grey area and (2) educational research, especially school based research. Much research is ongoing in schools without reference to privacy concerns, and without submission for review to REBs. In view of the increasing amounts of school-based research and research in Children's' Hospitals, this 'grey area' is in need of additional guidance.
- The need to keep the Document up-to-date, current, was stated. This was not a suggestion that the Document was out-of-date but recognition of a rapid pace of change and the increasing depth and scope of the privacy issue area.
- It was suggested that CIHR should help to harmonize competing demands of federal and various provincial legislation, especially in trials and health research consortia which cross jurisdictional lines, quite often international boundaries.
- A number of researchers asked for the information in the Document to be organized to facilitate rapid access; terms such as "chewable chunks", "Coles Notes", "PowerPoint" presentations, training modules in various media and on-going on-line communication among researchers, were used to describe what was wanted here.10
- Suggestions for improvement of the PBPs Document included:
- CIHR funding could be made contingent upon whether privacy concerns have been well handled in a funding proposal.
- Researchers asked for tools and guidelines to help them deal with privacy requirements efficiently and effectively in their proposals.
- There was a call for better ways and means for treating the privacy issue in the context of multidisciplinary and multi-jurisdictional research proposals such as proposals that involve institutions in several provinces, and, especially in clinical drug trials, the internationalization of research proposals).
- It was suggested that CIHR develop and use marketing, education and awareness programs aimed at the research community; their focus would be to help researchers understand that their interests are well-served if they are familiar with the Privacy Best Practices Document.
- The search for balance between privacy issues and personal health issues was a common theme. Some worry that patients are distracted from understanding risks to their health and security by lengthy explanations of confidentiality safeguards.
- Some see a conflict between the need to respect privacy and the wish to facilitate research. Some voiced concern that privacy issues may act to discourage useful research programs, particularly health related social science research, epidemiological studies on large populations, and secondary research using clinical case records where consent for research use cannot be secured.
- Among the suggestions to CIHR for better fostering the development of privacy best practices in Canada were:
- A "Tool Box" of cases, interpretations, etc., modelled on the case law approach
- frequently asked questions (FAQs)
- an ongoing discussion forum on-line
- additional training together
- more CIHR presentations
- It was clearly stated that the privacy issues related to the banking, storage and use of biological materials (blood and other human biological materials such as such as blood type, DNA code and the presence or absence of disease), are not covered sufficiently in this document. Several on the persons interviewed indicated they would like to have guidance on the storage and use of materials.
- Peer Review Committee Chairs pointed to the conflict of values that may occur between patient and privacy rights. Some people see it as in their interest to be research subjects; this includes the wish to have resulting information made available to their physicians or others. At the same time, people seek to limit sharing of their private medical information with employers or insurance companies. Better procedures, guidance, are wanted to accommodate both sets of rights.
- Those who use the Document report that they find it to be useful, most frequently as a reference document. Those who use the Document are more than 'quite satisfied' with the document. They find it to be well-formatted, readable, user-friendly and helpful.
- Document users find it "quite easy" or "very easy" to use. They attribute this to their familiarity with the subject matter, and also because it is well written and organized. The main reason that some found it "quite hard" to use is its length. There were also concerns about the challenge of summarizing such a vast and complex issue.
- Many of the persons interviewed see the Document as large and somewhat intimidating.
- Many were of the view that integrating the PBP Document in a non-mandatory way with the Tri-Council Policy Statement could help produce a knowledge set more useful to social science researchers.
- There are currently linkages between the CIHR Document and the Tri-Council Policy Statement. Although a small minority of researchers would like to see the PBPs become part of the Tri-Council policy, a substantial number felt that linkage was sufficient. The links from Tri-Council to the Document would usefully provide guidance and clarification for some of ambiguous, less specific, portions of the TCPS.
- The Document is recognized as one of many influences, policies, and laws in place in the jurisdictions in which the researchers work. Many would like to see the CIHR document linked to the Tri-council but they see it as a voluntary, not mandatory, adjunct. Most respondents liked the idea of linkage between the PBPs Document and the Tri-Council Ethical Conduct for Research Involving Humans policy statement, because the PBPs Document provides more detailed, clear, examples.
- The majority of respondents are associated with health research. The typical place of employment is a public institution employing researchers, such as a university, college, hospital or public service office. The roles played are, in large part, that of health researcher, research screener, research manager/coordinator or research advisor.
- The cultural and practical differences between social and biomedical researchers appear to call for different kinds of privacy policies and methods.
- There is a preference for paper copies of the document. Researchers and reviewers like to have their own hard-copy handy - a copy on which they have entered their personal notations.
- Several persons complained that they had difficulty obtaining paper copies of the document. Since many current users refer to the paper document, especially at meetings, the paper version is seen as much more useful than an on line version.
- The continuing use and development of electronic databases raised a host of security issues common to the public focus found in other (non-health) sectors such as banking and commercial enterprises. The continuing interconnection of databases to permit rapid access by health professionals was also seen as a requirement for adequate electronic data security.
- Some respondents are concerned that ethics and privacy are reviewed only in the context of research proposals, i.e. health research that is seeking funding. They worry that a lot of confidential information is on the unregistered "rogue databases" of individual researchers' laptops and therefore at risk in various ways.
- There are concerns about monitoring and auditing: do researchers put their professed safeguards of privacy into practice? This is both a policy issue (should it be done? if so, by whom?) and a practical one (where are the resources to do it?).
The findings for each of the seven groups are issued as stand-alone appendices. Where a pair of documents has been issued for a particular data set, the two documents are the report of the findings from the answers given and the other is the verbatim record of the answers given to certain of the free response questions.
The appendices are:
Document-Recipients: Interview Findings
Peer Review Chairs: Interview Findings
Peer Review Chairs: Text Responses
Ethics Community Stakeholders: Interview Findings
Ethics Community Stakeholders: Text Responses
REB Representatives: Interview Findings
REB Representatives: Text Responses
Focus Groups Report
Web Survey Report: Findings
Web Survey Report: Text Responses
CIHR Note: These Appendices are available on request to the CIHR Ethics Office. These documents are currently in English only.
- CIHR reports that there were addition recipients of the document. About 60 people (of the 60, many were invited to be interviewed and 8 agreed to be interviewed) asked for hard copies from the Ethics Office and other persons picked up copies on display at a range of different venues.
- The conditions are, essentially: a defined population; the specification of a frame for that population; the selection of a representative sample from the frame; and the obtaining of an adequate response rate from the representative sample. The ideal for a representative sample is a sample whose members are drawn at random from the frame. By adequate response rate is meant a sufficiently high percentage of responses such that the influence of the non-responses is not unacceptable in terms of its impact upon the confidence intervals for the findings.
- Here, as in many other of the respondent comments, it appears that many, if not most, researchers and PRC and REB Chairs, do not realize that many of the issues they raised during the interviews are already well treated by the Document. Notwithstanding this, the issue of approaching individuals to obtain consent is a problem for researchers, and therefore there is value to reviewing this requirement.
- The comment was received in this general nature. The suggestion is that CIHR consider how best to improve data security in a rapidly changing data storage environment.
- These were the suggestions offered. No one suggestion for how to make the information available was dominant.
- Date modified: