|
Given the importance of the Project Scheme competition to health researchers across career stages, we were pleased to have our national co-chairs invited to attend the rapidly-organized CIHR Summit in Ottawa on July 13, 2016 along with 50-60 other health researchers to discuss how to improve the peer review process in time for the Fall Project Scheme cycle. Two of our three co-chairs (Dr. Kristin Connor, Dr. Holly Witteman) were able to attend, as was ACECHR member Dr. Meghan Azad. In preparation for that meeting, we prepared a document based on wide consultation with all engaged ACECHR members and other early career investigators, using our Slack team, Facebook group, email lists of those who had participated in our survey, and other methods to reach out to as many early career investigators across Canada as possible. In our final consensus document, we recommended similar points as those that we had recommended previously, specifically: To address the disadvantages affecting ECIs, we recommend three immediate changes, before the next grant cycle: 1. Equity: Enforced equity in grant funding rates across all career stages and by sex, with changes to the Canadian Common CV to help support this.
2. Opportunity: Flexible grant applications allow ECIs to demonstrate what reviewers expect to see. Structured forms are suboptimal for all and they are especially hard on ECIs, who no longer have the flexibility or space to demonstrate aspects of a grant that reviewers expect to see from ECIs.
3. Expert review: Reviews from people who have (a) expertise in our fields and (b) sufficient time to adequately review our grants and offer constructive feedback.
Following these immediate changes, we further recommend to CIHR: 4. Transparency and accountability: Greater transparency and accountability from CIHR and reviewers. 5. Funding balance: Foundation Scheme restructured as a grant consolidation mechanism for those with 2+ or 3+ grants, in which researchers trade off a small amount of budget for greater stability and flexibility. To the federal government, we recommend: 6. Investment in health research: A return to more sustainable levels of health research funding in pursuit of (a) a healthy and stable health research enterprise, (b) young Canadians pursuing degrees and careers in health research and related fields, (c) a flourishing knowledge economy and (d) strong production of new knowledge and health benefits for Canadians. Our document and these recommendations were well-received at the meeting July 13, and we felt that the final recommendations reflected our priorities and the shared priorities of the Canadian health research community. Both Dr. Connor and Dr. Witteman were subsequently invited to serve on the Peer Review Working Group. Both attended the in-person working meeting in Ottawa on August 5, 2016, and one or both attended each teleconference, according to availability. Throughout this process, Drs. Connor and Witteman reported back to our Association and consulted with our members about proposals. CIHR posted the Chair's final message along with the final recommendations of the working group today. We are pleased with many of the recommendations. Specifically, the final recommendations reflect our recommendations 2 and 3 from our CIHR Summit recommendations report, and they partially reflect recommendation 1. We are pleased about the Working Group’s final recommendations for: Equalization for ECIs. Equalized success rates in the Project Competition for ECI applicants such that ECI success rates match those of the overall competition. The additional 30 million dollars that was allocated to CIHR by the federal government and that was planned to be used “with a focus on early career investigators” will help to pay for this. We support this use of those funds. Some CCV flexibility. A partial solution to address the inequities within the Canadian Common CV (CCV) by allowing those who took a leave from research during the time limits of the CCV to append a file detailing an equivalent amount of time’s worth of productivity. Format flexibility. An open format application, with unlimited references and letters of support. Expert reviewers. Greater likelihood of expert reviewers by having Chairs and Scientific Officers (SOs) verifying and/ or selecting reviewer assignment and by allowing more options for applicants to specify the types of expertise required to review their application. We recommend to all applicants that they select from those lists (Descriptors, Themes, Suggested Institutes, Areas of Science, Methods/Approaches, and Study Populations and Experimental Systems) carefully, and use the Other option as needed. These selections may help to determine which reviewers will be assigned your grant. Helpful reviews. Greater likelihood of helpful Stage 1 reviews. We hope that the unstructured review format combined with reviewers’ knowledge that even if they don’t go to the face-to-face meeting, their review--with their name attached--will be reviewed by the Chair/SO and may well be presented at a face-to-face meeting by another peer reviewer. An ability to respond to reviewers more quickly. Although it can’t be put in place for the Fall cycle, the idea to allow a 1-page rebuttal/response between Stage 1 (triage) and Stage 2 (face-to-face) may help ECIs iterate their grant applications into the funded pile more quickly. We will be watching to see: Transparency in results. We recommended to CIHR that this time, when they provide applicants with their results, that they provide full data, including the raw scores from each reviewer, the distribution of scores that that reviewer assigned to all of her/his applications, the rank assigned by each reviewer, and how many grants the reviewer had in her/his pile. Transparency in methods. We further recommended that CIHR be fully transparent about how scoring and ranking works. Transparency in data. CIHR has told us they are working on a data access strategy. We look forward to seeing this promise enacted so that the research community can perform their own analyses. Transparency in analysis. The Working Group recommended that CIHR undertake analyses to evaluate recommendations for which the Working Group lacked data, that CIHR share the results with the Canadian health research community, and that CIHR make changes as required to address any problems identified in those analyses. Transparency in reviewers. The Working Group recommended that, immediately after each grant cycle, CIHR post the lists of clusters (unnamed) along with the names of reviewers in each cluster. Appropriateness of reviewers. The Working Group discussed ideas such as allowing applicants to indicate which cluster(s) from the previous year might have been most appropriate to review the proposal they are submitting now. This will not be immediately possible due to the fact that the dynamic clusters (dynamic panels) are being used for the first time this fall. Members of the Working Group and CIHR staff also raised concerns about setting expectations among applicants that their application will be assigned to the selected cluster, which may not be possible for a variety of reasons, including that the clusters are expected to change somewhat from cycle to cycle, in response to the applications submitted. We will be watching to see what strategies CIHR uses to ensure that applications are assigned to the most appropriate reviewers and we encourage the use of applicant preference as an element in that strategy. [We edited this post on Sep 13, 2016 at 14:45 to add this paragraph, which was inadvertently left out when we drafted the post.] ECI participation in peer review. Many Canadian ECIs hold CIHR funds and/or have experience reviewing grants. We would like to see ECIs recruited to the College of Reviewers as soon as possible so that ECIs can both serve the community by providing expert review and can also benefit from the learning and networking aspects of grant review. If ECIs are going to be invited to observe a face-to-face meeting, we’d like to see this start in the Fall cycle. Time allowed for between-stage response/rebuttal. We remind CIHR that the Working Group discussed the need to set the between-stage response/rebuttal dates well in advance and to provide a minimum of 3 weeks. Although some suggested that 48 hours would be enough, we believe strongly that this disadvantages people who do certain types of research (e.g., field work in areas with intermittent Internet access), who have heavy teaching loads, who have family responsibilities, and so on. Three weeks is a much more reasonable time limit. Equity. Equity is important because when there is systemic or structural bias in grant review, the best research is not being funded. CIHR and the College of Reviewers are charged with continuing to pursue fair, equitable grant review and allocation. We didn’t get as far on this point as we wanted during the Working Group (see below) but the Working Group recommended that CIHR, the College of Reviewers and others continue this critical conversation. We remind these people that there may well be other dimensions on which to consider equity that are not yet well-captured by CIHR (e.g., race, ethnicity, disability). We are pleased to see the efforts of the Appropriate Review Practices Reference Group on Indigenous Health Research at CIHR. We highlight that the Ginther report in the US offers relevant findings regarding grants submitted to the National Institutes of Health by people of colour. We don’t know whether or not similar issues as those identified in the Ginther report exist in Canada because CIHR is not collecting those important data. Canadians deserve to know whether or not public funds are being used in a way that respects the Canadian Human Rights Act, and that if they are not, that prompt action will be taken to remedy problems. We note here that although we had initially recommended a mechanism to ensure sex/gender equity, data shared by CIHR suggested that the Project Scheme demonstrated parity by sex of nominated principal applicant, meaning that there was no drop-off in success rates for female researchers at any career stage. This further reinforces the hypothesis that the poor success rates for mid-career and senior women in the Foundation Scheme is a function of the gendered structure of that competition, particularly Stage 1, in which 75% or more of the score is from criteria (leadership, significance of contributions, productivity) for which there is empirical evidence of gender bias in other contexts. Imbalances in Canada Excellence Research Chairs and Canada Research Chairs (CRC) also need to be addressed. We offer a suggestion that if CIHR made allocation of Chairs conditional not only on the current criteria such as that around research funding obtained by institutions, but also on whether or not institutions become signatories to or earn awards from the Athena SWAN program, the problems faced by women researchers in Canada would be resolved far more quickly than they are progressing now. We are disappointed: Circular definition of excellence. We continue to be disappointed by the misuse of the term excellence in discussions with CIHR. When one defines "excellence" as "ranks highly in the system we designed", one cannot then make claims that the system one designed is good at identifying excellence. We are confident that all awardees deserved the grants they received, but this does not mean that those who were not funded were not excellent. They simply did not rank as highly in this system. It is worth noting that evidence from the National Institutes of Health suggests that there may be no meaningful difference in productivity between grants ranked in the 3rd to the 20th percentile, underlining that, outside of the few applications at the very top of the pile, peer reviewers may be unable to predict which projects are going to be most impactful. No mid-career equalization. We have consistently recommended and advocated for equitable funding across career stages, including MCIs. Funding data suggest that ECIs and MCIs are disadvantaged in the new system, and we have recommended corrections accordingly. Our continued recommendation for such equity across all career stages may be seen in all of our policy and other documents of April 24, 2016, May 23, 2016, July 4, 2016, July 12, 2016, and in the draft equity policy document that we distributed to the Peer Review Working Group by email in advance of our call Aug 25, 2016: 
Unfortunately, the working group as a whole did not agree. The working group discussed the health research community's potential discomfort with providing a possible ‘boost’ to people 15 years into their faculty career. We offered as an option that there could be equalization for the group of investigators 5-10 years in. This would align this equalization mechanism with the time limits for Tier 2 Canada Research Chairs and with the CIHR Gold Leaf award for New Investigators, both of which list a researcher in the first 10 years of their appointment as being eligible. However, this recommendation on our part was not endorsed by the Working Group. We are very disappointed that equalization will extend only to those who are between zero and five years into their independent careers. We are very concerned that mid-career investigators will be significantly disadvantaged in the next competition. We remind the leadership at CIHR and the College of Reviewers that someone who started her faculty position in 2011 began her career in a time that included two cancelled competitions and implementation of untested programs, all at a time of ever-lower grant success rates. Although nearly all Canadian health researchers are feeling the pain of low funding, those who are currently MCIs, particular those in years 5-10, have not had the same opportunities to launch their careers as those further ahead. We will not stand by and watch this cohort of excellent researchers be disadvantaged, simply because they happened to start their positions in a time of significant change. We also further remind leadership at CIHR and the College of Reviewers that at the meeting July 13, there was strong support for providing additional funding to both ECIs and MCIs. Attendees in the room expressed discomfort with the idea of providing a ‘boost’ to those nearly 15 years into their faculty appointment; however, after some discussion, there was agreement that providing additional funding to those 0-5 years into their career (ECIs) and those 5-12 years into their career (MCIs) would help to address some of the disadvantages faced by researchers in these groups. Senior members of the Peer Review Working Group described periods in which more junior investigators consistently had approximately a 5% higher success rate, simply through the culture of the panels. For example, when the overall success rate was 25%, for younger investigators it would end up being 30%. When the overall rate was 30%, for younger investigators it would be about 35%. (These are the actual example numbers, reflecting success rates at that time, used by the senior scientists.) Finally, we remind administrators at institutions that, when it comes to health research, assistant professors, researchers and scientists are facing the most challenging funding climate in memory. Institutions should carefully consider their criteria for tenure, contract renewal, extension of start-up funds, and other policies in light of the current funding situation:  Comments are closed.
|
|||
RSS Feed
