|
We sent the following letter to Dr. McInnes, interim president of the Canadian Institutes of Health Research, in response to the announcement of a delay to the upcoming Project Grant cycle. [May 7: edited from original to list only primary affiliations in signature] 
Our national co-chairs are meeting with the International Peer Review Expert Panel tomorrow. Here is our brief report without appendices:
And the version with appendices:
One of our national co-chairs, Holly Witteman, prepared a document this summer for the CIHR Working Group about funding equity. She has since updated the document to include newly available data and removed the sections that were specific to the Working Group. This 4-page document is freely available here:
Given the importance of the Project Scheme competition to health researchers across career stages, we were pleased to have our national co-chairs invited to attend the rapidly-organized CIHR Summit in Ottawa on July 13, 2016 along with 50-60 other health researchers to discuss how to improve the peer review process in time for the Fall Project Scheme cycle. Two of our three co-chairs (Dr. Kristin Connor, Dr. Holly Witteman) were able to attend, as was ACECHR member Dr. Meghan Azad. In preparation for that meeting, we prepared a document based on wide consultation with all engaged ACECHR members and other early career investigators, using our Slack team, Facebook group, email lists of those who had participated in our survey, and other methods to reach out to as many early career investigators across Canada as possible. In our final consensus document, we recommended similar points as those that we had recommended previously, specifically: To address the disadvantages affecting ECIs, we recommend three immediate changes, before the next grant cycle: 1. Equity: Enforced equity in grant funding rates across all career stages and by sex, with changes to the Canadian Common CV to help support this.
2. Opportunity: Flexible grant applications allow ECIs to demonstrate what reviewers expect to see. Structured forms are suboptimal for all and they are especially hard on ECIs, who no longer have the flexibility or space to demonstrate aspects of a grant that reviewers expect to see from ECIs.
3. Expert review: Reviews from people who have (a) expertise in our fields and (b) sufficient time to adequately review our grants and offer constructive feedback.
Following these immediate changes, we further recommend to CIHR: 4. Transparency and accountability: Greater transparency and accountability from CIHR and reviewers. 5. Funding balance: Foundation Scheme restructured as a grant consolidation mechanism for those with 2+ or 3+ grants, in which researchers trade off a small amount of budget for greater stability and flexibility. To the federal government, we recommend: 6. Investment in health research: A return to more sustainable levels of health research funding in pursuit of (a) a healthy and stable health research enterprise, (b) young Canadians pursuing degrees and careers in health research and related fields, (c) a flourishing knowledge economy and (d) strong production of new knowledge and health benefits for Canadians. Our document and these recommendations were well-received at the meeting July 13, and we felt that the final recommendations reflected our priorities and the shared priorities of the Canadian health research community. Both Dr. Connor and Dr. Witteman were subsequently invited to serve on the Peer Review Working Group. Both attended the in-person working meeting in Ottawa on August 5, 2016, and one or both attended each teleconference, according to availability. Throughout this process, Drs. Connor and Witteman reported back to our Association and consulted with our members about proposals. CIHR posted the Chair's final message along with the final recommendations of the working group today. We are pleased with many of the recommendations. Specifically, the final recommendations reflect our recommendations 2 and 3 from our CIHR Summit recommendations report, and they partially reflect recommendation 1. We are pleased about the Working Group’s final recommendations for: Equalization for ECIs. Equalized success rates in the Project Competition for ECI applicants such that ECI success rates match those of the overall competition. The additional 30 million dollars that was allocated to CIHR by the federal government and that was planned to be used “with a focus on early career investigators” will help to pay for this. We support this use of those funds. Some CCV flexibility. A partial solution to address the inequities within the Canadian Common CV (CCV) by allowing those who took a leave from research during the time limits of the CCV to append a file detailing an equivalent amount of time’s worth of productivity. Format flexibility. An open format application, with unlimited references and letters of support. Expert reviewers. Greater likelihood of expert reviewers by having Chairs and Scientific Officers (SOs) verifying and/ or selecting reviewer assignment and by allowing more options for applicants to specify the types of expertise required to review their application. We recommend to all applicants that they select from those lists (Descriptors, Themes, Suggested Institutes, Areas of Science, Methods/Approaches, and Study Populations and Experimental Systems) carefully, and use the Other option as needed. These selections may help to determine which reviewers will be assigned your grant. Helpful reviews. Greater likelihood of helpful Stage 1 reviews. We hope that the unstructured review format combined with reviewers’ knowledge that even if they don’t go to the face-to-face meeting, their review--with their name attached--will be reviewed by the Chair/SO and may well be presented at a face-to-face meeting by another peer reviewer. An ability to respond to reviewers more quickly. Although it can’t be put in place for the Fall cycle, the idea to allow a 1-page rebuttal/response between Stage 1 (triage) and Stage 2 (face-to-face) may help ECIs iterate their grant applications into the funded pile more quickly. We will be watching to see: Transparency in results. We recommended to CIHR that this time, when they provide applicants with their results, that they provide full data, including the raw scores from each reviewer, the distribution of scores that that reviewer assigned to all of her/his applications, the rank assigned by each reviewer, and how many grants the reviewer had in her/his pile. Transparency in methods. We further recommended that CIHR be fully transparent about how scoring and ranking works. Transparency in data. CIHR has told us they are working on a data access strategy. We look forward to seeing this promise enacted so that the research community can perform their own analyses. Transparency in analysis. The Working Group recommended that CIHR undertake analyses to evaluate recommendations for which the Working Group lacked data, that CIHR share the results with the Canadian health research community, and that CIHR make changes as required to address any problems identified in those analyses. Transparency in reviewers. The Working Group recommended that, immediately after each grant cycle, CIHR post the lists of clusters (unnamed) along with the names of reviewers in each cluster. Appropriateness of reviewers. The Working Group discussed ideas such as allowing applicants to indicate which cluster(s) from the previous year might have been most appropriate to review the proposal they are submitting now. This will not be immediately possible due to the fact that the dynamic clusters (dynamic panels) are being used for the first time this fall. Members of the Working Group and CIHR staff also raised concerns about setting expectations among applicants that their application will be assigned to the selected cluster, which may not be possible for a variety of reasons, including that the clusters are expected to change somewhat from cycle to cycle, in response to the applications submitted. We will be watching to see what strategies CIHR uses to ensure that applications are assigned to the most appropriate reviewers and we encourage the use of applicant preference as an element in that strategy. [We edited this post on Sep 13, 2016 at 14:45 to add this paragraph, which was inadvertently left out when we drafted the post.] ECI participation in peer review. Many Canadian ECIs hold CIHR funds and/or have experience reviewing grants. We would like to see ECIs recruited to the College of Reviewers as soon as possible so that ECIs can both serve the community by providing expert review and can also benefit from the learning and networking aspects of grant review. If ECIs are going to be invited to observe a face-to-face meeting, we’d like to see this start in the Fall cycle. Time allowed for between-stage response/rebuttal. We remind CIHR that the Working Group discussed the need to set the between-stage response/rebuttal dates well in advance and to provide a minimum of 3 weeks. Although some suggested that 48 hours would be enough, we believe strongly that this disadvantages people who do certain types of research (e.g., field work in areas with intermittent Internet access), who have heavy teaching loads, who have family responsibilities, and so on. Three weeks is a much more reasonable time limit. Equity. Equity is important because when there is systemic or structural bias in grant review, the best research is not being funded. CIHR and the College of Reviewers are charged with continuing to pursue fair, equitable grant review and allocation. We didn’t get as far on this point as we wanted during the Working Group (see below) but the Working Group recommended that CIHR, the College of Reviewers and others continue this critical conversation. We remind these people that there may well be other dimensions on which to consider equity that are not yet well-captured by CIHR (e.g., race, ethnicity, disability). We are pleased to see the efforts of the Appropriate Review Practices Reference Group on Indigenous Health Research at CIHR. We highlight that the Ginther report in the US offers relevant findings regarding grants submitted to the National Institutes of Health by people of colour. We don’t know whether or not similar issues as those identified in the Ginther report exist in Canada because CIHR is not collecting those important data. Canadians deserve to know whether or not public funds are being used in a way that respects the Canadian Human Rights Act, and that if they are not, that prompt action will be taken to remedy problems. We note here that although we had initially recommended a mechanism to ensure sex/gender equity, data shared by CIHR suggested that the Project Scheme demonstrated parity by sex of nominated principal applicant, meaning that there was no drop-off in success rates for female researchers at any career stage. This further reinforces the hypothesis that the poor success rates for mid-career and senior women in the Foundation Scheme is a function of the gendered structure of that competition, particularly Stage 1, in which 75% or more of the score is from criteria (leadership, significance of contributions, productivity) for which there is empirical evidence of gender bias in other contexts. Imbalances in Canada Excellence Research Chairs and Canada Research Chairs (CRC) also need to be addressed. We offer a suggestion that if CIHR made allocation of Chairs conditional not only on the current criteria such as that around research funding obtained by institutions, but also on whether or not institutions become signatories to or earn awards from the Athena SWAN program, the problems faced by women researchers in Canada would be resolved far more quickly than they are progressing now. We are disappointed: Circular definition of excellence. We continue to be disappointed by the misuse of the term excellence in discussions with CIHR. When one defines "excellence" as "ranks highly in the system we designed", one cannot then make claims that the system one designed is good at identifying excellence. We are confident that all awardees deserved the grants they received, but this does not mean that those who were not funded were not excellent. They simply did not rank as highly in this system. It is worth noting that evidence from the National Institutes of Health suggests that there may be no meaningful difference in productivity between grants ranked in the 3rd to the 20th percentile, underlining that, outside of the few applications at the very top of the pile, peer reviewers may be unable to predict which projects are going to be most impactful. No mid-career equalization. We have consistently recommended and advocated for equitable funding across career stages, including MCIs. Funding data suggest that ECIs and MCIs are disadvantaged in the new system, and we have recommended corrections accordingly. Our continued recommendation for such equity across all career stages may be seen in all of our policy and other documents of April 24, 2016, May 23, 2016, July 4, 2016, July 12, 2016, and in the draft equity policy document that we distributed to the Peer Review Working Group by email in advance of our call Aug 25, 2016:
Unfortunately, the working group as a whole did not agree. The working group discussed the health research community's potential discomfort with providing a possible ‘boost’ to people 15 years into their faculty career. We offered as an option that there could be equalization for the group of investigators 5-10 years in. This would align this equalization mechanism with the time limits for Tier 2 Canada Research Chairs and with the CIHR Gold Leaf award for New Investigators, both of which list a researcher in the first 10 years of their appointment as being eligible. However, this recommendation on our part was not endorsed by the Working Group. We are very disappointed that equalization will extend only to those who are between zero and five years into their independent careers. We are very concerned that mid-career investigators will be significantly disadvantaged in the next competition. We remind the leadership at CIHR and the College of Reviewers that someone who started her faculty position in 2011 began her career in a time that included two cancelled competitions and implementation of untested programs, all at a time of ever-lower grant success rates. Although nearly all Canadian health researchers are feeling the pain of low funding, those who are currently MCIs, particular those in years 5-10, have not had the same opportunities to launch their careers as those further ahead. We will not stand by and watch this cohort of excellent researchers be disadvantaged, simply because they happened to start their positions in a time of significant change. We also further remind leadership at CIHR and the College of Reviewers that at the meeting July 13, there was strong support for providing additional funding to both ECIs and MCIs. Attendees in the room expressed discomfort with the idea of providing a ‘boost’ to those nearly 15 years into their faculty appointment; however, after some discussion, there was agreement that providing additional funding to those 0-5 years into their career (ECIs) and those 5-12 years into their career (MCIs) would help to address some of the disadvantages faced by researchers in these groups. Senior members of the Peer Review Working Group described periods in which more junior investigators consistently had approximately a 5% higher success rate, simply through the culture of the panels. For example, when the overall success rate was 25%, for younger investigators it would end up being 30%. When the overall rate was 30%, for younger investigators it would be about 35%. (These are the actual example numbers, reflecting success rates at that time, used by the senior scientists.) Finally, we remind administrators at institutions that, when it comes to health research, assistant professors, researchers and scientists are facing the most challenging funding climate in memory. Institutions should carefully consider their criteria for tenure, contract renewal, extension of start-up funds, and other policies in light of the current funding situation:  Want to make your own images? Here are templates for you to use. Images: If you want to add an image, make sure you use an image that is shareable. There are lots of sites that offer images that are in the public domain, sometimes referred to as CC0. (Creative Commons 0.) The images we used in our campaign are all from Pixabay. If you use that site, make sure you don't grab one of the Shutterstock images by mistake. You can tell because the Shutterstock images have a watermark. Here is a list of other places to find images in the public domain, courtesy of the Harvard Law School Library. How do I edit these? Use an image editor if you have one, or use whatever program you use to mark up and edit pdfs. Hashtag: When you post these on Twitter, Facebook or elsewhere, don't forget to add the hashtag #InvestInCanadianHealthResearch. Tags: Consider tagging relevant political leaders, e.g., @JustinTrudeau, @janephilpott, @ScienceMin @Bill_Morneau. Reason to Invest template, short version (has less vertical space)
Reason to Invest template, tall version (has more vertical space)
Two of our national co-chairs will be attending CIHR's working meeting on Wednesday, July 13, 2016. In advance of the meeting, we prepared a report of early career investigator (ECI) concerns and recommendations regarding CIHR peer review and funding.
We (ACECHR national officers Kristin Connor, Michael Hendricks & Holly Witteman) had a collegial and productive call with CIHR representatives Michel Perron and Jane Aubin on July 4. CIHR will be releasing more information in the coming days and weeks, but we are able to share the following information. All paragraphs that begin with "CIHR’s response:" have been reviewed and, if necessary, edited by CIHR representatives. We are posting their edited version almost exactly as written by them, having corrected only a few typos.
What we asked for Urgent action. We continue to emphasize to CIHR that early career investigators (ECIs)--along with mid-career researchers and senior researchers facing funding gaps--cannot wait for the results of a pilot to be analyzed and plans made for adjustments to the process. People’s careers, laboratories, staff, and trainees are in jeopardy now. CIHR’s response: They are planning to use the additional $30M to help address critical gaps in funding. They indicated they too will monitor for any potential systemic bias in peer review (including potential bias against early career investigators, women, etc.) once the data are in from the current competitions. They indicated that there was no formal structural interventions in the past for these groups and that the competitions ‘self regulated’. They indicated that there is mention of this in their peer review manual and a new training module on gender bias. They also indicated that addressing all discrepancies through funding adjustments has the potential to become very complicated. There are many groups that may require attention, e.g., Aboriginal research, linguistic minorities, etc. Clarity about use of $30M. We asked (a) whether they planned to use it all this year or spread it over competitions, (b) whether it will be allocated only to ECIs (as per wording “focus on” ECIs) or will also include gapped mid-career investigators (as per our recommendation), whether they (c) if they have a target for a ratio of full grants to bridge grants. CIHR’s response: Overall, they restated the commitment of allocating the $30M towards early career investigators but to also implement these funds in order to allow for a sustainable competition funding over time (in other words not all at once as that would disadvantage ECI’s in future competitions since the $30M would be fully assigned immediately). As a result any available funds could be used to address any imbalances this competition which wont be known until final results are in and they don’t yet know what that might require, if it’s necessary. They did say that t (b) it will include other gapped investigators, and (c) there are no specific targets for full:bridge. Access to data & algorithms. We raised the issue that when applicants don’t trust the system, it creates many problems. We asked for more responsiveness to researcher concerns and more transparency. Regarding the latter point, we are specifically asking for (a) access to raw, anonymized data to conduct analyses and (b) transparency around how funding decisions are made, currently including the mathematical formulae and/or normalization algorithm detailing how consolidated ranks are calculated when different reviewers have different numbers of applications in their piles. CIHR’s responses: (a) They are committed to open data – and as soon as practically possible - but also need to comply with privacy legislation and ensure that data are appropriately anonymized. They are working on coming up with a timeline for making data freely available. (b) They will provide this algorithm (but did not specify a time frame.) Career development opportunities for ECIs. Participating in or viewing peer review panels has traditionally been recommended to ECIs as a method to learn about peer review. What sorts of opportunities will ECIs be offered going forward to learn more about peer review? CIHR’s response: They do not currently have a plan for allowing ECIs to observe peer review, but are considering avenues like filming a mock review. What is the purpose of peer review? To clarify possible sources of disagreement, we asked them, in their view, what is the purpose of peer review? We raised this in the context of career development activities like sitting on peer review panels and receiving expert feedback. CIHR’s response: As with other funding agencies, the fundamental purpose of peer review is to rank applications for funding. The other aspects are secondary. CIHR hopes that applicants are getting expert feedback prior to submission. (We did not have a chance to respond to this, though we would argue that while this is certainly very important and of course ECIs and others do this, colleagues and others can only offer so much, and even the most senior colleagues cannot always specify what might tip an application over the funding line in one direction or another, particularly in a changing funding environment.) We also sent a follow-up email suggesting that CIHR include the following 2 questions in the project scheme survey: To add to surveys 1.[for applicants] Based on the reviews I received, I believe that all my assigned reviewers had sufficient expertise to review my application. [for reviewers] I believe I had sufficient expertise to review all the applications I was assigned. [for virtual chairs] I believe all the reviewers within my virtual panel had sufficient expertise to review the applications they were assigned. [Likert, anchors: strongly disagree, strongly agree] (optional) Comments: 2.I believe the Project Scheme First Live Pilot funded the best applications. [Likert, anchors: strongly disagree, strongly agree] (optional) Comments: We expect that those whose applications are funded are more likely to respond positively; however, applicants' views will still offer useful insights one way or the other and reviewers’ views and virtual chairs’ views are likely to be very informative. As we promised to our members and also to CIHR, we are posting the response from CIHR officials to our most recent letter. The response can be seen in full below. It references our letter, as well as concerns raised by one of our members about equity between male and female applicants to the Foundation Scheme. The member tweeted these concerns on May 24, asked the CIHR representatives about any actions the CIHR might be taking, and also submitted a rapid response to the paper referenced in the letter below.
Thanks to support from HealthCareCAN, eleven members of our association were able to meet with representatives from the Canadian Institutes of Health Research (CIHR) as well as a representative from the office of the Minister of Science in Ottawa on May 4, 2016. We had good, frank conversations. After follow-up consultations with other members, we summarized our position in a letter to these same CIHR representatives, who welcomed the letter and confirmed that they will respond. We will post their response here when we receive it.
May 23, 2016 Follow-up letter after our meeting with CIHR representatives May 4, 2016 Members of the Association of Canadian Early Career Health Researchers (ACECHR) had the opportunity to meet with Canadian Institutes of Health Research (CIHR) representatives in Ottawa on May 4, 2016, to discuss our concerns about current and future funding programs. We appreciate the time these representatives spent with us, as well as the information they shared. In light of this conversation, we would like to take this opportunity to summarize our view on the current Project Scheme competition. Most importantly, we reiterate the urgency of our concerns. CIHR is running pilot programs, but these are not our pilot careers. We do not have years to wait for retrospective analyses on the impact of competition reforms on funding allocation. The careers and research programs of competitive early career researchers are in immediate danger of becoming collateral damage of these reforms. It appears that CIHR does not currently have sufficient protective measures in place to mitigate such potential unintended consequences. 1. The approximately equal success rates across career stages in the former Open Operating Grants Program (OOGP) were an extremely positive feature. Not only did such equality allow CIHR to meet Objective 4(j) of the CIHR Act[1] but, as recently recruited faculty, we can attest that perceptions of fairness and opportunity for early-career investigator (ECI) funding success was an important recruiting tool for Canadian universities. This feature of the OOGP was a product of the specific culture of the face-to-face review panels—it was not formally stated in any peer review instructions or criteria. We are concerned that this history of equitable success rates across career stages may be lost in the Project Scheme. We propose a mechanism that ensures equal success rates across career stages for the Project Scheme. Failure to have such a mechanism in place has a potentially devastating impact on ECI applicants. The implementation of such a mechanism is simple and requires no changes to the review process. We propose that this mechanism be used if needed. In other words, if the Project Scheme success rates are equivalent across career stages relative to the number of unique nominated principal applicants (NPAs) at each career stage, the mechanism would not be used. If, however, success rates are unbalanced across career stages, we propose that success rates be adjusted such that they are proportional to the number of unique applicants. For example, if 18% of unique NPAs are ECIs, then ECIs should receive 18% of grants awarded. Similar measures to ensure New Investigator[2] success were implemented by the National Institutes of Health in 2009 and successfully lowered entry barriers at a time when funding success rates had dropped for New Investigators to a “dismal” 15 percent[3]. 2. The “new” $30 million allocated to the Project Scheme should be targeted to those most negatively affected by the reforms. This cohort includes ECIs, as well as non-ECI researchers affected by aspects of the reforms such as cancelled competitions. Therefore, we recommend that the following priorities guide the use of these new funds: a. The entire $30M should be applied to the first Project Scheme competition (March 2016), with an emphasis on full awards rather than bridge grants. We suggest an approximately 75:25 ratio of spending on full awards versus bridge funds. b. Eligibility should be restricted to those applicants who have never held CIHR operating grants as NPAs, or are facing an imminent funding gap, and whose current application is not already in the fundable range of the first Project Grant Pilot. In other words, these awards should be in addition to the success rate equalization described in Point 1 above, intended to “rescue” the early-career and mid-career investigators who have fallen through the cracks during the transition period. We believe the measures outlined in this document are essential for CIHR to fulfill its mandate to build capacity as well as to meet its stated intention of maintaining a similarly-sized population of CIHR-funded researchers in the new programs as in the OOGP. We look forward to working with CIHR to ensure sustainable funding for investigators who represent the future of health research, safeguarding Canada’s long-standing success in this domain. [1] “…(j) building the capacity of the Canadian health research community through the development of researchers and the provision of sustained support for scientific careers in health research;” [2] "... a PD/PI is identified as a New Investigator if he/she has not previously competed successfully for an NIH-supported research project other than the following early stage or small research grants or for the indicated training, infrastructure, and career awards." [3] “Affirmative action at the NIH” http://www.nature.com/neuro/journal/v12/n11/full/nn1109-1351.html The Executive Summary is below, while the comprehensive report is available in PDF format. Executive Summary Health research funding has become increasingly competitive in Canada. Between 2005-06 and 2014-15, success rates for full term open operating grants at the Canadian Institutes of Health Research (CIHR) halved from 30% to 15%. Grants deemed fundable by reviewers went from being funded about half the time to about one fifth of the time. Low funding rates undermine health researchers’ ability to generate new knowledge and improve Canadians’ health outcomes. These falling success rates have had negative impacts on many scientists. Early career investigators (ECIs), defined as those within the first 5 years of their independent careers, have been particularly hard hit. Between 2008-09 and 2014-15, CIHR funding awarded to ECIs declined by 38%. Adding to these concerns, recent changes to CIHR funding programs (‘CIHR reforms’) threaten to remove a full third of total funding awarded to ECIs annually. The Association of Canadian Early Career Health Researchers (ACECHR) organized and ran an informal survey Mar 17-24, 2016 to gather personal accounts from ECIs about their experiences in the current funding environment. In one week, we received 143 responses from verified early career health researchers in Canada who hold competitive positions and have a history of research success. Highlights of our findings include:
In respondents’ words: I am Canadian and always wanted to come back to Canada after my training but am starting to think that I have made a mistake. -Respondent 40 (page 37) I am falling behind scientists in other countries [...] I am deeply worried that this will end my career in research right as it is meant to be taking off. -Respondent 7 (page 42) There is a feeling among our trainees that there is no future for them in science in Canada. Some leave the country, and others decide to pursue other fields. -Respondent 113 (page 45) Some of my highest quality colleagues are leaving Canada because of the uncertain funding climate. I am considering the same. -Respondent 42 (page 55) If Canada wishes to foster the future of health research in our country, a recognition of this significant ECI disadvantage and a meaningful resolution must be reached immediately or the investment made to train today's most ambitious and successful young researchers in our country will be lost. Time is of the essence. -Respondent 41 (page 59) |
|||||||||||||||||||||||||||||||||||
RSS Feed
