Two of our national co-chairs will be attending CIHR's working meeting on Wednesday, July 13, 2016. In advance of the meeting, we prepared a report of early career investigator (ECI) concerns and recommendations regarding CIHR peer review and funding.

We (ACECHR national officers Kristin Connor, Michael Hendricks & Holly Witteman) had a collegial and productive call with CIHR representatives Michel Perron and Jane Aubin on July 4. CIHR will be releasing more information in the coming days and weeks, but we are able to share the following information. All paragraphs that begin with "CIHR’s response:" have been reviewed and, if necessary, edited by CIHR representatives. We are posting their edited version almost exactly as written by them, having corrected only a few typos.
 
What we asked for
 
Urgent action. We continue to emphasize to CIHR that early career investigators (ECIs)--along with mid-career researchers and senior researchers facing funding gaps--cannot wait for the results of a pilot to be analyzed and plans made for adjustments to the process. People’s careers, laboratories, staff, and trainees are in jeopardy now.
 
CIHR’s response: They are planning to use the additional $30M to help address critical gaps in funding. They indicated they too will monitor for any  potential systemic bias in peer review (including potential bias against early career investigators, women, etc.) once the data are in from the current competitions.  They indicated that there was no formal structural interventions in the past for these groups and that the competitions ‘self regulated’.  They indicated that there is mention of this in their peer review manual and a new training module on gender bias. They also indicated that addressing all discrepancies through funding adjustments has the potential to become very complicated. There are many groups that may require attention, e.g., Aboriginal research, linguistic minorities, etc.
 
Clarity about use of $30M. We asked (a) whether they planned to use it all this year or spread it over competitions, (b) whether it will be allocated only to ECIs (as per wording “focus on” ECIs) or will also include gapped mid-career investigators (as per our recommendation), whether they (c) if they have a target for a ratio of full grants to bridge grants.
 
CIHR’s response: Overall, they restated the commitment of allocating the $30M towards early career investigators but to also implement these funds in order to allow for a sustainable competition funding over time (in other words not all at once as that would disadvantage ECI’s in future competitions since the $30M would be fully assigned immediately). As a result any available funds could be used to address any imbalances this competition which wont be known until final results are in and they don’t yet know what that might require, if it’s necessary. They did say that  t (b) it will include other gapped investigators, and (c) there are no specific targets for full:bridge.
 
Access to data & algorithms. We raised the issue that when applicants don’t trust the system, it creates many problems. We asked for more responsiveness to researcher concerns and more transparency. Regarding the latter point, we are specifically asking for (a) access to raw, anonymized data to conduct analyses and (b) transparency around how funding decisions are made, currently including the mathematical formulae and/or normalization algorithm detailing how consolidated ranks are calculated when different reviewers have different numbers of applications in their piles.
 
CIHR’s responses: (a) They are committed to open data – and as soon as practically possible - but also need to comply with privacy legislation and ensure that data are appropriately anonymized. They are working on coming up with a timeline for making data freely available. (b) They will provide this algorithm (but did not specify a time frame.)
 
Career development opportunities for ECIs. Participating in or viewing peer review panels has traditionally been recommended to ECIs as a method to learn about peer review. What sorts of opportunities will ECIs be offered going forward to learn more about peer review?
 
CIHR’s response: They do not currently have a plan for allowing ECIs to observe peer review, but are considering avenues like filming a mock review.
 
What is the purpose of peer review? To clarify possible sources of disagreement, we asked them, in their view, what is the purpose of peer review? We raised this in the context of career development activities like sitting on peer review panels and receiving expert feedback.
 
CIHR’s response: As with other funding agencies, the fundamental purpose of peer review is to rank applications for funding. The other aspects are secondary. CIHR hopes that applicants are getting expert feedback prior to submission. (We did not have a chance to respond to this, though we would argue that while this is certainly very important and of course ECIs and others do this, colleagues and others can only offer so much, and even the most senior colleagues cannot always specify what might tip an application over the funding line in one direction or another, particularly in a changing funding environment.)
 
We also sent a follow-up email suggesting that CIHR include the following 2 questions in the project scheme survey:
 
To add to surveys
 
1.[for applicants] Based on the reviews I received, I believe that all my assigned reviewers had sufficient expertise to review my application.
[for reviewers] I believe I had sufficient expertise to review all the applications I was assigned.
[for virtual chairs] I believe all the reviewers within my virtual panel had sufficient expertise to review the applications they were assigned.
[Likert, anchors: strongly disagree, strongly agree]
(optional) Comments:


2.I believe the Project Scheme First Live Pilot funded the best applications.
[Likert, anchors: strongly disagree, strongly agree]
(optional) Comments:
 
We expect that those whose applications are funded are more likely to respond positively; however, applicants' views will still offer useful insights one way or the other and reviewers’ views and virtual chairs’ views are likely to be very informative.

 As we promised to our members and also to CIHR, we are posting the response from CIHR officials to our most recent letter.

The response can be seen in full below. It references our letter, as well as concerns raised by one of our members about equity between male and female applicants to the Foundation Scheme. The member tweeted these concerns on May 24, asked the CIHR representatives about any actions the CIHR might be taking, and also submitted a rapid response to the paper referenced in the letter below.

Thanks to support from HealthCareCAN, eleven members of our association were able to meet with representatives from the Canadian Institutes of Health Research (CIHR) as well as a representative from the office of the Minister of Science in Ottawa on May 4, 2016. We had good, frank conversations. After follow-up consultations with other members, we summarized our position in a letter to these same CIHR representatives, who welcomed the letter and confirmed that they will respond. We will post their response here when we receive it.

May 23, 2016
 
Follow-up letter after our meeting with CIHR representatives May 4, 2016
 
Members of the Association of Canadian Early Career Health Researchers (ACECHR) had the opportunity to meet with Canadian Institutes of Health Research (CIHR) representatives in Ottawa on May 4, 2016, to discuss our concerns about current and future funding programs. We appreciate the time these representatives spent with us, as well as the information they shared. In light of this conversation, we would like to take this opportunity to summarize our view on the current Project Scheme competition.
 
Most importantly, we reiterate the urgency of our concerns. CIHR is running pilot programs, but these are not our pilot careers. We do not have years to wait for retrospective analyses on the impact of competition reforms on funding allocation. The careers and research programs of competitive early career researchers are in immediate danger of becoming collateral damage of these reforms. It appears that CIHR does not currently have sufficient protective measures in place to mitigate such potential unintended consequences.
 
1. The approximately equal success rates across career stages in the former Open Operating Grants Program (OOGP) were an extremely positive feature. Not only did such equality allow CIHR to meet Objective 4(j) of the CIHR Act[1] but, as recently recruited faculty, we can attest that perceptions of fairness and opportunity for early-career investigator (ECI) funding success was an important recruiting tool for Canadian universities. This feature of the OOGP was a product of the specific culture of the face-to-face review panels—it was not formally stated in any peer review instructions or criteria. We are concerned that this history of equitable success rates across career stages may be lost in the Project Scheme. We propose a mechanism that ensures equal success rates across career stages for the Project Scheme. Failure to have such a mechanism in place has a potentially devastating impact on ECI applicants. The implementation of such a mechanism is simple and requires no changes to the review process.

We propose that this mechanism be used if needed. In other words, if the Project Scheme success rates are equivalent across career stages relative to the number of unique nominated principal applicants (NPAs) at each career stage, the mechanism would not be used. If, however, success rates are unbalanced across career stages, we propose that success rates be adjusted such that they are proportional to the number of unique applicants. For example, if 18% of unique NPAs are ECIs, then ECIs should receive 18% of grants awarded. Similar measures to ensure New Investigator[2] success were implemented by the National Institutes of Health in 2009 and successfully lowered entry barriers at a time when funding success rates had dropped for New Investigators to a “dismal” 15 percent[3].
 
2. The “new” $30 million allocated to the Project Scheme should be targeted to those most negatively affected by the reforms. This cohort includes ECIs, as well as non-ECI researchers affected by aspects of the reforms such as cancelled competitions. Therefore, we recommend that the following priorities guide the use of these new funds:

a. The entire $30M should be applied to the first Project Scheme competition (March 2016), with an emphasis on full awards rather than bridge grants. We suggest an approximately 75:25 ratio of spending on full awards versus bridge funds.

b. Eligibility should be restricted to those applicants who have never held CIHR operating grants as NPAs, or are facing an imminent funding gap, and whose current application is not already in the fundable range of the first Project Grant Pilot. In other words, these awards should be in addition to the success rate equalization described in Point 1 above, intended to “rescue” the early-career and mid-career investigators who have fallen through the cracks during the transition period.
 
We believe the measures outlined in this document are essential for CIHR to fulfill its mandate to build capacity as well as to meet its stated intention of maintaining a similarly-sized population of CIHR-funded researchers in the new programs as in the OOGP.
 
We look forward to working with CIHR to ensure sustainable funding for investigators who represent the future of health research, safeguarding Canada’s long-standing success in this domain.


[1] “…(j) building the capacity of the Canadian health research community through the development of researchers and the provision of sustained support for scientific careers in health research;”

[2] "... a PD/PI is identified as a New Investigator if he/she has not previously competed successfully for an NIH-supported research project other than the following early stage or small research grants or for the indicated training, infrastructure, and career awards."

[3] “Affirmative action at the NIH” http://www.nature.com/neuro/journal/v12/n11/full/nn1109-1351.html

The Executive Summary is below, while the comprehensive report is available in PDF format. 

Executive Summary
Health research funding has become increasingly competitive in Canada. Between 2005-06 and 2014-15, success rates for full term open operating grants at the Canadian Institutes of Health Research (CIHR) halved from 30% to 15%. Grants deemed fundable by reviewers went from being funded about half the time to about one fifth of the time. Low funding rates undermine health researchers’ ability to generate new knowledge and improve Canadians’ health outcomes.

These falling success rates have had negative impacts on many scientists. Early career investigators (ECIs), defined as those within the first 5 years of their independent careers, have been particularly hard hit. Between 2008-09 and 2014-15, CIHR funding awarded to ECIs declined by 38%. Adding to these concerns, recent changes to CIHR funding programs (‘CIHR reforms’) threaten to remove a full third of total funding awarded to ECIs annually.

The Association of Canadian Early Career Health Researchers (ACECHR) organized and ran an informal survey Mar 17-24, 2016 to gather personal accounts from ECIs about their experiences in the current funding environment. In one week, we received 143 responses from verified early career health researchers in Canada who hold competitive positions and have a history of research success. Highlights of our findings include:

• 84% report that they are delaying starting potentially impactful research. Respondents report scaling back their research and losing competitiveness internationally.
• In total, respondents currently employ 204 staff and supervise 909 trainees but many report firing staff and being unable to fund their trainees or accept new trainees.
• 46%indicate that because of the current funding environment, they are considering leaving research, academia, or Canada.

In respondents’ words:

I am Canadian and always wanted to come back to Canada after my training but am starting to think that I have made a mistake. -Respondent 40 (page 37)

I am falling behind scientists in other countries [...] I am deeply worried that this will end my career in research right as it is meant to be taking off. -Respondent 7 (page 42)

There is a feeling among our trainees that there is no future for them in science in Canada. Some leave the country, and others decide to pursue other fields. -Respondent 113 (page 45)

Some of my highest quality colleagues are leaving Canada because of the uncertain funding climate. I am considering the same. -Respondent 42 (page 55)

If Canada wishes to foster the future of health research in our country, a recognition of this significant ECI disadvantage and a meaningful resolution must be reached immediately or the investment made to train today's most ambitious and successful young researchers in our country will be lost. Time is of the essence. -Respondent 41 (page 59)

​A PDF of the letter in French and in English is available here.

“Early-career investigators” (ECIs) are typically assistant professors at research universities and hospitals who have held their positions for 5 years or less. Like many other researchers in Canada, ECIs rely on CIHR for grant funding. However, increased competition, declining success rates, and recent changes to the grant programs at CIHR have severely disadvantaged early-career scientists. The Canadian Foundation for Innovation, the Canada Research Chairs program, and universities invest millions each year to recruit these scientists and equip their labs. These investments are squandered when ECIs are not given adequate opportunities to obtain funding to operate their labs and conduct research.

Under the previous CIHR grant system, ECIs received approximately 18% of the funds. Starting in 2015, however, CIHR transferred 45% of its grants budget to the new Foundation Scheme. In the first year of the Foundation,ECIs received just 5% of the funds awarded. While a quota guaranteed that 15% of grants were awarded to ECIs, these awards were on average a third the size of those given to established researchers. CIHR is the only major health research agency in the world whose current granting system awards systematically smaller grants to early-career investigators.

The remaining 55% of the CIHR budget goes into the new Project Scheme, launching this year. At current budget levels and with increased application pressure, success rates will be at an all-time low. Even if ECIs do as well in the Project Scheme as in the old system (~18%), an optimistic assumption, low funding in Foundation means that CIHR will decrease overall support for ECIs by over 30%, from approximately $90 million per year to $60 million.

ECIs face obstacles at grant review due to a lack of documented experience compared to established investigators. In the previous system, CIHR review panels had a culture of actively promoting ECI success. These in-person panels have been disbanded in favour of an online system where reviewers are not required to consider career stage, which is likely to reduce ECI success in the Project Scheme.

SOLUTIONS: Most funding agencies like CIHR have compensatory mechanisms to ensure ECI success, yet CIHR’s Project Scheme has none. At the U.S. National Institutes of Health (NIH), for example, the “fundable score” cut-off is adjusted so that ECI success rates match those of other applicants. This simple solution does not require a special funding program or changes to the peer review process, and it has worked well since its implementation in 2009.

On behalf of current and future new investigators in Canadian health and biomedical research, the signatories of this letter request immediate action to fulfill CIHR’s legal mandate to build research capacity through equitable investment in the next generation of scientists:

1. Guarantee that ECI success rates match those of non-ECI applicants in the Project Scheme through an NIH-style adjustment to the fundable score for ECIs.
2. Set a minimum ECI quota in Foundation of 15% of funds rather than 15% of grants.
3. Renew Canada’s commitment to health research by bringing CIHR’s budget back to a level where it can support the growth of discovery and innovation in health and biomedical sciences.

These steps are essential for the recruitment and retention of the best young scientists in Canada and for the future of Canadian health research.

List of signatories