Evaluation of the Banting Postdoctoral Fellowships (PDF) Program
Final Evaluation Report

August 2024

At the Canadian Institutes of Health Research (CIHR), we know that research has the power to change lives. As Canada's health research investment agency, we collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen our health care system.

Canadian Institutes of Health Research
160 Elgin Street, 9th Floor
Address Locator 4809A
Ottawa, Ontario K1A 0W9

This publication was produced by the Canadian Institutes of Health Research. The views expressed herein do not necessarily reflect those of the Canadian Institutes of Health Research.

Acknowledgements

Special thanks to all participants in this evaluation through end of award reports, surveys, focus groups, and interviews, the Vanier-Banting PDF Secretariat, Goss Gilroy Inc., Observatoire des sciences et des technologies, SSHRC’s Corporate Analytics and NSERC’s Governance, Risk and Compliance units, CIHR’s Financial Planning Unit, and CIHR’s Funding Policy and Analytics Unit. Additional thanks to the members of the Banting PDF Evaluation Advisory Committee – Susan Morris (NSERC/SSHRC), Barbara Szijarto (NSERC/SSHRC), Murielle Vergnhes (NSERC), Anna Engman (SSHRC), François Zegers and Sean Daley (VBS), Judy Ettinger (Innovation, Science and Economic Development Canada – ISED).

The Banting PDF Evaluation Team

Angel Mackenzie, Alison Croke, Gedeon Djissa, Alice Ndayishimiye, Mauve Patterson (student), Jenny Larkin, Kwadwo (Nana) Bosompra, Jean-Christian Maillet, Michael Goodyer, and Sarah Connor Gorber.

For more information and to obtain copies, please contact: evaluation@cihr-irsc.gc.ca.

Table of Contents

List of Tables

List of Figures

List of Acronyms

Acronym Meaning
ACTE Administrative Costs as a Percentage of Total Expenditures
ARC Average Relative Citation
ARIF Average Relative Impact Factor
Banting PDF Banting Postdoctoral Fellowships
BEAR Banting PDF End of Award Report
B5 Banting PDF Five Year Follow-Up Survey
CAD Canadian Dollars
CIHR Canadian Institutes of Health Research
CRC Canada Research Chairs
CRCC Canada Research Coordinating Committee
EDI Equity, Diversity and Inclusion
GBP British Pound Sterling
NSERC Natural Sciences and Engineering Research Council of Canada
PDF Postdoctoral Fellowships
SGBA+ Sex and Gender Based Analysis Plus
SSHRC Social Sciences and Humanities Research Council of Canada
TAP-MC Tri-Agency Programs Management Committee
TAP-SC Tri-Agency Programs Steering Committee
TBS Treasury Board of Canada Secretariat
EUR Euro
USD U.S. Dollars
Vanier CGS Vanier Canadian Graduate Scholarship

Executive Summary

Program Overview

The Banting Postdoctoral Fellowships (Banting PDF) program was announced in the 2010 federal budget as part of a broader strategy to increase Canadian capacity for research excellence, and is jointly administered by the Tri-agencies: the Canadian Institutes of Health Research (CIHR) , the Natural Sciences and Engineering Research Council (NSERC), and the Social Sciences and Humanities Research Council (SSHRC). The objectives of the Banting PDF program are to: attract and retain top-tier postdoctoral talent, both nationally and internationally; develop their leadership potential; and position them for success as research leaders of tomorrow. Banting PDFs are of two years’ duration, with a value of $70,000 per year, and 70 new fellowships are awarded per year, distributed equally among the three agencies.

Evaluation Objective, Scope and Methodology

The objective of the evaluation was to provide Tri-agency management with valid, insightful and actionable findings regarding the needs addressed by the program, the effectiveness of the program design in supporting outcomes, and the achievement of expected results over the period from 2014-15 to 2020-21. This is the second evaluation of the program; the first evaluation was completed in 2015. Building on the first evaluation, this evaluation used multiple lines of evidence including analyses of documents, end of award reports and other administrative and financial data, environmental scan, surveys, key informant interviews, focus groups, and case studies. The evaluation meets the requirements of the Treasury Board Secretariat (TBS) of Canada under the Policy on Results and the Financial Administration Act.

Key Findings

Relevance

There is a continued need for postdoctoral support that aims to attract, retain, and support the training of top-tier Canadian and international early-career researchers to position them for success as research leaders. As per the program authorities, the Banting PDF program is intended to meet this need. The Banting PDF program exists within a competitive global PDF environment where its objectives make it a unique tool for the federal government to attract top international research talent to Canada.

The Banting PDF program complements other prestigious federal programs within a suite of elite federal research capacity development programs (e.g., Vanier Canada Graduate Scholarships [Vanier CGS], Canada Research Chairs [CRC], Canada Excellence Research Chairs [CERC]). At the postdoctoral level, there are concerns of overlap between the Banting PDF program and other Tri-agency PDF programs (i.e., agency-specific PDFs) in terms of training support provided and having too many programs that are intended to achieve similar objectives.

There is a role for the federal government and granting agencies in attracting and retaining top-tier postdoctoral trainees to increase the supply of highly qualified researchers and enhance Canada’s research capacity to foster its economic and social progress. The Banting PDF program contributes to this objective, although the extent to which it contributes is limited due to the small number of awards (70 PDFs are awarded annually). The Banting PDF is aligned with federal government and Tri-agency strategic priorities that aim to attract, retain, and develop talent to strengthen Canadian research capacity.

Performance

The Banting PDF program is producing its outputs and achieving its expected immediate outcomes specific to Banting PDF recipients. However, evidence suggests that unfunded applicants who received other sources of postdoctoral support are achieving similar outcomes during their fellowship training. Therefore, it is not clear that the achievement of key outputs and immediate outcomes can be solely attributed to the Banting PDF.

Canadian researchers and institutional representatives consulted as part of the evaluation are aware of the Banting PDF as an attractive and competitive award. Available data indicates that awareness outside of Canada is limited. Banting PDF recipients and unfunded applicants perceive the Banting PDF as prestigious, and recipients report that receiving the award has led to research and other professional opportunities during their Banting PDF.

The Banting PDF program is selecting excellent candidates in Canada, although it is unclear whether these candidates are better than their unfunded applicant peers, or whether this is largely a reflection of the assessment criteria for “top-tier”. The Banting PDF program has seen increased uptake by international applicants; however, the program has not met its target of 50% international nominations. Available data indicates that the Banting PDF program is not effective in attracting international candidates from outside Canada, although it may play a role in retaining talent during their Banting PDF.

While the influence of the Banting PDF varies, recipients are devoting most of their time to research during their fellowship, consistent with norms for postdoctoral training. The Banting PDF may provide recipients with increased autonomy to conduct their research compared to other sources of PDF support.

Banting PDF recipients are establishing national and international collaborations and are engaged in a range of leadership development activities and additional training, but these outcomes cannot be clearly attributed to the Banting PDF. Although Banting PDF recipients reported greater professional leadership compared to unfunded applicants, both groups report comparable improvement in research, teaching, and service leadership.

The Banting PDF program is achieving its intermediate outcomes; however, the incremental contributions of the Banting PDF in relation to other PDF supports appears limited. Banting PDF recipients are recognized as representatives of Canadian research excellence, demonstrating production of impactful research, achievement of awards and grants, and development of patents and company start-ups. Banting PDF recipients are demonstrating better outcomes on some, but not all, measures of research excellence compared to their unfunded applicant peers. Banting PDF recipients, for example, are producing a higher number of papers annually on average, while Banting PDF recipients and unfunded applicants are producing conference presentations or publications at a similar rate at five-year follow-up. In terms of leadership, Banting PDF recipients are more likely to be working in research intensive careers, in tenure track academic positions, and in Canada. Banting PDF recipients are recognized as leaders in their fields and are demonstrating leadership outside research (e.g., active community outreach in promoting research, production of non-academic books). However, Banting PDF recipients are recognized by their supervisors as having inherent leadership potential and would likely have achieved those outcomes without the Banting PDF.

Findings indicate that key design features of the Banting PDF program may be limiting the effective achievement of intended outcomes. The Banting PDF application process is perceived to be administratively heavy, in terms of time and effort required, for both applicants and institutions. Institution and supervisor support during the Banting PDF application process varies and can pose challenges for applicants. Program requirements, including synergy with the host institution and demonstration of leadership and research excellence, may limit the attraction of top-tier international talent in favour of those who are more advanced in their research careers and already connected to the institution. While improvements have been made to the selection process at the Tri-agency level, concerns remain regarding the lack of transparency in the institution nomination and review processes, particularly synergy with the host institution and research excellence and leadership.

The Banting PDF’s value is sufficient for most recipients in Canada but is not competitive with some key international programs. Most recipients felt that the duration of the Banting PDF award was sufficient, although some felt that the award could be longer. Removal of the 25% cap on recipients holding the Banting PDF abroad appears to have led to a slight increase in recipients taking the award abroad. It is not clear that support for leadership development has increased despite this recommendation from the previous evaluation. The Banting PDF is unique among comparable PDF programs reviewed in its commitment to equity, diversity, and inclusion (EDI) in the program’s assessment, although improvements can still be made for equity-deserving groups. For example, reviewing program features that have been identified as potential barriers, such as the window of eligibility and mobility requirement.

Practices related to performance measurement, including collection and use of EDI data, linkage between administrative data and performance measurement tools, and some structural elements of these tools (e.g., length, inconsistency of scales), present challenges in measuring the performance of the Banting PDF program.

The Banting PDF program has been delivered in a cost-efficient manner during the evaluation period with the administrative costs as a percentage of total expenditures ranging from 4.3% to 5.6% and below the overall CIHR reference value of 6%. Before the COVID-19 pandemic, the administrative costs as a percentage of total expenditures increased by 27% between 2014-15 and 2019-20. During the COVID-19 period (2020-21), it has decreased by 18% due to the adjustments of delivery modalities such as holding selection committee meetings virtually.

The Banting PDF program was flexible in responding to the COVID-19 pandemic including providing additional funding, deferring and extending award period, and switching to a virtual review process. Nevertheless, the pandemic has had a negative impact overall on recipients’ ability to conduct research.

Recommendations

  1. The Banting PDF program should consider adjusting key features of the award (e.g., award value, allowances, and duration) to remain prestigious and competitive in comparison to other international programs.
  2. The Banting PDF program needs to clarify its objective of attracting international candidates to meet the program’s target of 50% international nominations.
  3. The Banting PDF program should take steps to increase awareness of the award among the international research community, including enhancing current activities and the monitoring of these activities.
  4. The Banting PDF program should develop specific leadership development and mentorship program elements during the tenure of the fellowship to better develop Banting PDF recipients’ leadership potential and position them as future research leaders.
  5. The Banting PDF program should improve application and selection processes to better ensure transparency, including:
    • Define and improve measurement of leadership and research excellence using an EDI lens in order to ensure alignment with the Tri-agencies’ strategic priorities related to research excellence and EDI.
    • Reduce weight of the synergy with the host institution review criterion.
    • Review program features, including the window of eligibility and mobility requirement, to ensure that barriers are reduced for equity-deserving groups.
  6. The Banting PDF program needs to improve end of award reporting to improve assessment of program performance and barriers to access.

Overview of Banting PDF Program

Program Description

The Banting PDF program was announced in the 2010 federal budget as part of a broader strategy to increase Canadian capacity for research excellence. The program is jointly administered by the three research granting agencies of the Government of Canada, referred to collectively as the Tri-agencies, namely: CIHR, NSERC, and SSHRC.

The Tri-agencies were allocated $45 million over the first five years to establish a new and prestigious PDF program with the aim to attract top-level talent to Canada, and since 2014-15 they have received approximately $10 million annually. Banting PDFs are of two years’ duration, with a value of $70,000 per year. The first program applicant intake was in November 2010, with the first Banting PDFs awarded in March 2011. The Banting PDF program supports 70 new recipients per year, with a total of 140 active recipients.

The Banting PDF program aims to develop high-level research capacity in Canada by attracting and supporting Canadian and international postdoctoral talent, and retaining them in research-intensive careers to ultimately contribute positively to Canada’s economic, social, and research-based growth (see Figure 1: Banting PDF Logic Model). The specific objectives of the program are to:

Application and Selection Process

The Banting PDF program administers an annual competition with the top 70 applicants being recommended for a Banting PDF. Each of the Tri-agencies designates a selection committee to be responsible for selecting the 23 or 24 most meritorious applicantsFootnote 1. Applicants must seek endorsement from their proposed host institution to apply, prepare, and submit an application as the Tri-agencies require applicants to complete their application in full collaboration with the host institution. The institutions have individual internal processes for endorsing and supporting applications (i.e., providing an institutional letter of support and supervisor’s statement) to be submitted to the Tri-agencies via the Vanier Banting Secretariat (VBS). Once received at the level of the Tri-agencies, eligible applications are reviewed and ranked by agency-specific selection committees (i.e., one committee per granting agency) in relation to three equally weighted selection criteria:

Subsequently, the list of 70 recommended applicants is submitted for final approval to the Vanier-Banting PDF Steering Committee (also referred to as the Tri-agency Programs Steering Committee [TAP-SC]), composed of the presidents of the Tri-agencies and deputy ministers of Health Canada and Innovation, Science and Economic Development Canada. The committee also approves a list of replacements should any of the 70 applicants selected for funding decline the fellowship.

The Banting PDF program is open to Canadian citizens, permanent residents of Canada, and foreign citizens who have applied to the program and meet the following criteria:

About the Evaluation

Purpose and Scope

The purpose of this evaluation is to provide Tri-agency senior management with valid, insightful, and actionable findings demonstrating the:

By addressing these issues, the evaluation will help inform Tri-Agency program management decision-making and planning regarding the Banting PDF program, and meet the evaluation requirements outlined in the Policy on Results and subsection 42.1 of the Financial Administration Act.

The evaluation of the Banting PDF program was conducted by the CIHR Evaluation Unit in collaboration with NSERC’s and SSHRC’s Evaluation Division and with support from the Evaluation Advisory Committee. The evaluation covers the period from 2014-15 to 2020-21, with the partial coverage of 2020-21 based on the availability of data at the time of the evaluation. The extent to which the program has achieved its expected immediate outcomes was measured by examining the experiences of 2014-15 to 2019-20 recipients during their award period. The extent to which expected intermediate outcomes have been achieved was measured through the inclusion of earlier cohorts of recipients (2010-11 to 2013-14) as more time has elapsed for these recipients to demonstrate achievement of intermediate outcomes. The evaluation design used a comprehensive approach with numerous lines of evidence to maximize depth of coverage of evaluation questions and rigour, and to triangulate data.

Evaluation Context

The evaluation of the Banting PDF program was conducted with consideration of the previous evaluation of the program and the broader Canadian research training landscape. The COVID-19 pandemic struck as this evaluation was being designed; therefore, the evaluation design was amended to capture any immediate or real-time impacts on program delivery and performance and identify any potential intermediate or longer-term impacts.

Previous Evaluation

This is the second evaluation of the Banting PDF program and builds upon the first Banting PDF evaluation [ PDF (1.9 MB) ] (CIHR, 2015a) which covered the period from 2010-11 to 2013-14.

The findings from the first evaluation supported the continued need for the Banting PDF program, and its alignment with roles and responsibilities of the federal government and mandates of the Tri-agencies. The first evaluation made the following recommendations which were agreed to by Tri-agency senior management in a management response to the evaluation:

The current evaluation assessed how the program has addressed these recommendations. It builds on the first evaluation in assessing the achievement of immediate outcomes and, given that the program has been implemented for a decade, provides a more thorough assessment of the program’s achievement of its intermediate outcomes.

Impact of COVID-19 Pandemic

The COVID-19 pandemic had an impact on the delivery of the 2020 Banting PDF competition. Given that the Banting PDF program relies on the same online applications system as all other research applications, COVID-19 related funding activities were prioritized which resulted in the minor delay of the Banting PDF application deadline from June 1 to June 15, 2020. Program management engaged with the research community to make real time adjustments to events related to the pandemic that had implications on program delivery. For example, several universities along with their laboratories temporarily closed, which were anticipated to negatively impact program performance (e.g., the ability of Banting PDF recipients to continue to conduct research, engage in collaborations, and access enhanced training). In response to this, the evaluation included a specific evaluation question to assess the impacts of COVID-19 – both current and prospective – on program design and delivery, and performance.

Evaluation Questions

The evaluation of the Banting PDF program was guided by five main evaluation questions with 14 sub-questions that assess the relevance and performance of the Banting PDF program. These questions were developed in consultation with the Evaluation Advisory Committee and key program stakeholdersFootnote 3. The relevance of the program is assessed by examining the needs addressed by the program, as well as its alignment with the mandates of the Tri-agencies and the priorities of the Government of Canada.

The performance aspect assessed the program’s achievement of expected outputs, immediate outcomes, and intermediate outcomes that are expected to occur within 10 years from the program’s inception. The ultimate outcomes of the program are expected to occur after 15 years from program inception and therefore are beyond the scope of this evaluation. The ability of the program to achieve its expected outcomes was assessed via questions and indicators related to program implementation, effective and efficient delivery, and the impact of the COVID-19 pandemic on the Banting PDF program.

The evaluation questions were addressed through a series of specific indicators linked to data sources and methods, including indicators that consider SGBA+ and EDI dimensions of the question. The evaluation questions and sub-questions are presented below.

Relevance

  1. Is there a continued need for the Banting PDF program and is the program aligned with federal government priorities?
    • 1.1. To what extent does the Banting PDF program address an ongoing need?
    • 1.2. To what extent does the Banting PDF program align with federal government and granting agencies' programs and priorities?

Performance

  1. To what extent is the Banting PDF program achieving its expected outputs and immediate outcomes?
    • 2.1. To what extent are national and international postdoctoral students and institutions aware of the Banting PDF as an attractive and competitive award?
    • 2.2. To what extent has the Banting PDF program attracted top-tier talent?
    • 2.3. To what extent are Banting PDF fellows devoting their time to conducting research during their fellowship?
    • 2.4. Are Banting PDF fellows establishing national and international collaborations?
    • 2.5. To what extent have Banting PDF fellows received enhanced training?
  2. To what extent is the Banting PDF program achieving its intermediate outcomes?
    • 3.1. To what extent have Banting PDF fellows been recognized as representatives of Canadian research excellence?
    • 3.2. To what extent have Banting PDF fellows been retained and undertaken research careers in Canada?
    • 3.3. Are Banting PDF fellows demonstrating leadership?
  3. Are effective and efficient means being used to achieve intended outcomes?
    • 4.1. How do the design and delivery features of the Banting PDF program facilitate the achievement of intended outcomes?
    • 4.2. To what extent has the Banting PDF program been delivered in a cost-efficient manner?
  4. To what extent has the COVID-19 pandemic had an impact on the delivery and performance of the Banting PDF program?
    • 5.1. To what extent has the Banting PDF program design and delivery been impacted by the COVID-19 pandemic?
    • 5.2. To what extent has the COVID-19 pandemic impacted intended outcomes related to fellows’ ability to conduct research?

Evaluation Methodology

The evaluation employed both quantitative and qualitative data collection methods and analyses. Consistent with best practices in program evaluationFootnote 4 as well as the Policy on Results, multiple lines of evidence were used to triangulate evaluation findings. The evaluation methods included a document and administrative data review; an environmental scan; a funding history analysis (based on administrative data from CIHR, NSERC, and SSHRC); a bibliometric analysis; and surveys of recipients and applicants (consisting of existing end of award reports: Banting End of Award Report [BEAR] and Banting Five-Year Follow-up [B5]; as well as equivalent surveys for comparable cohorts of unfunded applicants, the latter generated for the purpose of the evaluation). Additionally, the evaluation undertook key informant interviewsFootnote 5 with Banting PDF program management (TAP-SC members, program directors), selection committee members, recipients, unfunded applicants, recipients’ supervisors, host institution officials, and a representative of Global Affairs Canada; case studies of Banting PDF recipients from the 2012-13 and 2013-14 cohorts; as well as focus groups with recipients of agency-specific PDFs who did not apply for the Banting PDF.

Sex and gender-based analysis plus (SGBA+) and equity diversity and inclusion (EDI) considerations were built into the evaluation framework via specific evaluation subquestions and indicators. Additionally, the barriers to application and experience survey (henceforth referred to as the barriers survey) focused on barriers to access for recipients and unfunded applicants as well as those barriers experienced during the fellowship (recipients only), including barriers faced by members of equity-deserving groups. Note that wherever possible, the variable of self-reported “gender” was reported on; however, historical program application data specifies only “sex” of applicant and EDI data including gender self-identification has only been recorded since 2018. Thus, sex is used as a proxy for gender when historical program data are presented.

Please note that for the purposes of this report, recipients are defined as those who received a Banting PDF; whereas unfunded applicants (the comparator group) are defined as those who applied for but did not receive a Banting PDF. In some cases, unfunded applicants received other PDF funding. For example, in the bibliometric and funding history analyses, the comparator group is specifically unfunded applicants who have received an agency-specific PDF; in the applicant and barriers surveys, unfunded applicants may have received other funding, including but not limited to agency-specific PDFs, and those who indicate that they have received a PDF of some sort are asked survey questions related to their respective PDF. In key informant interviews, unfunded applicants may have received other funding but were not asked to discuss their alternative sources of support. Thus, “unfunded” refers to those who applied to but were not funded by the Banting PDF, although they may have other sources of postdoctoral support.

Given the large number of lines of evidence with varying sample sizes, the following qualifiers have been used to indicate the frequency of responses for consistency, for qualitative lines of evidence conducted by a contractor outside the CIHR evaluation team (i.e., focus groups and key informant interviews):

None A few Some Many Most Almost all All
(0 or no) (<20%) (20-39%) (40-59%) (60-79%) (80-99%) (100%)

Note that these qualifiers have been used to summarize statements about qualitative data; they should not necessarily serve as a measure of the importance of the respective finding. These qualifiers were also used to describe some of the responses to the barriers survey, where it was deemed important to represent the perspectives of survey respondents qualitatively (i.e., potentially over-representing responses from under-represented groups) rather than in aggregated data form.

Additional details about the methodology are provided in Appendix C: Methodology – Additional Details.

Limitations of this Evaluation

The evaluation leveraged a variety of data sources. The value of this evidence-based strategy lies in the efficiency of utilizing currently available data and synthesizing these data through a single evaluative lens. However, as with all evaluations, this evaluation encountered some limitations; the limitations and associated mitigations are discussed in more detail in Appendix C: Evaluation Limitations and Mitigation Strategies . The main limitations associated with this evaluation are:

Evaluation Findings

Relevance: Is there a continued need for the Banting PDF program and is the program aligned with federal government priorities?

Key Findings:

There is a continued need to attract and retain top tier national and international postdoctoral researchers in Canada

There is evidence that the original need identified for the Banting PDF program by the program authorities to focus on attracting the very best international postdoctoral researchers to Canada and give Canadian postdoctoral researchers the opportunity to further their education and refine their skills in Canada or abroad, remains. There is a continued need for the Banting PDF as the only federal government program designed to attract and retain top-tier early-career researchersFootnote 6, particularly international postdoctoral researchers, to Canada. All TAP-SC/Program Directors, selection committee members, and host institution representatives indicated a continuing need for the Banting PDF program. Most key informants among these groups identified the need for Canada to both attract and retain Canadian and foreign talent to support Canada’s research infrastructure (e.g., universities, governments) with the next generation of scientists.

“Awards such as Banting PDF provide funding but do more than this – recognition through an award like Banting PDF provides confidence, prestige to become independent investigators and future renowned researchers.”

A Banting fellow’s Supervisor

Key informant interviews noted that the Banting PDF is responding to a demand for PDFs that is growing in Canada. A few interviewees noted that there is a growing population of postdoctoral researchers compared to faculty positions and thus an increasing demand for PDF positions to increase marketability and allow candidates time to refine their skills while waiting for faculty or other independent research positions to become available. Supervisors noted that including PDFs on their team is critical, and that there is a lack of funding in Canada and programs like the Banting PDF create opportunities for independent research.

The growing demand for a PDF was also acknowledged by the agency-specific PDF recipients who did not apply to the Banting PDF. Many focus group participants across the Tri-agencies (CIHR, NSERC, and SSHRC) indicated that in their respective fields of research, it is necessary to complete a PDF to obtain an academic position. Many participants within the SSHRC focus group described it as a needed “transition” period that allows PhD graduates to figure out what’s next in their career, and a few participants emphasized that they needed more training after their PhD to fill knowledge gaps and complete their research projects before entering the workforce.

Despite a growing PDF demand, there has been a decrease in PDF support at the federal level which highlights a continued need for a program that provides federal PDF funding, including the Banting PDF. The annual number of newly funded agency-specific PDF opportunities has slightly decreased during the evaluation period, although the magnitude of the decrease varies across the agencies. For CIHR, the number of PDFs has decreased from 154 to 140 between 2014-15 and 2018-19 while slightly increasing to 148 in 2019-20Footnote 7. SSHRC PDFs have dropped significantly between 2015 and 2018, from 172 to 129, after which they increased to 182 in 2019 before decreasing again in 2020 to 133. For NSERC, the number of PDFs has remained consistent with a slight increase between 2015 and 2019, from 183 to 194; however, between 2019 and 2020, NSERC PDFs dropped significantly from 194 to 110 (See Table 1. Number of agency-specific PDFs awarded by year, 2015-2020). Compared with program authorities’ data, there was a notable decrease in agency-specific PDF opportunities since the Banting PDF was initiated in 2010 (i.e., NSERC supported approximately 500 PDFs, CIHR supported approximately 300 PDFs, and SSHRC supported approximately 250 PDFs). According to the 2016 National PDF Survey (Jadavji et al., 2016) which collected data from 2,109 postdoctoral researchers, 23% (n = 493) of respondents reported having received CIHR/SSHRC/NSERC PDF awards. The most frequently reported source of PDF support was supervisor funded PDFs, reported by 27% of respondents (n = 573)Footnote 8. This suggests that there is limited coverage of the PDF needs by the Tri-agencies in terms of PDF awards offered.Footnote 9

The Banting PDF is a unique tool for the federal government to attract top international research talent to Canada

Evidence suggests that despite a competitive national and international PDF environment, the Banting PDF has distinctive characteristics that make it a unique tool for the federal government to attract top international talent to Canada. PDF programs in Canada reviewed by the environmental scan such as the Alberta Innovates PDFs, and the Mitacs Accelerate and Elevate postdoctoral programsFootnote 10 aim to attract talented national and international researchers and provide them with opportunities to stay in the country and contribute to achieving greater prosperity in Canada. However, the Alberta Innovates PDF focuses on health-related research in Alberta, while the Banting PDF covers all disciplines and geographical areas within Canada and abroad (for Canadian and permanent resident applicants). The Mitacs PDF also appears limited in scope (recipients can only undertake their fellowship in Canada or another Mitacs partner countryFootnote 11).

Internationally, programs reviewed by the environmental scan aim to attract the best postdoctoral researchers from around the world, such as the Newton International Fellowships in the U.K. Similarly, the Rhodes Scholarship, Fulbright Postdoctoral Awards, U.S. National Science Foundation PDFs, Swiss National Science Foundation PDF, Wellcome Trust Early Career Awards, and the Schmidt Science Fellowship are all targeting top-tier candidates. The Human Frontier Science Postdoctoral program offers opportunities to top-tier early-career researchers to move to Canada or other eligible countries (as specified by the program) for their PDFs. Although this program contributes to bringing talented researchers into Canada and offers international opportunities to Canadians, it has a limited scope. While the Banting PDF covers all disciplines, the Long-Term Fellowship stream of the Human Frontier Science Postdoctoral program only covers biological disciplines and its Cross-discipline Fellowship stream is also limited (to non-life sciences such as physics, chemistry, mathematics, engineering or computer sciences)Footnote 12. The Rhodes Scholarship, which annually offers 11 grants for Canadians and permanent residents to study in the U.K., is not dedicated to postdoctoral funding but broadly covers graduate studies at the University of Oxford, demonstrating a limited contribution in fulfilling the needs addressed by the Banting PDF program.

Furthermore, while recipients and unfunded applicants had submitted applications to other PDFs (84% of Banting PDF recipients would have pursued a postdoctoral position even without the Banting PDF, BEAR data; 64% of unfunded applicants obtained another postdoctoral position, B5 data), documents reviewed reveal that the Banting PDF program has distinct objectives that make it attractive to applicants. Specifically, the program allows the three federal granting agencies to offer prestigious PDFs at an internationally competitive level of funding to attract and retain top-tier post-doctoral talent from Canada and abroad. Interview data corroborate the finding that the Banting PDF objectives (e.g., attraction of international talent) and criteria are distinctive when compared to other federal PDF funding programs, which make it a unique program in the portfolio of the Tri-agencies (CIHR, NSERC, and SSHRC). Some interviewees noted that a unique aspect of the Banting PDF is the recruitment of foreign nationals to Canada.

Some host institution officials interviewed highlighted that the Banting PDF program is important for host institutions to be able to compete with universities in other countries, despite there being a limited number of Banting PDF awards. International recipients were much more likely to report that they would have carried out their PDF outside Canada had it not been for the Banting PDF, with almost half (40%, n = 24) of international recipients reporting this compared to a small number (14%, n = 14) of Canadians and permanent residents who held their Banting PDF in Canada (BEAR data).

However, findings from the 2016 National PDF Survey indicated that attracting highly qualified postdoctoral researchers may be influenced by Canada’s postdoctoral performance on the global stage and Canada might not be successfully accessing the global postdoctoral market (Jadavji, et al., 2016). For example, when comparing the experiences of Canadian postdoctoral researchers with postdoctoral experience reported in other countries, the survey found that there is a need for better supportFootnote 13 for international PDFs in Canada, and therefore supporting the need for PDF programs like the Banting PDF. The survey revealed that 29% of postdoctoral researchers (out of 2,109 respondents) were work permit holders which was below the 2009 (39%) and 2013 (38%) figures, indicating a downward trend in international postdoctoral researchers in Canada.

The Banting PDF program complements other prestigious federal programs, though there are concerns of overlap with Tri-agency PDF programs

Documents reviewed reveal that the Banting PDF program was intended to complete the federal suite of programs (Vanier CGS and CRC) which are all aimed at increasing the supply of highly qualified research personnel in Canada, and branding Canada worldwide as a destination of choice and as an innovative nation known for quality research and research training. According to the program authorities, the Banting PDF was developed in response to a significant gap in the continuum of support at the postdoctoral level for excellent candidates evidenced by the fact that there were no federal programs targeted specifically towards elite applicants at this level of training, as well as by the fact that Tri-agency PDF programs directly funded only about 20% of the qualified Canadian applicants.

Key informants considered the Banting PDF to be complementary to other talent programs in funding the postdoctoral training phase and bridging researchers from trainees to independent researchers. Interviews revealed that Vanier CGS, Banting PDF, and CRC are complementary in sharing a similar rationale as they provide direct funding that is higher value, and allow recipients more choices (in supervisor, project)Footnote 14. Most interviewees perceived the Banting PDF program as addressing the need for funding, referred to as “a crucial bridge,” following doctoral studies and prior to establishing a research career.

“I don’t see need for the Banting PDF program; Agency PDFs could have been tweaked. We hear from many in the research community who feel the program is not necessary. The program has created another infrastructure within the system.”

Program Management

Interviews also revealed that the Banting PDF is complementary to agency-specific PDFs in providing higher level funding to attract top-tier talent. However, about half of interviewed members of program management indicated that although the Banting PDF program is unique in its objective of attracting international talent and related eligibility criteria, selection criteria (leadership), and higher value of the award, there were concerns of “overcrowding” of PDF funding programs by the Tri-agencies, and that the fact that the program is not achieving its unique intended objectives exacerbates this potential for overlap. Program management also raised concerns that this overcrowding of programs results in inefficiency for the agencies in managing several similar but different programs, as well as inefficiency and confusion in the research community such as around which program to apply to and the investment of time to prepare several applications.

These findings align with the Fundamental Science Review Panel recommendation that there should be a reinvigoration and harmonization of scholarship and PDF programs, and optimization of the use of current awards to attract international talent. The report also recommended that the Vanier CGS and the Banting PDF programs could be most effective if they were used exclusively for international recruitment and exchange opportunities in a similar vein to the Fulbright and Rhodes Scholar programs, raising the international profile and impact of both programs. The Panel found the current mix of agency-specific and Tri-agency programs where awards vary considerably by value, duration, and international portability to be “puzzling.” With the proliferation of programs under the granting agency umbrellas, these arrangements were found to be unduly complex and arguably inefficient. The Panel also suggested that they provide only a limited number of opportunities to bring international students and postdoctoral researchers to Canada (Advisory Panel on Federal Support for Fundamental Science, 2017).

There is a role for the federal government and granting agencies in attracting and retaining top-tier postdoctoral trainees

The role for the federal government and granting agencies in supporting top-tier trainees, both nationally and internationally, to contribute to strengthening the research infrastructure and foster progress in Canada is clear based on documents reviewed and interviews. The federal government has recognized the importance of supporting the next generation of research and researchers when it announced the creation of the Banting PDF in the 2010 budget (Government of Canada, 2010), and this importance was further reaffirmed in the 2018 budget (Government of Canada, 2018b). The Science, Technology and Innovation Strategy also outlines the need for Canada to support the attraction, development, and retention of research and researchers (Government of Canada, 2014). All program management members interviewed indicated that there is a role for the Canadian federal government in attracting and retaining top-tier talent to support the country’s research capacity.

The role of the Tri-agencies in supporting top-tier national and international postdoctoral trainees is reflected in their respective Acts: Canadian Institutes of Health Research Act, 2000; the Natural Sciences and Engineering Research Council Act 1985; and the Social Sciences and Humanities Research Council Act, 1985. As Canada’s health research investment agency, CIHR strives to ensure a strong foundation of future health research leaders and is committed to offering programs that increase the supply of highly qualified research personnel. Additionally, as per CIHR’s mandate, the agency was designed to respond to the evolving needs for health research and seeks to transform health research in Canada by building research capacity in under-developed areas and training the next generation of health researchers (Government of Canada, 2000).

NSERC works with universities to remove barriers, develop opportunities, and attract new expertise to make Canada’s research community thrive. It also encourages young talent and aims to create unparalleled training opportunities for the next generation of scientists and engineers. It forges international collaborations that connect Canadian researchers to the global network of ideas and aims to attract the world’s brightest minds to our country to increase the impact of science, technology, and innovation in Canada (Government of Canada, 2021). SSHRC supports world-class research in the social sciences and humanities to advance knowledge and understanding in order to meet Canada’s current and future challenges and promote new opportunities for Canadians. SSHRC supports training the next generation of talented, creative thinkers and doers as well as promote the training of highly skilled people develop talent (SSHRC, 2020).

The Banting PDF is aligned with federal government and Tri-agency strategic priorities

The Banting PDF program aligns with several federal government priorities. The program also aligns closely with strategic outcomes and priorities of CIHR, NSERC, and SSHRC to build research capacity by aiming to attract, support, and train top-tier postdoctoral researchers to carry out research.

The Banting PDF program is aligned with the previous and current CIHR Strategic Plans. In CIHR’s Strategic Plan for 2014-15 to 2018-19 “Roadmap II”, Strategic Direction 1 (promoting excellence, creativity, and breadth in health research and knowledge translation) focused on building a solid foundation for the future (CIHR, 2015b). This was achieved by creating more opportunities to train the next generation of researchers, through the Banting PDF program. In the current CIHR Strategic Plan for 2021 to 2031, “A Vision for a Healthier Future” (CIHR, 2021), the Banting PDF program is aligned with some of the priorities and strategies identified. For example, Priority A: Advance research excellence in all its diversity, through championing a more inclusive concept of research excellence (Strategy 1), is aligned with the Banting PDF program. Also, the CIHR Strategic Plan for 2021 to 2031 has an objective of enhancing national and international collaboration (Strategy 4 of Priority A), which is a key outcome of the Banting PDF. The other priority that aligns with the Banting PDF program is Strategy 3 of Priority B: Strengthen Canadian health research capacity by enhancing training and career support (CIHR, 2021). This is very similar to the Banting PDF program’s expected outcome of recipients receiving enhanced training to prepare them for research careers in Canada.

The Banting PDF program was aligned with the 2020 NSERC Strategic Plan in the latter’s emphasis on the importance of launching a new generation of talent (second priority) with a goal of mobilizing Canada’s future brain trust (NSERC, 2020). NSERC’s most recent Strategic Plan (NSERC 2030) [ PDF (5.5 MB) - external link ])  specified the importance of deepening Canada’s wellspring of research talent (Pillar 2: Expand, diversify, and nurture Canada’s talent pool) with a recognition that the research ecosystem relies on attracting and retaining talented individuals. This pillar of the Strategic Plan goal to train the next generation of talent, and to prepare and empower those who are driven to contribute to the global knowledge economy (NSERC, 2022). This objective aligns with Banting PDF’s objective of positioning recipients for success as research leaders of tomorrow, with one of the program’s expected outcomes that recipients undertake research careers in Canada.

For SSHRC, the Banting PDF program aligns with some of its strategic priorities. SSHRC’s 2016-20 Strategic Plan [ PDF (1.2 MB) - external link ] (SSHRC, 2016) was found to align with the Banting PDF program; more specifically, with Strategic Objective 1: Enable Excellence in a Changing Research Landscape, and Strategic Objective 2: Create Opportunities for Research and Training Through Collaborative Initiatives. The first priority of SSHRC’s 2020-25 Strategic Plan [ PDF (19.4 MB) - external link ] (SSHRC, 2020) is to enhance Canada’s global leadership in social sciences and humanities research. This first priority is expected to be achieved by enabling interdisciplinary and international research collaborations. This is consistent with the Banting PDF program’s expected outcome for recipients to establish international collaborations. The Banting PDF program is also aligned with the second priority of the Strategic Plan: that is, to grow Canada’s research talent (SSHRC, 2020). This priority is focused on the importance of developing, increasing, and maintaining a talent pool of highly skilled researchers. Finally, the Banting PDF program is aligned with the third objective of SSHRC’s strategic plan, which is strengthening the research enterprise in Canada by building an inclusive research community (SSHRC, 2020).

The Banting PDF program also aligns with the Tri-agencies’ priority to integrate EDI considerations in research funding as stated in the Tri-agency EDI Action Plan for 2018-25. According to this action plan, in order to achieve world-class research, the Tri-agencies must address systemic barriers that limit the full participation of talented individuals and create a culture where embedding EDI considerations into all aspects of research is second nature. The Banting PDF program aligns with the priority of the Tri-agency EDI Action Plan, as it provides conditions for EDI to be embedded into all aspects of the program. For example, in the review criteria, the Banting PDF program provides for reviewers to consider nontraditional career paths and research metrics (e.g., leadership, applied research) into their assessment of research excellence. SGBA+ is the process by which the Banting PDF program ensures sound EDI principles are applied to research design, methods, analysis and interpretation, and/or dissemination of research findingsFootnote 15. However, it is unclear how the Banting PDF program has integrated the Tri-agency EDI priorities into the definition of top-tier postdoctoral talent.

Performance: To what extent is the Banting PDF program achieving its expected outputs and immediate outcomes?

Key Findings:

The Banting PDF is seen as an attractive and competitive award in Canada, although awareness outside of Canada may be limited

Based on key informant interviews and case studies, awareness of the Banting PDF program appears to be high in Canada, but there remain opportunities to improve awareness of the program internationally. Canadian researchers interviewed are aware of the Banting PDF as a prestigious and competitive award and they confirmed broader awareness of the Banting PDF within the research community in Canada, but it is not clear how well the Banting PDF is known outside of Canada. Two U.S. based supervisors had not heard of the award prior to supervising a Banting PDF recipient, and multiple stakeholdersFootnote 16 interviewed in the case studies (n = 6) either in the U.S. or Europe had only become aware of it after encountering a Banting PDF recipient. Similarly, key informants indicated that the Banting PDF is recognized within Canada, although there is limited awareness abroad: some selection committee members, supervisors, and members of program management felt that the Banting PDF could be better advertised outside of Canada in order to attract international postdoctoral trainees.

The previous Banting PDF program evaluation identified that the program was not well known outside of Canada, but that awareness levels were expected to increase with time. According to documents reviewed, as of 2018, the Banting PDF website is the primary source of program information and the main promotional tool. Findings from a review of the initiatives undertaken as part of the joint Vanier CGS and Banting PDF communications strategy (developed every three years) indicated that awareness of the program had increased, including social media and web metrics: visits to the Banting PDF program website have increased 20% from 2014 to 2017Footnote 17. Global Affairs Canada continues to promote the Banting PDF among other scholarships and fellowships through a number of promotional and dissemination effortsFootnote 18. The VBS conducts an annual promotional blitz to the Times Higher Education World University Rankings and the main international funding agencies to raise awareness of the Banting PDF program, including use of social media, updates on the programs’ website, distribution of promotional materials, and direct contact with international institutions. These activities contributed to a 94% increase of international web traffic on the Banting PDF program website in 2017-18; however, it is unknown whether these activities are contributing to an increase in applicants from top-ranked universities.

The number and proportion of total applications from international applicants have increased over the 2014-20 period, suggesting that international awareness is increasing, although limited, and that a wider pool of international talent is being attracted (further detail provided under later section discussing attraction of top-tier candidates). Over the 2014-20 period, the number of applications submitted by male applicants was consistently slightly higher than the number of applications submitted by female applicants.

Recipients consider the Banting PDF to be a prestigious award that can lead to other opportunities

“It makes you want to take other opportunities, right? You're like, okay, people believe in me. So I'm going to do keep doing the things I need to do to build my career in this direction and keep looking for those opportunities.”

Recipient Included in Case Studies

The Banting PDF is generally perceived as a prestigious Canadian award, consistent with findings from the previous evaluation (CIHR, 2015a). According to key informants across groups and recipients included in case studies, the prestige and competitiveness of the Banting PDF appears to be mostly recognized within Canada, and for recipients included in the case studies this was a key factor in the decision to apply for the award. Key informants from the interviews and case studies also reported that the Banting PDF program increases recipients’ confidence and sense of validation and may facilitate opportunities for leadership development; however, key informants also indicated that Banting PDF recipients are recognized as inherent leaders who would have likely pursued these developmental opportunities irrespective of the Banting PDF program. Two Banting PDF recipients included in the case studies felt that the biggest appeal of the Banting PDF was the prestige of the award itself rather than the monetary value.

Recipients surveyed at five-year follow-up felt that the Banting PDF was very prestigious (M = 4.3 out of 5, SD = 0.79; 87% reported “great” or “very great” extentFootnote 19, n = 126; B5 data). Focus group participants across Tri-agencies (who received an agency-specific award but did not apply to the Banting PDF) cited the competitiveness of the Banting PDF and the limited number of awards as the leading reason for not applying to the program. However, these participants also perceived that the agency-specific PDF was competitive and prestigious enough to help elevate their CV to the level necessary to compete confidently for a faculty position.

The Banting PDF program is selecting excellent candidates in Canada, although it is unclear whether these candidates are better than their unfunded applicant peers

Evidence suggests that the Banting PDF recipients are high quality candidates with strong applications overall, although unfunded applicant peers are also high quality candidates in terms of measures such as previous funding or publications. Thus, the Banting PDF program’s definition of “top-tier” may be more reflective of the specific review criteria of the Banting PDF (i.e., equal weighting of research proposal, synergy with host institution’s priorities, and research excellence and leadership) rather than the overall quality of the candidates themselves.

Administrative data, case studies, and interviews indicate that Banting PDF candidates are competitive and the calibre of Banting PDF recipients is excellent. According to administrative data analysis, the overall success rate across the 2014 to 2020 period was 12%, ranging from 11-13% over the period. This is similar, although slightly lower, than the success rates from the previous evaluation period of 2010 to 2014 (average: 13.9%; CIHR, 2015a), and demonstrates that Banting PDF candidates continue to be competitiveFootnote 20. Case studies indicated that Banting PDF recipients had strong applications for the Banting PDF, reflecting a variety of strengths that included strong publication records, strong and/or novel research proposals, institutional support and synergy with the host institutionFootnote 21, and demonstration of leadership. The excellent calibre of Banting PDF recipients was also noted in interviews by a few members of program management, and all Banting PDF recipients included in case studies were consistently described by their supervisors and stakeholders as having strong personal qualities such as good communication, innovation, motivation, and an ability to inspire others, which were identified as key components of their success.

Banting PDF recipients are only slightly outperforming unsuccessful applicants who received a Tri-agency PDF at the time of application to the Banting PDF, in terms of the number of published papers. The bibliometric analysis found that for the most recent cohorts of the Banting PDF (2018-19 to 2020-21 competition years), recipients produced a slightly higher number of average papers in the three years prior to application to the Banting PDF compared to their comparison group (Banting PDF recipients: M = 1.53; unfunded applicants who received an agency-specific PDF: M = 1.36)Footnote 22.

There is evidence that unfunded applicant peers are also strong candidates based on award funding prior to the Banting PDF, and prior award funding alone is not a predictor of achieving a Banting PDFFootnote 23. The Tri-agency funding history analysis found that approximately half of both Banting PDF recipients and unfunded applicants who received an agency-specific PDF had received Tri-agency funding prior to applying for the Banting PDF. For CIHR and SSHRC, there was no difference between recipients and unfunded applicants who received an agency-specific PDF in terms of the proportion of individuals who received prior Tri-agency funding (CIHR: 48% vs. 43%; SSHRC: 41% vs. 44%), and for NSERC, the unfunded Banting PDF applicants who received an agency-specific PDF actually received significantly more Tri-agency fundingFootnote 24 prior to their Banting PDF application compared to Banting PDF recipients (72% vs. 56%). SSHRC Banting PDF recipients who received awards prior to the Banting PDF had received a higher amount of total funding compared to their unsuccessful applicant peers who received agency-specific PDFs, although there was no difference in total amount of prior award funding between these groups for CIHR and NSERC Banting PDF.

Available evidence indicates that the Banting PDF program has not been effective in attracting international candidates from outside Canada

The Banting PDF program has seen increased uptake by international applicants; however, the program has not met its target of 50% international nominations. While there is evidence that excellent international candidates are taking up the Banting PDF in Canada, it is not clear whether top international candidates are being attracted from outside Canada.

Approximately one third of applicants during the 2014-15 to 2019-20 period have been Canadians applying to remain in Canada (31-37%, average: 34%, n = 1,198), one third have been Canadians applying to take the award abroad (28-36%, average: 33%, n = 1151), and just under one third have been international applicants, including those already in Canada, based on the institution where they completed their PhDFootnote 25 (27-40%, average: 33%, n = 1,166). However, the number of international applicants has increased year over year (from 156 in 2014-15 to 220 in 2019-20; see Figure 2: Applications by citizenship and location of applicant, 2014-2020), as has the proportion of applications from international applicants (from 27%, n = 156 in 2014-15 to 40%, n = 220 in 2019-20). The previous evaluation found that the proportion of international applications during the first two years of the program was 40% of total applications and had fallen to 27% in 2014-15 (CIHR, 2015a). Thus, it appears that over the ten-year period of the program, international applications decreased after the first two-years  and have since increased again to the same level as at the start of the program. In its response to the Fundamental Science Review, the Ministerial Response has tasked the VBS with reaching a target of 50% of nominations from international applicants for the Vanier CGS and Banting PDF. Although the program has made progress towards this target for international attraction, it has yet to reach this goal.

In terms of awarded fellowships, administrative data analysis revealed that of the 420 Banting PDFs awarded over the competition years 2014-15 to 2019-20, just over one quarter of recipients (27%, n = 115) were international citizens who were hosted at an institution in Canada. The number and proportion of international recipients remained relatively consistent over the 2014 to 2020 period, ranging from 23 to 31% of recipients, with the lowest proportion funded in 2017 (16%, n = 22) and the highest in 2014 (31%, n = 22).

“The program hasn’t been successful in attracting top-tier international talent. Probably because of design. There are specific deadlines and it’s easier and safer for postdocs already in Canada to apply.”

TAP-SC member/Program Director Interviewee

However, available administrative data indicates that the Banting PDF program is not effective in attracting international candidates from outside Canada, although it may play a role in retaining both Canadian and international talent during their Banting PDF. Administrative dataFootnote 26 from 2014-15 to 2019-20 indicated that 70% (n = 80) of international recipients were applying from an institution within Canada.  Additionally, of all Banting PDF recipients between competition years 2014-15 and 2019-20 (including international recipients as well as Canadians and permanent residents, n = 420) only 8% (n = 35) of recipients were international citizens applying from an institution outside of Canada. Among key informants, program management and selection committee members had mixed views as to whether the Banting PDF program has attracted international top-tier talent, although most selection committee members interviewed believed that the Banting PDF program helps retain top-tier Canadian and permanent resident postdoctoral trainees in Canada.Many recipients (Canadian/permanent resident and international) indicated that the Banting PDF program played a role in their decision to pursue a PDF in Canada, while many other recipients indicated that they made the decision to pursue a PDF in Canada regardless of whether they received the Banting PDF.

Administrative data also indicated that over the 2014-20 competition year period, almost one-half (43%, n = 181) of recipients were Canadians who held their Banting PDF in Canada (37%, n = 156, remained in Canada, 6%, n = 25, returned to Canada after studying abroad), while close to one third (30%, n = 124) of recipients were Canadians who held their Banting PDF abroadFootnote 27. This is a slight increase since the removal of the 25% cap on Banting PDFs that may be taken abroad (removed for the 2017-18 competition, based on a recommendation from the first evaluation).

The Banting PDF experience varies, though the award may provide recipients with increased autonomy to conduct research compared to other sources of support

Banting PDF recipients reported on the end of award report that they spent two-thirds of their time, on average (M = 68%, SD = 15.2, n = 225), on research activities associated with their research program, although unfunded applicants reported a similar amount of time spent on research during their respective PDFs (M = 64%, SD = 19.2, n = 79). This is consistent with reported time spent on research by recipients and applicants in the previous evaluation (CIHR, 2015a).

Most recipients in interviews and case studies remarked that the Banting PDF program allowed them to focus their energy and resources in conducting research, as opposed to being distracted by the need to find additional funds to cover their salary and/or research costs. In almost all cases studied(n = 8), the recipient and/or the supervisor reported that the Banting PDF had allowed the recipient to have increased autonomy in their research, which likely enabled them to take their research in new directions or be more innovative compared to a funding source (e.g., supervisor’s grant) that may have more rigid expectations. It is also not clear whether this sense of autonomy is unique to the Banting PDF, or whether this is simply a characteristic of PDF funding in general.

Three out of nine Banting PDF recipients included in the case studies reported that the research environment and the research they undertook would likely have been the same had they been funded through other sources. Some (n = 2), however, were not sure that their research would have been able to be funded through other sourcesFootnote 28.

Banting PDF recipients are establishing collaborations, although it is not clear whether these can be solely attributed to the Banting PDF

According to survey and end of award report data, both recipients and unfunded applicants were establishing collaborations during their PDFs. On average, both recipients and unfunded applicants (Recipients: n = 181, unfunded applicants: n = 49) established 3 collaborations (SD = 1.8) during their respective PDFs, with a range of 1 to 7 collaborations (BEAR data). At five-year follow-up, the average number of current collaborations did not significantly differ between recipients (M = 7.9, SD 10.7; Range: 1-100; n = 121) and unfunded applicants (M = 5.6, SD = 4.7; range: 1-25; n = 50; B5 data).

The extent to which the Banting PDF program has an influence on international collaborations appears to be limited. When asked about the location of formal collaborations, more than half of Banting PDF recipients reported international collaborations; however, a higher proportion of unfunded applicants reported international collaborations compared with recipients: 75% (n = 42) of unfunded applicants compared to 61% (n = 111) of Banting PDF recipients (BEAR data). Bibliometric analysis found no differences in the total number and rate of international and interinstitutional collaborations between Banting PDF and unfunded applicants who received an agency-specific PDF for NSERC, SSHRC, or CIHR from the 2010-11 to 2013-14 cohorts, over a ten-year period (2010 to 2020). While recipients from case studies and interviews felt that the Banting PDF may have facilitated collaboration and networking, for the most part Banting PDF recipients were also characterized by key informants as inherent leaders who would have likely created opportunities for collaborations without the Banting PDF.

Leadership development and additional training offered to Banting PDF recipients vary, particularly opportunities for teaching and professional development

Banting PDF recipients and their supervisors reported some additional supports and opportunities to develop leadership, although the range and types of supports vary.

The Banting PDF program has explicit objectives to develop research, teaching, and service leadership. All interviewed supervisors indicated that their Banting PDF recipients had participated in leadership activities, including teaching and supervising of graduate students; participating in university committees and national/international associations; leading multiple research projects and grant applications; establishing new initiatives; and disseminating their research through scientific publications, books, and conferences.

However, according to end of award report data, Banting PDF recipients and unfunded applicants reported similar levels of involvement and improvement in research, teaching, and service leadership activities during their respective PDFs, with both groups reporting the greatest involvement and improvement in research activities, and the least in service activities. See Figure 3: Extent of improvement in research leadership activities; Figure 4: Extent of improvement in teaching leadership activities; Figure 5: Extent of improvement in service leadership activities.

One fifth of recipients (19%, n = 27) and a slightly higher proportion of unfunded applicants (28%, n = 65) indicated in the barriers survey that a lack of teaching leadership opportunities was a barrierFootnote 29 during their PDF, with some respondents who provided comments (6% of recipients, n = 12; 10% of unfunded applicants, n = 24) specifying that they received no or extremely limited opportunities to teach. Banting PDF case study recipients also reported variability in their teaching opportunities. Given that the Banting PDF program has an expected outcome to develop teaching leadership, it is surprising that there is such variability and limited teaching leadership development.

It is unclear whether Banting PDF recipients are receiving more professional development opportunities than their unfunded applicant peers, and there are no explicit objectives for professional development outlined in the Banting PDF program. There seems to be variation in the definition of professional development activities with some overlap across other areas of leadership developmentFootnote 30, such that professional development can be understood as an overarching element of training and leadership development (e.g., project management, career development) which may be a less tangible form of leadership development than research, teaching, or service.

Key informant interviews and case studies emphasized that various professional development opportunities (formal and informal) have been facilitated by having been a Banting PDF recipient. However, most recipients included in case studies did not report having received any specific professional development opportunities that were unique to the Banting PDF. A similar proportion of recipients (68%, n = 152) and unfunded applicants (69%, n = 61) reported via end of award report and survey that they had benefitted from professional development training during their PDFs. However, a higher proportion of recipients (60%, n = 133) indicated that they were offered career preparation (e.g., Common CV preparation, interview skills), compared to unfunded applicants who reported that they were offered this during their PDFs (48%, n = 37; BEAR data).

Recipients reported a moderate to great improvement in personal/professional leadership development (M = 3.7, SD = 0.9), which was significantly higher than the average improvement reported by unfunded applicants (M = 3.2, SD = 0.9, p < 0.010; BEAR data).

Few respondents reported receiving research training outside academia (recipients: 16%, n = 35; unfunded applicants: 8%, n = 6; BEAR data). The two Banting PDF case study recipients currently working outside academia reported that work they had done during the Banting PDF provided them with transferable skills applicable to their current work, although they attributed these skills to postdoctoral training in general.

Most interviewed supervisors did describe supports, such as a travel allowance and a research stipend, that were offered specifically to Banting PDF recipients. A few interviewed recipients reported receiving additional financial support from their host institution, most of whom specified this additional support was specifically for Banting PDF recipients. Banting PDF recipients included in case studies (six out of nine) reported that they had increased opportunities to travel to attend conferences or workshops, often because their supervisor allocated additional resources to the recipient since the supervisor did not have to fund their PDF.

However, most host institution officials, most recipients, and many supervisors interviewed (key informants, case studies) indicated that the same types of training and supports are offered to all postdoctoral researchers, irrespective of which type of PDF support they received, with variability in opportunities largely dependent on the individual supervisor. This is consistent with findings from the previous evaluation (CIHR, 2015a).

When asked at five-year follow-up about the extent to which they felt their leadership development during their fellowship had prepared them for a leadership position in their career, recipients reported experiencing this to a moderate to great extent (M = 3.5 out of 5, SD = 1.2; 46% great or very great extent), while unfunded applicants reported a similar experience (M = 3.4, SD = 1.1; 37% great or very great extent; B5 data). However, most interviewed Banting PDF recipients did not attribute their status as a Banting PDF to their participation in leadership development activities.

One third (31%) of Banting PDF recipients reported that there were specific ways that the Banting PDF program could better prepare recipients for their careers. Suggestions included professional development, networking, additional funding from various sources, and provision of faculty positions for Banting PDF recipients (BEAR data). According to document review, suggestions made by program management for professional skills development include workshops and webinars organized by the Tri-agencies, although there was an interest from the institutions to lead these activities. The extent to which these suggestions have been implemented is unclear.

Performance: To what extent is the Banting PDF program achieving its intermediate outcomes?

Key Findings:

Banting PDF recipients are recognized as representatives of Canadian research excellence

Banting PDF recipients interviewed as key informants and as part of case studies identified a number of key achievements in their research careers, including securing research grants and awards (e.g., CRC); publishing articles in high impact journals (including the two case study recipients working outside academia); presenting at international conferences; establishing an independent research program, lab or company; contributing to scientific advances in their field; applying for patents; and building international collaborations.

Two Banting PDF recipients included in the case studies reported that patents (nine patents reported by one recipient) had resulted from their research started during the Banting PDF. One of these recipients also reported that two start-up companies had been developed through this research, one of which had received over $100 million in seed funding. Case study recipients in academia reported dissemination of their work outside of academia in the form of public outreach (e.g., a documentary, talks for the public), reports written to inform policy, and collaborations with industry (n = 3)Footnote 31.  The two recipients in non-academic positions disseminate their work through science communications to the public and advising international governments (respectively).

Banting PDF recipients are performing better on some, but not all, measures of research excellence when compared to unfunded applicants

Banting PDF recipients have been comparable or slightly more productive in terms of research outputs at five-year follow-up when compared with unfunded applicants, according to survey data. However, there are no meaningful differences in terms of bibliometric outputs or Tri-agency funding, specifically when recipients are compared with unfunded applicants who received an agency-specific PDF. At five-year follow-up, a slightly higher proportion of recipients reported producing peer reviewed journal publications (98%, n = 124) compared to unfunded applicants (88%, n = 51), and recipients produced a higher number of publications on average ( M = 17, SD = 18) compared to recipients ( M = 11, SD = 11). Similar proportions of recipients and unfunded applicants reported producing conference presentations or publications (88%, n = 111 and 86%, n = 50, respectively); however, recipients reported producing a higher number of these outputs, on average ( M =26.5 presentations/publications per recipient, SD = 29.5), compared to unfunded applicants ( M =8.0, SD = 5.0; B5 data). Both recipients and unfunded applicants reported producing other outputs (e.g., books/book chapters, media interviews; BEAR data)Footnote 32, although a greater proportion of recipients reported producing books/book chapters (63%, n = 79) at five-year follow-up, compared to unfunded applicants (47%, n = 27; B5 data). As expected, SSHRC recipients (29%, n = 20) more frequently reported producing other knowledge products attributed to the Banting PDF which do not fall in the categories of outputs provided in the end of award report (e.g., creative pieces, performances), compared to CIHR (5%, n = 4) and NSERC recipients (7%, n = 5; BEAR data).

The bibliometric analysisFootnote 33 found that Banting PDF recipients produced a similar although slightly higher number of papers annually, on average, compared to agency-specific PDF recipients (who had applied unsuccessfully to the Banting PDF) across all three agencies (CIHR: 2.1 vs. 2.0 papers; NSERC: 1.9 vs. 1.7 papers; SSHRC: 0.6 vs. 0.4 papers). However, there were minimal differences between the two groups in other bibliometric indicators of research excellence, namely Average Relative Citation (ARC) scores and Average Relative Impact Factor (ARIF)Footnote 34. These findings suggest that Banting PDF recipients may demonstrate slightly higher productivity than their agency-specific PDF recipient peers, consistent with self-reported survey findings (B5 data) but are not necessarily outperforming their peers on bibliometric measures of research excellence.

In terms of grant funding, Banting PDF recipients more frequently reported having received research grants in the role of Principal Investigator at five-year follow-up compared to unfunded applicants (94% vs. 81%), and recipients reported receiving a higher number of these grants on average (6.1 vs. 3.7; B5 data). Seven out of nine Banting PDF recipients included in the case studies described having received grants following their Banting PDF, including one of the individuals currently working in a non-academic position who indicated that they had been a co-applicant on several grants in collaboration with academic researchers. However, an analysis of Tri-agency funding history data of Banting PDF recipients and unfunded applicants who received an agency-specific PDFFootnote 35 reveals that there is no meaningful difference between Banting PDF recipients and agency-specific PDF recipients in the average value of the grants received and the proportion of each group that received grant funding.Footnote 36 Footnote 37

Up to 10% of both Banting PDF recipients and agency-specific PDF recipients from NSERC and CIHR received CRCs, and no SSHRC agency PDF recipients in the sample had received CRCs. A significantly higher proportion of CIHR Banting PDF recipients received a CRC (10%) compared to unfunded applicants who received a CIHR agency-specific PDF (5%, p = 0.012). For NSERC and SSHRC, a limited number of Banting PDF recipients and unfunded applicants who received agency-specific PDFs received awardsFootnote 38 after the Banting PDF/agency PDF (NSERC - Banting PDF recipients: 7%, agency-specific PDFs: 3%; SSHRC - Banting PDF recipients and agency-specific PDFs: 1%). CIHR Banting PDF recipients and unfunded applicants who received a CIHR agency-specific PDF received awards after their fellowship more frequently than their NSERC and SSHRC counterparts, and a significantly higher proportion of CIHR Banting PDF recipients (23%) received awards after the fellowship compared to unfunded applicants who received a CIHR agency-specific PDF (14%, p < 0.01)Footnote 39.

Banting PDF recipients are more likely than unfunded applicants to be working in research intensive careers, in tenure track academic positions, and in Canada

At five-year follow-up, the most frequently identified barrier to current employment was a limited supply of relevant positions or strong competitionFootnote 40. Despite concerns about the “postdoc pile-up” phenomenon (Van Benthem et al., 2020)Footnote 41, almost all interviewed Banting PDF recipients indicated that the award played a significant role in reinforcing their decision to pursue a research career and enabled them to stand out in a competitive job market. As observed in the previous evaluation, Banting PDF recipients reported that the Banting PDF had a greater influence on their desire to pursue a research career compared to unfunded applicants and their respective PDFs (recipients: M =4.3 out of 5, SD = 0.9; unfunded applicants: M =4.0, SD = 0.9 p < 0.01). At the end of their PDF, almost all recipients reported that they were fairly or very satisfied with their career to date (91%, n = 193), with lower satisfaction reported by unfunded applicants (77%, n = 129; BEAR data).

Survey data indicated that the Banting PDF has contributed positively to recipients’ career outcomes. Recipients were significantly more likely to be working in the university sector at the end of their Banting PDF compared to unfunded applicants after a similar period of time (82%, n = 155 vs. 62%, n = 82; BEAR data) and at five-year follow-up (89%, n = 118, vs. 67%, n = 78, p < 0.001Footnote 42; B5 data; see Figure 6: Sector of employment for Banting PDF recipients and unfunded applicants at end of their fellowship and five-year follow-up). Of those working in the university sectorFootnote 43 at five-year follow-up, almost all Banting PDF recipients (84%, n = 89) surveyed were in either tenured or tenure-track positions, which was slightly higher than for unfunded applicants (75%, n = 54). This difference was not significant from among those working in the university sector; however, from overall employment across sectors (including government, academic sector, hospitals and other health care providers, private sector/industry, non-profit organization), a significantly higher proportion of Banting PDF recipients (67%, n = 89) reported holding academic tenure/tenure track positions compared to unfunded applicants (48%, n = 54Footnote 44, p = 0.002, see Figure 7: Percentage of employed Banting PDF recipients and unfunded applicants in tenured/tenure-track positions). Thus, taken together the data suggest that Banting PDF recipients are more likely to be employed in academia and also obtain tenure-track positions if they remain in academia. CIHR recipients were less likely to be in tenured or tenure-track positions (79%, n = 15) compared to NSERC (100%, n = 17) and SSHRC (93%, n = 27) recipients, although numbers are small when broken down by agency (B5 data). Recipients who participated in interviews and case studies perceived that Banting PDF recipients have a distinct advantage when competing for academic positions, since the Banting PDF program is a highly regarded credential on a resume. Although all nine Banting PDF recipients selected for the case studies were in tenure track academic positions or non-academic leadership positions, some reported that it took several years to obtain a position.

Banting PDF recipients identified their jobs as being more research intensive compared to unfunded applicants (89%, n = 160, vs. 71%, n = 97 very or extremely research intensive) at the end of their PDF (BEAR data). Among those who indicated that research was a main activity related to their position at five-year follow-up, the average number of reported hours per week spent on research and related tasks by recipients was 31.7 (SD = 16.6; Range: 2-90, n = 123), which was higher than the 27.0 hours on average per week reported by unfunded applicants (SD = 13.7; Range: 1-70, n = 84, p = 0.03; B5 data).

Available evidence suggests that Banting PDF recipients prefer to remain in Canada, and that the Banting PDF is having a positive influence on retention. Following their PDF, two thirds of both recipients (65%, n = 122) and unfunded applicants (63%, n = 81) reported via survey that they were working in Canada (BEAR data). However, at five-year follow-up, recipients were slightly more likely to be employed in Canada than unfunded applicants (70%, n = 75, vs. 61%, n = 49; B5 data; see Figure 8: Proportion of Banting PDF recipients employed within and outside Canada, at end of their fellowship and five-year follow-up). Looking at citizenship of Banting PDF recipients, a slightly larger percentage of Canadian recipients reported having a position located in Canada at the end of their PDF (68%, n = 94) compared to foreign citizen recipients (57%, n = 26).

At five-year follow-up, most (86%, n = 57) Canadian recipients reportedly were working in Canada compared to only about one third (36%, n = 8) of foreign citizen recipients. Of those recipients who held their Banting PDF outside Canada, most (87%, n = 37) intended to return to Canada in the foreseeable future; this proportion was slightly lower for unfunded applicants who held PDFs outside Canada (74%, n = 37). Canadian Banting PDF recipients working outside Canada reported that they intended to return to Canada as a resident in the foreseeable future much more frequently (73%, n = 32) compared to international recipients (10%, n = 2). Based on the small number of foreign citizen recipients these results need to be interpreted with caution and it is difficult to draw conclusions regarding retention of foreign citizens.

Almost all of those interviewed who had completed their Banting PDFs were currently employed (or had been offered positions) as faculty members at Canadian universities, and six out of nine of the Banting PDF recipients included in case studies were currently employed in Canada – five in academia, and one in provincial government. Three of the five recipients (two interviewed, three case studies) who were employed outside of Canada were looking to return to Canada. All three case study recipients currently employed outside Canada described the competitive academic job market and lack of job openings in Canadian universities as the major obstacle to returning to Canada.

Banting PDF recipients are recognized as leaders in their fields, although they are also recognized as having inherent leadership potential

Findings from various lines of evidence indicate that Banting PDF recipients are perceived as leaders, both within and outside their respective research areas (based on self-report and observations of others) and demonstrate greater performance on some measures of leadership in comparison with their unfunded applicant peers. At five-year follow-up, surveyed recipients rated themselves as research leaders to a significantly greater extent (M = 3.6 out of 5, SD = 1.2) than unfunded applicants (M = 3.2 out of 5, SD = 1.2, p = 0.03; B5 data).

Surveyed Banting PDF recipients and unfunded applicants reported having participated in invited presentations at local, national, and international levels, although recipients reported these presentations slightly more frequentlyFootnote 45. In terms of teaching leadership, recipients and unfunded applicants reported engaging in unpaid teaching and guest lecturing (75%, n = 98) slightly more frequently than applicants (59%, n = 59) at five-year follow-up (B5 data). All nine Banting PDF recipients included in the case studies continue to teach and mentor, either as part of their teaching duties as academics or in their non-academic positions, and are recognized for their commitment to producing the next generation of researchers. This is despite their mixed, and in some cases limited, opportunities to teach during their Banting PDF.

With respect to service leadership, recipients were also slightly more likely to report engaging in peer review (93%, n = 122) and committees or working groups (73%, n = 96) at five-year follow-up compared to applicants (85%, n = 87 and 62%, n = 64, respectively; B5 data). Five out of nine Banting PDF recipients involved in the case studies had taken an active interest in EDI issues within and/or outside of academia, serving on committees or working groups dedicated to addressing EDI issues, or organizing conferences with an EDI lens. Four out of nine demonstrated active community outreach in promoting their research area outside of academia to the public, and two recipients were currently writing non-academic books.

All nine Banting PDF recipients involved in case studies are recognized by others, including their supervisors and research stakeholders, as leaders in their field of research, having received national and international recognition, including recipients (n = 2) working outside of academia. Interviewed supervisors reported that the Banting PDF recipients have made contributions to their respective fields via their scientific publications, books, and other forms of knowledge production and dissemination. According to key informant and case study interviews with supervisors and stakeholders, Banting PDF recipients are also recognized as inherent leaders, possessing personal qualities that would have allowed them to be successful and take on opportunities to demonstrate leadership irrespective of receiving the Banting PDF.

Performance: Are effective and efficient means being used to achieve intended outcomes?

Key Findings:

The Banting PDF application process is perceived to be administratively heavy for both applicants and institutions

The analysis of end of award data show that three quarters (74%, n = 133) of Banting PDF recipients were satisfied with the Banting PDF application process. There were agency-level differences in satisfaction with the application process: NSERC recipients reported the highest satisfaction with the application process (88% very or extremely satisfied, n = 53), followed by CIHR recipients (78%, n = 45), with SSHRC recipients reporting lowest satisfaction (64%, n = 37). By comparison, only one-third (32%, n = 63) of unfunded applicants were satisfied with the application process (BEAR data).

Interviewed host institution officials and program documents reviewed reveal that the application was “too heavy” for applicants, and both sources also identified the considerable burden the application placed on the institution. For institutions, concerns included the time-consuming nature of application and review processes for the institutions (due to number of applications assigned and length of applications). For applicants, concerns included a lack of clarity around the guidelines and the role of the arm’s length referee; a need to streamline the supervisor’s statement; and the length of the Common CV template (though it was noted there have been improvements to the Common CV, these issues still exist). These issues have also been identified by unfunded applicants and recipients (via interviews and survey data). Some interviewed unfunded applicants and recipients reported that the application process was cumbersome, particularly the multiple steps involved (e.g., internal and Tri-agency adjudication) and multiple components of the application (e.g., institutional support letter, arm’s length referee, supervisor Ysupport letter, endorsement step). A few interviewees, including host institution officials and unfunded applicants, noted that internal deadlines were challenging, especially for international students.

Findings from focus groups conducted with those who had not applied to the Banting PDF (but had received an agency-specific PDF) indicated that the application process was perceived to be labor-intensive and challenging to complete in the allocated time frame. Further, focus group participants expressed frustration with the substantial differences between the agency-specific PDF and the Banting PDF application processes, particularly with respect to the submission deadlines and the fact that application materials for one award could not be used either in full or in part for the other. Some participants felt that the agency-specific PDF and Banting PDF should be combined into one program where top-rated applicants would be granted a Banting PDF.

Institution and supervisor support during the Banting PDF application process varies and can pose challenges for applicants

The degree to which Banting PDF applicants are supported by the institution and/or their supervisor varies and may impact the application process. A much higher proportion of unfunded applicants (39%, n = 132) who responded to the barriers survey indicated that access to institutional support for the application process was a barrier to their Banting PDF application, compared to recipients (17%, n = 29). Some recipients and unfunded applicants interviewed reported that they had received significant support and mentorship in navigating the application process from their supervisor and award offices, while others, particularly those from earlier cohorts, indicated that they did a lot of the work on their application on their own without much support. A few interviewed host institution officials stated that a weakness of the application process was that sometimes applicants may fall through the cracks due to lack of university resources to support them. Three out of nine Banting PDF recipients included in case studies noted the challenges in obtaining the resources from the department or institution (e.g., to specify the research support, professional development, and leadership opportunities that would be provided) required for their application. In terms of supervisor support, several interviewees (one host institution official, and one case study recipient and their supervisor) highlighted the importance of informal networks in establishing a supervisor and institutional connections which could introduce biases in the nomination process.

Concerns remain around the lack of transparency in the institution nomination and selection processes

Some improvements have been made to the Banting PDF selection processes, although perspectives of program stakeholders on the current state of these processes are mixed. According to survey data, unfunded applicants reported lower satisfaction with the selection process compared to recipients; in particular, the decision/peer review process and the eligibility requirementsFootnote 46.

Among selection committee members and the few members of program management interviewed who could speak to effectiveness of the Banting PDF selection process, most felt the process was working well. According to document review, several selection process elements have been adapted over the evaluation period including changes to binning processes for scoring applications, the implementation of an unconscious bias learning module for referees and reviewers, and addition of the requirement for the arm’s length referee. The update to the binning process has been received positively by the review committee and seems to be less likely to deprioritize higher quality applications.

Other concerns related to the selection process included the lack of transparency in the nomination process and timing of the decision process. More than one quarter (28%, n = 126) of unfunded applicants who responded to the barriers survey indicated that the transparency of the institutional nomination process was a barrier to their application, compared to a small proportion of Banting PDF recipients (14%, n = 24). There is limited available information to assess the institution’s nomination process because the institution ultimately decides which candidates are considered “top-tier” and thus move forward to the next stage of application. The institutional nomination process is neither transparent nor standardized across institution; therefore, it is not possible to evaluate the full range of applicants including those who apply at the level of the institution. This presents an obstacle to the assessment of equity and access to the Banting PDF program.

The selection criteria of synergy with host institution, and leadership and research excellence may be limiting the attraction of top-tier talent

The document review identified concerns among the TAP committees regarding the weighting and assessment of the three Banting PDF review criteria: synergy with host institution priorities, applicant’s research proposal, and research excellence and leadershipFootnote 47. In particular, most selection committee members felt that the leadership criterion should have the least weight, while one felt that the synergy should carry the least weight. Some felt that the weighting of the three criteria was appropriate, although these committee members still felt they could be better supported by the Banting PDF program as it remains challenging to select individuals from a highly qualified pool of applicants. Here, it is important to note that the environmental scan found that the Banting PDF program has similar assessment criteria to other comparable programs (6 out of 18 programs reviewed included research excellence, quality of research program, and synergy with host institutions in their assessment criteriaFootnote 48). Recipients and unfunded applicants reported via interviews that they felt the review criteria were clear and transparent. However, a proportion of recipients and unfunded applicants reported that the synergy and research excellence/leadership criteria lacked clarity in their assessment and posed barriers to application. In addition, these review criteria may be biased towards those already at the institution or those who have had a longer period of time to demonstrate leadership and research excellence, thereby limiting the success of candidates who would otherwise be considered “top-tier”.

Leadership and Research Excellence

More than one-third of unfunded applicants indicated via the barriers survey that the assessment of research excellence (35%, n = 154) and leadership (41%, n = 176) were barriers to their application, compared to a much lower proportion of recipients who indicated this (6%, n = 10 and 7%, n = 12, respectively). Of those who provided comments, one quarter of unfunded applicants and recipients combined (25%, n = 31) felt that there was a lack of feedback or clarity on how research excellence is assessed, and more than half (53%, n = 49) noted a similar lack of transparency and clarity of the criteria for assessment of leadership. It should be noted that no feedback is provided to applicants on their applications.

A few interviewed host institution officials commented that the assessment of research excellence and leadership is discipline-specific. Host institution officials also reported variation in the measurement of research excellence within institutions, particularly the extent to which assessment at the institutional level relied on traditional measures (e.g., number of publications, journal impact factor, first author publications)Footnote 49. All selection committee members discussed the challenges associated with assessing research excellence in terms of research outputs, notably because certain outputs are rated differently across disciplines; that is, some disciplines expect a high number of publications whereas others recognize different types or volume of outputs. A few recipients and unfunded applicants commented in response to the barriers survey that there was too much focus on publications or productivity in assessment of research excellence (15%, n = 12), that there was a bias against the type of productivity in their field due to different publication behaviours (14%, n = 10), and that interdisciplinary research was disadvantaged (10%, n = 8). Nearly half of unfunded applicants (40%, n = 173) felt that perceived bias in the decision process related to applicant’s research area was a barrier to their Banting PDF application compared to a few (5%, n = 8) recipients. Due to the challenges related to comparing research outputs across disciplines, some selection committee members emphasized the importance of the arm’s length referee’s assessment of the candidate’s research proposal, even though unfunded applicants note concerns related to this application requirement. Selection committee members struggled to provide suggestions to improve the assessment of research excellence, although all emphasized the need to ensure more accurate assessments across the disciplines.

With respect to the assessment of leadership, all interviewed applicants, supervisors, reviewers, institutions, and selection committee members recommended that the Banting PDF program leadership criterion be clarified as the current definition and relevant instructions to applicants and reviewers have led to some inconsistent interpretations. This finding is consistent with findings from the document review. Some members of program management felt that cultural differences and past paid work leadership opportunities should be considered in the reviews; however, the selection committees did not want to be too prescriptive in assessing services outside the academic domain. Based on comments specific to the assessment of leadership, a few unfunded applicants felt that their leadership was unfairly evaluated (12%, n = 10), or that there was a bias toward a particular type of leadership (9%, n = 8) such as formal or research leadership.

Further, several (n = 6)Footnote 50 respondents to the barriers survey described the expectation to demonstrate leadership and research excellence as being unreasonable for someone at the end of their PhD. Thus, it may be that in its effort to select the candidates with the most complete CVs, the Banting PDF program is biased towards selecting individuals who have been able to demonstrate a longer period of productivity and leadership beyond their PhDs rather than selecting those who are demonstrating the greatest success at the end of their PhDs. The analysis of end of award report data indicated that half (51%, n = 55) of recipients in the sample applied to the Banting PDF at least a year after completing their PhDFootnote 51. A further 22% (n = 24) applied in the same year (BEAR data). Given the gap between the Banting PDF application deadline (fall) and earliest start of funding (spring of the following year), these data indicate that almost three quarters of Banting PDF recipients did not go straight into the Banting PDF after completing their PhD, or at least would likely not have received the funding until 6 months or more following the completion of their PhDFootnote 52. Thus, the Banting PDF review criteria may place a greater burden on applicants in order to demonstrate that they are “top-tier”, which may be difficult to demonstrate without prior postdoctoral experience.

Synergy with Host Institution’s Priorities

The TAP-SC has frequently discussed the program’s “institutional commitment and demonstrated synergy between applicant and institutional strategic priorities” criterion. Committee members have stated that they would like less weight placed on this criterion than the other two criteria. They also indicate that the intent of this criterion should be clarified because it can be confusing for applicants and difficult for selection committee members to assess. A higher proportion of unfunded applicants (33%, n = 141) than recipients (14%, n = 23) who responded to the barriers survey felt that the assessment of synergy with the host institution’s priorities was a barrier to their Banting PDF application. Of those who provided comments, more than two thirds of both recipients (72%, n = 8) and unfunded applicants (71%, n = 42) felt that this criterion was unclear or lacked transparency.

The synergy criterion could also be favouring candidates who are already at the institution where they intend to hold the Banting PDF which potentially limits the attraction of top-tier candidates from outside Canada and from other institutions within Canada. According to administrative data from 2014-15 to 2019-20, more than half of Banting PDF applicants were applying from the same institution as their proposed host institution (53%, n = 1870). The proportion of candidates already at their proposed host Banting PDF institution was similar for international applicants (50%, n = 587) and Canadian applicants (55%, n = 1283), although the number of applicants already at their host institution has decreased slightly over the period under review (65%, n = 376 in 2014 to 53%, n = 292 in 2019-20; see Table 2. Proportion of Banting PDF applicants (funded and unfunded) already at host institution at the time of application, by citizenship, 2014-15 to 2019-20).

Although the program requires applicants to hold their Banting PDF at a different institution from the one where they did their PhD (except in cases with valid justification), eight out of nine recipients in the case studies had already started their research at the institution where they held their Banting PDF. Seven out of nine cases had begun postdoctoral work at the institution when they applied for the Banting PDF, and the eighth recipient had spent a year on exchange working with their supervisor on the same line of inquiry. The only Banting PDF recipient from the case studies who applied straight out of their PhD was applying to a research institute that was an affiliate of the institution where they completed their PhD.

The environmental scan found that like the Banting PDF, other programs where the host institutions must endorse applicants also have no control over the internal process leading to this endorsement. However, some programs offer more flexibility than the Banting PDF in terms of the host institution affiliation. For example, the U.S. National Science Foundation’s Social, Behavioral and Economic PDF program does not require applicants to secure an affiliation at the time of application and allows the applicant to provide proof of affiliation later, before starting the program. Other programs like the Schmidt Science Fellowship offer placements for recipients after selection.

The Banting PDF’s value is sufficient for most recipients in Canada but is not competitive with some key international programs

The value of the Banting PDF is largely perceived by recipients and other stakeholders as being sufficient and competitive within Canada, providing a living wage, facilitating more opportunities for conferences and workshops, and providing recipients with autonomy in their research. While the Banting PDF is competitive against some comparable national and international PDFs, the value of the Banting PDF is lower than some other prestigious international PDF programs. Consistent with the finding that the Banting PDF is competitive nationally but less so internationally, the 2020 CAPS survey (Sparling et al., 2022) found that the median salary for postdoctoral researchers studying in Canada was $51,375, whereas the median salary for Canadian postdoctoral researchers studying abroad was close to the same value as the Banting PDF at $68,500. Thus, the value of the award may limit the Banting PDF program’s ability to attract top-tier international talent, as well as impacting the competitiveness of the award for Canadian recipients taking the award abroad.

Canada’s Fundamental Science Review Panel (2017) found that levels of postdoctoral support in Canada are inconsistent with those globally. Canada’s award values have remained too static, and the Panel repeatedly heard from researchers that in order to recruit the best talent from abroad, we must be competitive internationally. The Panel’s recommendation was to reinvigorate and harmonize scholarships and PDF programs across agencies and highlighted more consistent and more generous levels of support (value and duration) are needed. This recommendation was echoed in the 2020 CAPS survey report. Moreover, the recent Report of the Advisory Panel on the Federal Research Support System (2023) also made a recommendation that funding for graduate students and postdoctoral researchers should be increased to an internationally competitive level. Although this report was released late in the reporting phase of the evaluation, its review was included as a key policy document for both the Banting PDF program as well as PDF funding at the federal level going forward. Given the program’s objectives, the Banting PDF seems well positioned to fulfill the role of an internationally competitive PDF for Canada.

Overall, interviewed program management and host institution officials identified few issues related to the value of the award and most felt this feature was competitive and sufficient to attract top-tier talent. Some interviewees (program management) suggested that the program could be made more distinctive from other PDFs not just by the amount of the award, but in other ways (e.g., by awarding a small research grant or with a more explicit link to academia). A few noted that the higher salary, while beneficial for the individual and good for recruitment, doesn’t translate into a different experience per se.

Almost all Banting PDF recipients surveyed (86%, n = 180) were satisfied (rated very satisfied or extremely satisfied) with the monetary value of the award (M = 4.2 out of 5, SD = 0.9; BEAR data)Footnote 53. All nine Banting PDF recipients included in the case studies indicated that the value of the award was important, with more than half reporting that the funding facilitated more opportunities to attend conferences and workshops (n = 6) and allowed for greater autonomy in their research as a result of having their own funding (n = 5). In addition, for five out of nine case study recipients who held their Banting PDF in cities with higher cost of living, the Banting PDF provided a living wage that offered greater financial security and enabled personal circumstances which in turn allowed for more time to conduct research.

However, a few Banting PDF recipients interviewed felt that the value of the Banting PDF award should be increased (e.g., indexed to inflation or relative to location), and one quarter of both recipients (25%, n = 42) and unfunded applicants (23%, n = 109) indicated that the monetary value of the award was a barrier when considering application (per the barriers survey). Of those who provided comments, most (88%) felt that the value was too low. Specific responses as to why the value was too low included: value relative to cost of living and/or inflation (36%, n = 31); not internationally competitive (21%, n = 18); value compared to other postdoctoral researchers (7%, n = 6); lack of research budget or value is too low for research requirements (8%, n = 7); and, taxation of the award is a challenge (7%, n = 6). The two Banting PDF case study recipients who held the award in the U.S. either received a salary top up by the institution or relied on their spouse’s income because of the poor exchange rate at the time. The case study recipient who held their award in Europe did note that it would have been difficult to take a regular agency-specific PDF abroad due to the lower value in relation to cost of living in Europe.

The Banting PDF annual award value of $70,000 is competitive, with the same or higher value, when compared to other PDF programs in CanadaFootnote 54 and some internationallyFootnote 55. However, the environmental scan found that 6 out of 15 international programs offered higher award values. For example, the Human Frontier Science PDF offers a more competitive award value and longer duration in Canada than the Banting PDF: annual stipend of around $70,000Footnote 56 complemented with a child allowance ($6,990 CAD per child annually) and a relocation allowance ($9,552 CAD). Internationally, the Banting PDF remains less competitive in terms of award value than other prestigious PDF programs such as the National Science Foundation PDF, Swiss National Science Foundation PDF, US National Institutes of Health’s K99/R00 Fellowships (Pathway to Independence Awards), Schmidt Science Fellowships, and Applied Harvard Society Junior Fellows, which range from $109,000 to $136,000 CADFootnote 57. The lower value of the Banting PDF value compared to these prestigious international programs reduces the ability of the program to be competitive internationally in attracting top-tier postdoctoral talent.

Most recipients felt that the duration of the Banting PDF award was sufficient, although some felt that the award could be longer

The duration of the Banting PDF was generally deemed to be sufficient, with three quarters of Banting PDF recipients (74%, n = 133) reporting satisfactionFootnote 58 with the length of the award (BEAR data). However, some recipients felt that the award could be longer and the length of the award may present a barrier to accomplishing research goals.

One fifth of recipients (20%, n = 34) and unfunded applicants (19%, n = 88) who responded to the barriers survey felt that the duration of the award was a barrier. Of those who provided comments, some specified that it was too short to meet the requirements of their research projects (27%, n = 16) or in comparison to other PDFs (23%, n = 14). Five out of nine recipient cases reported that the duration of the Banting PDF was long enough to complete the research outlined in their proposal, although all five recipients also acknowledged that they had additional time (i.e., during their PhD or a previous PDF) to begin the work they continued during the Banting PDF, and that without this time the two-year duration may not have been long enough. Three out of nine recipients included in case studies and a few interviewed recipients felt that the duration of the award should be longer.

Perspectives on the length of the award from recipients included in the case studies differed by agency: SSHRC case study recipients (n = 3) all felt that the duration was sufficient, although two felt that a longer period may have been beneficial. NSERC case study recipients (n = 3) all felt that the award could be longer. CIHR case study recipients (n = 3) all felt the award length was sufficient to accomplish their research. That said, two CIHR recipients acknowledged that the award duration could have been longer, given they were able to secure subsequent grant funding afterward that enabled them to continue their research, while the third recipient secured a job by the end of the Banting PDF. CIHR Banting PDF recipients reported much lower satisfaction with the length of the award (58% very or extremely satisfied, n = 34) via survey compared to SSHRC and NSERC recipients (83%, n = 48 and 82%, n = 48; BEAR data). Thus, the agency-specific differences in satisfaction with the Banting PDF reflect differences in needs regarding the length of a PDF by research discipline (e.g., natural sciences and engineering and health researchers may require longer PDF awards than the social sciences and humanities) which may be something to consider for future program design.

The views of supervisors and stakeholders included in case studies were mixed in terms of the appropriateness of the length of the award. A few (n = 3) felt that the fellowship should be three years. One SSHRC supervisor felt that the two-year period was long enough, assuming the research idea is well developed prior to the start of the PDF period. Most of the other PDF awards reviewed in the environmental scan have a duration of two or three years, making the duration of the Banting PDF, at two years, comparable with some of the awards in the PDF ecosystem. However, a number of other programs have awards with a duration of three years: the Human Frontier Science Postdoctoral program, Swiss National Science Foundation PDF, Marie-Curie Actions (Global stream), Applied Harvard Society of Fellows, Alberta Innovates PDF, and Mitacs (Accelerate stream). The document review found evidence of support for increasing the length of the Banting PDF award from two years to three or four years. For example, in response to VBS consultation seeking input to increase international attraction, a CRC member indicated that increasing the award duration would be the most important change the program could implement to recruit international applicants.

The removal of the 25% cap on recipients holding the Banting PDF abroad appears to have led to a slight increase in recipients taking the award abroad

A significant change was made to the Banting PDF program over the evaluation period: the 25% cap on recipients holding their awards abroad was removed in 2017Footnote 59. This change was implemented for the 2017-18 competition year and all future Banting PDF competitions. According to committee meeting documents, this change has helped to ensure that the best applications are funded, and better align the program with other PDF programs. Application data review indicates that there has been a slight increase in the percentage of awards held abroad since the removal of the 25% cap: 34% of awards were held abroad in 2017-18 competition year (the first year the change was implemented) and 30% of awards on average have been taken abroad since the removal of the capFootnote 60.

It is not clear that support for leadership development has increased despite this recommendation from the previous evaluation

The previous evaluation recommended that the Banting PDF program should develop guidance regarding leading practices for the support of Banting PDF recipients to develop their leadership potential and position them for success as research leaders of tomorrow (CIHR, 2015a). Steps to address this recommendationFootnote 61 were reported by the VBS as being fully implemented through a memo at the launch of the 2016-17 competition, reminding institutions of the importance of providing support to the recipients beyond what is offered to regular postdoctoral researchers. Nevertheless, there are currently no formal leadership development or mentorship requirements in the Banting PDF program profile and logic model.

In comparison, the environmental scan found that training, mentorship, and career development plans were identified as essential features in 9 out of 18 peer programs. For example, PDFs that offer financial support specifically for training include Alberta Innovates PDFFootnote 62, Marie-Curie Actions, National Science Foundation (U.S.), Schmidt Science Fellowships, and Wellcome Trust U.K. The Wellcome Trust provides that the host organization should give 10 days a year to the recipient for training and continuing professional development. The review did not find evidence of a clear or consistent process to ensure that host institutions provide the opportunities they committed to offer to recipients during their PDF. One supervisor and one stakeholder (a former Banting PDF supervisor) included in the case studies specified that the Banting PDF program could enhance its requirements for mentorship to ensure that supervisors are doing an adequate job of meeting the needs of their trainees.

Improvements to the Banting PDF end of award reports would improve the assessment of program outcomes

The Banting PDF program has developed and implemented an effective performance measurement and monitoring system using an end of award report (BEAR) and a five-year follow-up survey (B5). As with the Vanier CGS Program, the Banting PDF program’s end of award and follow-up survey instruments are best practices among Tri-agency funding programs and have collected a large volume of data that has supported program delivery as well as this evaluation. However, issues remain with these performance measurement tools that can limit their utility in assessing program performance. For example, some structural elements of the BEAR and B5 (e.g., length, inconsistency of scales), lack of linkage between administrative data and these instruments, limited collection of EDI data, and limited measurement of key issues such as leadership outcomes and barriers at application may restrict the program’s ability to assess program outcomes.

The Banting PDF is unique among PDF programs reviewed in its commitment to equity, diversity, and inclusion, although improvements can still be made

Overall, Banting PDF recipients tend to report minimal experience with barriers related to EDI issues; however, unfunded applicants across lines of evidence more frequently report experiencing barriers which may have an impact on their ability to apply for and be assessed by the program.

Efforts regarding EDI considerations in the Banting PDF program are ongoing. Some members of program management and half of selection committee members interviewed observed that there is effort on the part of the program to include EDI considerations in program design and delivery through communications and training modules, and host institution officials indicated that EDI is taken into consideration in their selection and endorsement process: Tri-agency materials on EDI are shared and some institutions host EDI training workshops for reviewers. Also, a page on Equity, Diversity and Inclusionwas  added to the Banting PDF website in 2018  which outlines EDI and SGBA+ considerations in program design and delivery. The environmental scan found that the use of the SGBA+ tool during the application assessment makes the Banting PDF program unique among similar programs (although it is worth noting that this unique tool is also a feature of the agency-specific PDFs)Footnote 63. The unconscious bias learning module was viewed as successful overall by program management, although there is general agreement that the program needs to demonstrate more commitment to EDI issues, particularly beyond gender. Several suggestions for improvement across interview respondent groups included examination of eligibility and selection criteria that may generate disadvantages from an EDI perspective, particularly the assessment of leadership, and collection of more data on the profile of applicants and success rates of different EDI applicant groups. The views of program management and selection committee members were mixed as to whether the program should have EDI targets, noting that it would be difficult to meet inclusion objectives without increasing the number of awards.

Eligibility and Application Process

Some interviewed Banting PDF recipients recognized that EDI is considered in the Banting PDF application and selection processes, but were uncertain how or whether this was applied in practical terms in the adjudication. None of the interviewed recipients or unfunded applicants reported that they had experienced or known of someone who had experienced an EDI-related barrier in their interactions with the Banting PDF program. However, a few barriers were identified for equity-deserving groups, related to the application process and specific program design features.

The program design feature most identified as a potential barrier to equity-deserving groups was the requirement to move to a different research environment (usually necessitating a move to a different city, province, or country) in order to complete the Banting PDF, as reported by SSHRC and NSERC agency-specific PDF recipients who participated in focus groups and three individuals (one stakeholder, two Banting PDF recipients) involved in case studies. Document review indicates that program management has frequently discussed the program’s mobility requirement, debating its value and indicating that it is difficult to evaluate. Program management and SSHRC and NSERC focus group participants noted that while this requirement can expose researchers to new people and ideas, it can disadvantage individuals with disabilities, family responsibilities, and/or fewer financial resources. The mobility requirement was reported as a barrier for over one quarter of unfunded applicants (29%, n = 137) and one fifth of recipients (19%, n = 32; barriers survey), with comments most commonly citing family reasons (52%, n = 45). There is currently a provision for the applicant to provide justification for remaining in the same environment, which program management has supported to allow for circumstances outside the applicant’s control (e.g., caregiving responsibilities).

The window of eligibility to apply for the Banting PDF was also viewed as a potential barrier. A much higher proportion of applicants (about one quarter, 28%, n = 147) indicated in response to the barriers survey that the window of eligibility to apply to the Banting PDF following PhD or health professional degree completion was a barrier to their application, compared to few recipients (8%, n = 14). A few host institution officials in interviews suggested extending or revising the window of eligibility.

The document review found that some concerns remain regarding gender equity in the application process. In 2018, no significant differences for funding decisions based on gender of applicant were found across agencies, and there was no trend in scoring patterns between women reviewing women, and men reviewing women. However, in 2020, program management raised concern with ongoing variability of success rates between men and women relative to their respective postdoctoral research populations. As a result, an equity champion was introduced, and messaging and communications around EDI were strengthened. An additional suggestion from these discussions was to create structured referee letters and analyze selection criteria which may introduce biasFootnote 64 (see Figure 9: Number of Banting PDF applications and success rate by sex, 2014-15 to 2019-20). More recent VBS data indicates that this alignment occurred for the 2021-22 competition, and the proportion was relatively aligned for 2022-23. That said, women unfunded applicants reported lower satisfaction with the eligibility requirements compared to men, via end of award report.Footnote 65 Program management concluded at a 2020 meeting that the variances by agency appear to be a result of differences in postdoctoral populations (more women postdoctoral researchers in Social Sciences/Humanities and Health, more men postdoctoral researchers in Natural Sciences and Engineering), consistent with data from the 10,000 PhDs Project (Reithmeier et al., 2019).

Regarding the language of applications, the committees have noted via documents reviewed that the number of submitted French applications remains low (see Figure 10: Number of Banting PDF applications and success rate by language, 2019-20). The communication strategy has been updated to include components on increasing promotional activities towards diverse groups and languages. An analysis in 2018 of bias against language (which had insufficient data for CIHR and NSERC) found no evidence that funding decisions depended on language for SSHRC. Between 2014-15 and 2016-17, document review indicated that applications with French as the preferred language were recommended less frequently for funding than their English counterparts. However, starting in 2017-18 to 2020-21, applications with French as the preferred language had a higher success rate relative to total applications, when compared with the success rate for English applications, across all agencies.

There is evidence of evolving Indigenous considerations within program documentation. Several requirements have been introduced during the evaluation period for Indigenous applicants and research, including flagging research on application as involving Indigenous peoples, informing host institutions of appropriate principles, and consultation of existing guidelines during the selection processFootnote 66. Program management has kept Indigenous considerations as continuous actions between TAP meetings. Thus, steps have been taken to improve assessment of application for Indigenous candidates; however, there have been a limited number of applicants to the Banting PDF program in recent years who identified as IndigenousFootnote 67.

Respondents who self-identified as visible minorities were also slightly more likely to rate several aspects of the Banting PDF application process as posing a barrier, including choosing an appropriate Banting PDF supervisor (25%, n = 36) and restrictions related to holding other funding (22%, n = 32), compared to those who did not identify as visible minorities (16%, n = 73, and 13%, n = 62, respectively; barriers survey data). Obtaining reference letters was noted as a barrier to application for a greater proportion of survey respondents who identified as having a sexual orientation other than heterosexual (37%, n = 48) and applicants who were non-Canadian citizensFootnote 68 (34%, n = 82)Footnote 69. One Banting PDF recipient included in the case studies with a disability felt that there was more that the Tri-agencies could do in terms of the structure of the application and training of reviewers which would allow for EDI considerations to be disclosed, and two interviewees (one host institution official, one supervisor) commented that the application process presented a greater burden for students with disabilities.

Review Criteria

In terms of the assessment of review criteria, a qualitative analysis of leadership in the Vanier CGS program, the Banting PDF program, and CIHR’s Foundation Grant programs found some differences in the way leadership was described for male vs. female applicants (CIHR, 2018). The varying definitions of leadership between men and women could further complicate the issue of the lack of a clear definition of leadership within the Banting PDF program.

According to the barriers survey, respondents who identified as having a disability were more likely to report that assessment of both research excellence and leadership were barriers to their Banting PDF application (40%, n = 20) compared to those who did not self-identify a disability (26%, n = 135 and 31%, n = 160, respectively), and those who self-identified as visible minorities were considerably more likely to rate the assessment of leadership (41%, n = 52) as a barrier compared to those who did not identify as visible minorities (13%, n = 62).

Most interviewed selection committee members agreed that the Reviewer Evaluation Guide is equally applicable to international applicants and citizens or permanent residents of Canada. However, one committee member indicated that applicants from Canada tend to have access to a lot more funding than many international students, making the former more competitive for the Banting PDF. Some NSERC and SSHRC focus group participants also remarked that financially disadvantaged individuals face systemic barriers that are further perpetuated by the standards of success and leadership dictated by academia (e.g., holding summer studentships instead of summer employment, and gaining an advantage from having received previous funding).

The Banting PDF has been delivered in a cost-efficient manner

The delivery of the Banting PDF program during the evaluation period from 2014-15 to 2020-21 has been cost-efficient, as evidenced by the total program administrative costs as a percentage of total program expenditures (ACTE; See Table 3. Banting PDF administrative costs as a percentage of total program expenditures). The ACTE has increased by 27% (from 4.4% to 5.6%) between 2014-15 and 2019-20. At its highest value of 5.6%, the percentage is close to the CIHR overall administrative cost ratio (6%). From 2019-20 to 2020-21, the ACTE dropped by 18% to reach 4.6% of total expenditures due to the COVID-19 pandemic, which resulted in a significant decrease in direct operating and maintenance costs driven primarily by the move from in-person to virtual selection committee meetings. For example, the Tri-agencies during 2020-21 delivered selection committee meetings virtually, which reduced the Direct Operating and Maintenance costs from $94,010 in 2019-20 to $5,740 in 2020-21 (VBS data).

The administrative costs increased from $447,220 in 2014-15 to $577,180 in 2019-20 resulting from an increase in direct salaryFootnote 70 ($278,260 to $345,120), employee benefit plan costs ($55,650 to $93,180), accommodation costs ($36,170 to $44,870), and direct operating and maintenance costs ($77,130 to $94,010). Although overall delivery remains cost-efficient with the ACTE below the reference value, the significant increase observed during the evaluation period suggests a loss in cost-efficiency in the delivery of the Banting PDF. The increase in the delivery costs, while the number of recipients per year remains unchanged, has led to an increase in the administrative cost per award. From 2014-15 to 2019-20, the administrative cost per Banting PDF funded increased by 29%, from $3,194 to $4,123. This notable increase indicates that the management of the Banting PDF has become more expensive over the years, driven by the increase in employee benefit plan costs (67%), direct salary (24%), accommodation costs (24%), and direct operating and maintenance costs (22%). The increase in employee benefit plan costs has resulted from the change of the employee benefit plan rate from 20% of direct salary to 27% beginning 2019. However, the decline observed in 2020-21 due to COVID-19 suggests that using innovative means, such as using virtual selection committee meetings, can reduce delivery costs.

Performance: To what extent has the COVID-19 pandemic had an impact on the delivery and performance of the Banting PDF program?

Key Findings:

The Banting PDF was flexible in responding to the COVID-19 pandemic

According to documents reviewed, the Tri-agencies were adaptable and employed a flexible approach with the design and delivery of the Banting PDF program in response to the COVID-19 pandemic. In April 2020, the federal government announced new funding of $291.6M for traineesFootnote 71 whose research and educational programs were affected due to restrictions imposed by the COVID-19 pandemic (Government of Canada, 2020). Additionally, postdoctoral trainees were given the option to defer the start of their award or request an unpaid interruption for reasons related to the pandemic, and Banting PDF recipients whose awards ended between March and August 2020 were offered a four-month extension.

Most interviewed members of program management and host institution officials identified minimal impacts to the program due to COVID-19. However, some unfunded applicants and recipients felt they were negatively impacted because the window of eligibility (n = 8) and the duration of the award (n = 6) were too short due to the impact of COVID-19 (barriers survey). Although the Tri-agencies made some changes to accommodate recipients affected by the pandemic, document review indicated that recipients with Banting PDFs ending outside the eligible dates could also have benefitted from these changes. A decision was made to change the face-to-face selection committee meeting to a virtual format to reduce the impact of the pandemic on program delivery. However, this change raised some concerns noted in documents reviewed, such as capacity to chair, timing for review of each application, and engagement amongst committee members. All selection committee members interviewed noted that this change had a detrimental impact on meetings, and most agreed that face-to-face meetings are preferred.

The COVID-19 pandemic had a negative impact on recipients’ ability to conduct research

Consistent with impacts across the research ecosystem, the COVID-19 pandemic had a negative impact on Banting PDF recipients’ ability to conduct research. Document review, key informants across groups, and Banting PDF recipient respondents to the barriers survey all noted restricted laboratory access due to COVID-19 which reduced recipients’ ability to conduct their research. Supervisors and recipients interviewed noted reduced opportunities for collaboration, and supervisors and recipients (via the barriers survey) noted reduced ability engage with supervisors as negative impacts of the COVID-19 pandemic. One case study recipient reported difficulty adapting to virtual teaching and establishing a lab during the pandemic.

Conclusions and Recommendations

Conclusions

Relevance

There is a continued need for postdoctoral support that aims to attract, retain, and support the training of top-tier Canadian and international early-career researchers to position them for success as research leaders. As per the program authorities, the Banting PDF program is intended to meet this need. The Banting PDF program exists within a competitive global PDF environment where its objectives make it a unique tool for the federal government to attract top international research talent to Canada.

The Banting PDF program complements other prestigious federal programs within a suite of elite federal research capacity development programs [e.g., Vanier Canada Graduate Scholarships (Vanier CGS), Canada Research Chairs (CRC), Canada Excellence Research Chairs (CERC)]. At the postdoctoral level, there are concerns of overlap between the Banting PDF program and other Tri-agency PDF programs (i.e., agency-specific PDFs) in terms of training support provided and having too many programs that are intended to achieve similar objectives.

There is a role for the federal government and granting agencies in attracting and retaining top-tier postdoctoral trainees to increase the supply of highly qualified researchers and enhance Canada’s research capacity to foster Canada’s economic and social progress. The Banting PDF program contributes to this objective, although the extent to which it contributes is limited due to the small number of awards (i.e., 70 PDFs are awarded annually). The Banting PDF is aligned with federal government and Tri-agency strategic priorities that aim to attract, retain, and develop talent to strengthen the Canadian research capacity.

Performance

The Banting PDF program is producing its outputs and achieving its expected immediate outcomes specific to Banting PDF recipients. However, evidence suggests that unfunded applicants who received other sources of postdoctoral support are achieving similar outcomes. Therefore, it is not clear that the achievement of key outputs and immediate outcomes can be solely attributed to the Banting PDF.

Canadian researchers and institutional representatives consulted as part of the evaluation are aware of the Banting PDF as an attractive and competitive award. Available data indicates that awareness outside of Canada is limited. Banting PDF recipients and unfunded applicants perceive the Banting PDF as prestigious, and recipients report that receiving the award has led to research and other professional opportunities during their Banting PDF.

The Banting PDF program is selecting excellent candidates in Canada, although it is unclear whether these candidates are better than their unfunded applicant peers, or whether this is largely a reflection of the assessment criteria for “top-tier”. The Banting PDF program has seen increased uptake by international applicants; however, the program has not met its target of 50% international nominations. Available data indicates that the Banting PDF program is not effective in attracting international candidates from outside Canada, although it may play a role in retaining talent during their Banting PDF.

While the influence of the Banting PDF varies, recipients are devoting most of their time to research during their fellowship, consistent with norms for postdoctoral training. The Banting PDF may provide recipients with increased autonomy to conduct their research compared to other sources of PDF support.

Banting PDF recipients are establishing national and international collaborations and are engaged in a range of leadership development activities and additional training, but these outcomes cannot be clearly attributed to the Banting PDF. Although Banting PDF recipients reported greater professional leadership compared to unfunded applicants, both groups report comparable improvement in research, teaching, and service leadership ability.

The Banting PDF program is achieving its intermediate outcomes; however, the incremental contributions of the Banting PDF in relation to other PDF supports appears limited. Banting PDF recipients are recognized as leaders in their fields and are demonstrating leadership outside research (e.g., active community outreach in promoting research, production of non-academic books). However, Banting PDF recipients are recognized by their supervisors as having inherent leadership potential and would likely have achieved those outcomes without the Banting PDF.

Findings indicate that key design features of the Banting PDF program may be limiting the effective achievement of intended outcomes. The Banting PDF application process is perceived to be administratively heavy, in terms of time and effort required, for both applicants and institutions. Institution and supervisor support during the Banting PDF application process varies and can pose challenges for applicants. Program requirements, including synergy with host institution and demonstration of leadership and research excellence, may limit the attraction of top-tier international talent in favour of those who are more advanced in their research careers and already connected to the institution. While improvements have been made to the selection process at the Tri-agency level, concerns remain around the lack of transparency in the institution nomination and review processes, particularly synergy with the host institution and research excellence and leadership.

The Banting PDF’s value is sufficient for most recipients in Canada but is not competitive with some key international programs. Most recipients felt that the duration of the Banting PDF award was sufficient, although some felt that the award could be longer. Removal of the 25% cap on recipients holding Banting PDFs abroad appears to have led to a slight increase in recipients taking the award abroad. It is not clear that support for leadership development has increased despite this recommendation from the previous evaluation. The Banting PDF is unique among comparable PDF programs reviewed in its commitment to EDI in the program’s assessment, although improvements can still be made for equity-deserving groups. For example, reviewing program features that have been identified as potential barriers, such as the window of eligibility and mobility requirement.

Practices related to performance measurement, including collection and use of EDI data, linkage between administrative data and performance measurement tools, and some structural elements of these tools (e.g., length, inconsistency of scales), present challenges in measuring the performance of the Banting PDF program.

Recommendations

  1. The Banting PDF program should consider adjusting key features of the award (e.g., award value, allowances and duration) to remain prestigious and competitive in comparison to other international programs.
  2. The Banting PDF program needs to clarify its objective of attracting international candidates to meet the program’s target of 50% international nominations.
  3. The Banting PDF program should take steps to increase awareness of the award among the international research community, including enhancing current activities and the monitoring of these activities.
  4. The Banting PDF program should develop specific leadership development and mentorshipprogram elements during the tenure of the fellowship better develop Banting PDF recipients’ leadership potential and position them as future research leaders.I
  5. The Banting PDF program should improve application and selection processes to better ensure transparency, including:
    • Define and improve measurement of leadership and research excellence using an EDI lens in order to ensure alignment with the Tri-agencies’ strategic priorities related to research excellence and EDI.
    • Reduce the weight of the synergy with host institution review criterion.
    • Review program features, including the window of eligibility and mobility requirement, to ensure that barriers are reduced for equity-deserving groups.
  6. The Banting PDF program needs to improve end of award reporting to improve assessment of program performance and barriers to access.

Appendix A: Tables and Figures

Table 1. Number of Agency-specific PDFs awarded by year, 2015-2020

  2015 2016 2017 2018 2019 2020 Total
CIHR 154 155 131 116 140 148 844
NSERC 183 199 192 204 194 110 1082
SSHRC 172 123 122 129 182 133 1003

Note: competition year is different for CIHR vs NSERC/SSHRC: 2015 in this case is for NSERC and SSHRC, it is equivalent to 2014-15 for CIHR

Note: Due to lack of complete data across agencies at the time of request, 2021 was not included.

Source: CIHR Funding Analytics; NSERC Governance, Risk and Compliance; SSHRC Research Training Portfolio

Table 2. Proportion of Banting PDF applicants (funded and unfunded) already at host institution at the time of application, by citizenship, 2014-15 to 2019-20

  Citizenship 2014 2015 2016 2017 2018 2019 Total
Already at host institution at time of applicationFootnote * Canadian & Permanent Resident 272 (64%)Footnote * 215 (50%) 188 (51%) 210 (53%) 218 (55%) 180 (54%) 1283 (55%)
International 104Footnote ** (67%) 80 (44%) 84 (46%) 106 (50%) 101 (47%) 112 (51%) 587 (50%)
Total 376 (65%) 295 (48%) 272 (49%) 316 (52%) 319 (52%) 292 (53%) 1870 (53%)
Total number of Banting PDF applicants Canadian & Permanent Resident 425 431 368 394 398 333 2349
International 156 182 183 212 213 220 1166
Total 581 613 551 606 611 553 3515

Note: Calculation based on the proportion of total number of Banting PDF applicants whose institution at time of application and host institution match

Source: Vanier-Banting Secretariat

Table 3. Banting PDF administrative costs as a percentage of total program expenditures

  2014-15 2015-16 2016-17 2017-18 2018-19 2019-20 2020-21
Total award expenditures (a) $9,800,000 $9,800,000 $9,800,000 $9,800,000 $9,800,000 $9,800,000 $9,800,000
Total administrative costs (b)Footnote * $447,217 $445,291 $460,376 $499,457 $526,808 $577,177 $475,246
Total program expenditure (c=a+b) $10,247,217 $10,245,291 $10,260,376 $10,299,457 $10,326,808 $10,377,177 $10,275,246
Percentage of administrative costs to total expenditures (d=(b/c)% 4.4% 4.3% 4.5% 4.8% 5.1% 5.6% 4.6%

Source: Award expenditure and administrative costs data from Vanier Banting Secretariat and CIHR Finance.

Figure 1: Banting PDF Logic Model

Figure 1 long description

The Banting Postdoctoral Fellowships program was launched by the Federal Government in 2010 with the vision: “To attract and retain top—tier postdoctoral talent, both nationally and internationally, to develop their leadership potential and to position them for success as research leaders of tomorrow, positively contributing to Canada’s economic, social and research-based growth through a research-intensive career.”

The following components describe the Banting Postdoctoral Fellowships program logic model:

Activities:

  • Program management and administration.
  • Development and dissemination of program branding, communications and marketing strategies.

Outputs:

  • Top-tier applicants are attracted and recruited.
  • Banting fellowships awarded.
  • Communication/promotion materials produced.
  • Communication/promotion materials disseminated.
  • Client services delivered, program management reports produced, delivered and disseminated.

Immediate Outcomes:

  • Banting fellows receive enhanced research training
  • Banting fellows devote the majority of their time to conduct research
  • Banting fellows establish national and international collaborations
  • Increased national and international awareness of Banting PDFs as an attractive and competitive award

Intermediate Outcomes:

  • Banting fellows are retained and undertake research careers in Canada
  • Banting fellows attain leadership positions in organizations
  • Banting fellows demonstrate leadership
  • Banting fellows are recognized as representatives of Canadian research excellence

Ultimate Outcomes:

  • Canada’s research enterprise has a reliable supply of highly qualified/trained researchers
  • Canada is a destination of choice for quality research training

Assumptions:

  • Research institutions are aware of objectives and nature of B-PDF program.
  • Research institutions engage and participate as intended.
  • Attraction and retention are important to all activities and outcomes.

External Influences:

  • Government of Canada budget & priorities
  • Budget and priorities of research institutions.
  • Canadian & global economic climate
  • Priorities and need in research communities

The following should be noted:

Research careers can be undertaken in a variety of sectors such as Industry, Government, Academia, Non-profit organizations and Health.

The program’s performance measurement strategy separates leadership into three broad categories, each with its own characteristics and activities: Research Leadership; Academic Leadership; and, Service Leadership (see Mazutis, Morris and Olsen, 2011). While the first two categories are relatively straightforward, Service Leadership is not. For university faculty, “service” primarily means participating in departmental or institutional committees. However, the performance measurement strategy also includes activities such as advising students, mentoring junior colleagues, or becoming involved in community civic groups, agencies and organizations.

Figure 2: Applications by citizenship and location of applicant, 2014-2020

Figure 2 long description
  Canadians and permanent residents (retained) Canadians and permanent residents (abroad) Internationals (attracted) Internationals (retained)
2014-15 215 206 41 115
2015-16 216 215 86 96
2016-17 199 169 86 97
2017-18 186 208 82 130
2018-19 201 197 89 124
2019-20 179 154 85 135

Note: Data are based on location of applicant’s institution at time of application. “Retained” refers to those already in Canada at time of application who remained in Canada.

Figure 3: Extent of improvement in research leadership activities for Banting PDF recipients (during the Banting PDF) and unfunded applicants (during their PDF)

Figure 3 long description
  Recipients Unfunded Applicants
Developing theoretical knowledge of the discipline 3.93 3.73
Developing analytical techniques/experimental methods 3.78 3.97
Coordination of research resources (laboratory equipment, instruments, etc.) 2.95 3.24
Writing manuscripts 4.24 3.76
Presenting findings to different audiences 3.93 3.61
Interdisciplinary research 3.42 3.06
Collaborative research with industry and/or government researchers 2.11 2.06
Earning presentation/paper/research awards 2.96 3.03

Source: Banting End of Award Report data and Applicant Equivalent survey

Figure 4: Extent of improvement in teaching leadership activities for Banting PDF recipients (during the Banting PDF) and unfunded applicants (during their PDF)

Figure 4 long description
  Recipients Unfunded Applicants
Communications/presentations 3.35 3.09
Supervising students 3.27 3.52
Developing course materials 2.17 2.16
Developing teaching methods 2.15 1.87
Teaching a university course 2.15 2.07

Source: Banting End of Award Report data and Applicant Equivalent survey

Figure 5: Extent of improvement in service leadership activities for Banting PDF recipients (during the Banting PDF) and applicants (during their fellowship)

Figure 5 long description
  Recipients Unfunded Applicants
Taking on an administrative role (e.g., as an editor for a journal, or as a representative to a faculty/departmental/institutional committee) 2.43 2.1
Taking on a leadership position with a student group (e.g., union, association) 1.79 1.82
Participating in institutional associations 2.03 1.9
Organizing or participating in volunteer activities 2.23 2.27
Participating in campus media 2.28 1.48
Participating in civic initiatives 1.92 1.31
Taking on a leadership role in campus activities (e.g., athletic, artistic or cultural) 1.66 1.48
Educating youth or community groups 1.6 1.83
Promoting awareness for a particular cause 1.88 1.55
Developing and executing a fundraising campaign 1.94 1.24
Taking on a leadership position in a community organization 1.62 1.39
Fulfilling a position in a professional organization 1.95 1.86

Source: Banting End of Award Report data and Applicant Equivalent survey

Figure 6: Sector of employment for Banting PDF recipients and applicants at end of their fellowship and five year follow-up

Figure 6 long description
  End of Fellowship Five Year Follow-Up
Recipients (n = 188) Unfunded Applicants (n = 132) Recipients (n = 132) Unfunded Applicants (n = 113)
University 82% 62% 89% 69%
Industry 3% 14% 3% 17%
Hospitals and other health care providers 4% 9% 2% 3%
Government 3% 5% 2% 6%
Non-profit organization 4% 4% 2% 5%
Other 3% 6% 2% 0%

Source: Banting End of Award Report, Banting 5-Year Follow-Up data and Applicant Equivalent surveys

Figure 7: Proportion of Banting PDF recipients employed within and outside Canada, at end of their fellowship and five-year follow-up

Figure 7 long description
  End of Fellowship Five Year Follow-Up
Recipients (n = 80) Unfunded Applicants (n = 107) Recipients (n = 129) Unfunded Applicants (n = 188)
In Canada 70% 61% 65% 63%
Outside Canada 30% 29% 35% 37%

Source: Banting 5-Year Follow-Up data and Applicant Equivalent survey

Figure 8: Percentage of employed Banting PDF recipients and applicants in tenured/tenure-track positions

Figure 8 long description
  Recipient (n = 132) Unfunded Applicant (n = 113)
Prefer not to answer/Missing 9% 8%
Other employment 24% 44%
Tenure/tenure-track 67% 48%

Note: proportions (recipient vs. unfunded applicant at end of fellowship and at five-year follow-up, and recipient at end of fellowship vs recipient vs five year follow-up) do not significantly differ (using z-test of proportions)

Source: Banting End of Award Report, Banting 5-Year Follow-Up data and Applicant Equivalent survey

Figure 9: Number of Banting PDF applications and success rate by sex, 2014-15 to 2019-20

Figure 9 long description
  Male Female
Number of Applications Success Rate Number of Applications Success Rate
2014-15 293 14% 288 10%
2015-16 343 12% 269 10%
2016-17 298 12% 237 12%
2017-18 308 11% 274 11%
2018-19 308 13% 286 9%
2019-20 270 15% 258 10%

Source: Vanier-Banting Secretariat

Figure 10: Number of Banting PDF applications and success rate by language, 2019-20

Figure 10 long description
  English French
Number of Applications Success Rate Number of Applications Success Rate
2014-15 526 12% 55 9%
2015-16 557 12% 56 5%
2016-17 502 13% 49 8%
2017-18 551 11% 55 20%
2018-19 554 10% 57 21%
2019-20 509 12% 44 18%

Source: Vanier-Banting Secretariat

Appendix C: Methodology – Additional Details

Additional details about the multiple lines of evidence and methodology used in the evaluation are presented in this appendix.

Environmental Scan & Document Review

The environmental scan focuses on the relevance and performance of the Banting PDF program and provides information to address the following evaluation sub-questions:

The environmental scan involved a review of program documentation from CIHR, NSERC, and SSHRC and from other national and international PDF programs comparable to the Banting PDF. Specifically, this included program information related to, but not limited to, the following: objectives, eligibility requirements, key assessment criteria, award value per annum, duration, location, number of new awards per annum, and EDI considerations.

The document review component involved the analysis of relevant Vanier Banting PDF Secretariat documents (e.g., TAP-MC and TAP-SC meeting books, communication reports, webpages) as well as relevant grey literature, such as reports disseminated by the three federal granting agencies (CIHR, NSERC and SSHRC), and other relevant federal government documents. The specific documents reviewed (n = 78) were selected based on relevance and recency of available documents provided to the Evaluation Unit from the Vanier Banting PDF Secretariat.

Cost-Efficiency Analysis

The cost-efficiency analysis informed the cost-efficiency of the program delivery from 2014-15 to 2020-21, covering the evaluation period. The efficiency analysis assessed the evolution of the administrative costs as a percentage of total expenditures (ACTE) during the evaluation period. Total expenditures were determined by combining the total award expenditures and total administrative costs. The total administrative costs include the direct salary costs of the Tri-Agency staff involved in the implementation of the program, 20% to 27% percentage of the employee benefits plan (20% was considered up to 2018 and 27% after), 13% of accommodation plan and direct operating and maintenance costs which are non-salary costs. The ACTE to total expenditures is the share of management costs in total expenditures. It is thus obtained by dividing the total administrative costs by the total expenditures.

Administrative Data Analysis & Funding History Analysis

A review of Banting PDF program records and administrative data from CIHR, NSERC, and SSHRC provided information on application and success rates, and program expenditures, as well as applicant characteristics of such as affiliated institution, preferred language, gender, and funding history, which helped contextualize the program.

A separate analysis was conducted of all available funding data for Tri-agency grants and awards received by Banting PDF recipients and unfunded applicants of Banting PDF who had also received an agency-specific PDF from CIHR, NSERC, and SSHRC. As the first competition year for the Banting PDF was 2010, cohorts included Banting PDF and Tri-agency PDF recipients from the 2010-2020 period, and additional funding from competition years 2000 to 2020 was included in the funding history. This allowed the funding history data to capture any awards received by the first Banting PDF cohort (and comparable agency-specific PDF cohort) prior to their application, assuming that those at the fellowship level could have previously received a Master’s, Doctoral, and/or additional postdoctoral fellowship awards, as well as grants and awards achieved after their Banting PDF or agency-specific PDF, to the extent possible depending on how recently they received their fellowship. It is important to note that this funding history only included Tri-agency funding, and therefore could not capture an individual’s complete funding profile, nor would it capture funding received by individuals based outside Canada. Analysis of funding history data involved z-tests of proportions; t-tests of average funding amounts; and logistic and linear regressions with awards/grants received and award/grant value as outcome variables, Banting PDF application status (recipient vs. unfunded applicant) as predictors, and language and gender used as moderator variables in separate analyses. Analyses were conducted in SPSS (using the Hayes Process tool for moderation models). Alpha was set at 0.05 for statistical tests, although “marginal” significance close to the threshold of 0.05 was noted.

Banting PDF End of Award Report (BEAR), Banting PDF Five-Year Follow-Up (B5) & Applicant Equivalent Surveys

The performance measurement tools targeting Banting PDF recipients included the following: the BEAR and B5. The BEAR was a survey meant to collect data from applicants immediately following completion of their fellowship. The BEAR was typically sent out to recipients at three points during the year (spring, summer, fall/winter), as their fellowship was ending. Data for the BEAR that were utilized for the purposes of this evaluation consisted of 234 respondents (from a population of 327, response rate: 71.6%), including respondents from the earliest BEAR cohort examined in this evaluation (i.e., those who applied to the Banting PDF in competition year 2014-15 and completed their fellowship in 2017), up to the most recently collected data for Banting PDF recipients who completed their fellowship in 2020. Recipients who responded to the BEAR were most frequently awarded their Banting PDF from CIHR (34%, n = 79), followed by NSERC (33%, n = 77), and SSHRC (31%, n = 74). These data were collected prior to the evaluation (with the most recent 2020 list of eligible recipients collected during the early stages of the evaluation) by the CIHR Results and Impact Unit.

The B5 was implemented in 2018, and was meant to measure longer term outcomes by tracking Banting PDF recipients five years after completion of their Banting PDF. Thus, the first cohort of Banting PDF recipients, starting with those who held the Banting PDF from 2011-12 to 2012-13 who received the B5 in 2018. The B5 was sent out once a year to all Banting PDF recipients who had completed their fellowships within the calendar year five years prior. B5 data utilized for the purposes of this evaluation included 136 respondents (from a population of 282, response rate: 48.2%), representing the earliest B5 cohort from 2018, up to and including eligible B5 respondents for 2021 (i.e., those that concluded their Banting PDF in 2016). Data collection for 2018 and 2019 had been conducted by the Results and Impact unit, with 2020 data collection delayed due to the COVID-19 pandemic. The CIHR Evaluation Unit administered the survey to the 2020 and 2021 B5 cohorts in order to make the most complete data available for the evaluation. Twenty-four percent of recipients who responded to the B5 were awarded their Banting PDF through SSHRC (n = 33), followed by 19% awarded from CIHR (n = 26), and 16% awarded by NSERC (n = 22).

In order to provide a counterfactual to the Banting PDF recipient cohorts targeted in the BEAR and B5, two equivalent applicant surveys were created as part of the evaluation. The aim of these surveys was to assess career outcomes and training experiences from fellowships other than the Banting PDF of individuals who had unsuccessfully applied to the Banting PDF program, to compare against the outcomes and experiences of Banting PDF recipients. These surveys were also meant to target similar cohorts of unfunded applicants. Thus, the BEAR equivalent targeted a more recent cohort of unfunded applicants from competition years 2014-16 (i.e., those expected to have “finished” an equivalent fellowship in 2017-19; n = 203 from a sampling frame of 1459, response rate = 13.9%); and the B5 equivalent targeted an earlier cohort of unfunded applicants from competition years 2011-13 (i.e., those expected to have “finished” an equivalent fellowship in 2013-15; n = 116 from a sampling frame of 1382, response rate = 8.4%). Those who responded to the BEAR applicant equivalent survey had most frequently applied through CIHR (43%, n = 84), followed by 35% applying through NSERC (n = 72), and 23% to SSHRC (n = 47), while those who responded to the B5 applicant equivalent survey had most frequently applied through NSERC (40%, n = 46), followed by CIHR (34%, n = 39), and SSHRC (22%, n = 25).

Survey responses were analyzed using frequencies (counts, percentages) and descriptive statistics (means, standard deviation), with t-tests of mean differences and z-tests of proportions used where appropriate. T-tests were chosen as the primary statistical test for 5-point Likert-type scales, in particular, treating the scales as interval/ratio measures. Alpha was set at 0.05 for statistical tests, although “marginal” significance close to the threshold of 0.05 was noted.

Bibliometric Analysis

Bibliometrics indicators are recognized as valuable measures of scientific productivity and quality and therefore were used to address the following evaluation questions:

The bibliometric study, conducted by the Observatoire des sciences et des technologies (OST), examined the scientific productivity of Banting PDF applicants, using reconstituted publications files for both successful and unsuccessful applicants who have received Banting PDF and agency-specific PDFs, respectively – over all available years following Banting PDF application for the 2010-11 to 2013-14 cohorts, and for the three-year period preceding their Banting PDF application for the 2018-19 to 2020-21 cohorts. It also examined differences in co-authorships. Comparisons were made between these two broad groups of applicants: (a) recipients of the Banting PDF, (b) unsuccessful Banting PDF applicants funded by agency-specific PDFs from any of the Tri-agencies.

The bibliometric data is drawn from the Canadian Bibliometric Database (CBDTM) built by the Observatoire des sciences et des technologies (OST) by using Clarivate Analytics’ Web of Science (WoS). The WoS includes three databases (the Science Citation Index Expanded™ [SCIE], the Social Sciences Citation Index™ [SSCI], and the Arts & Humanities Citation Index™ [AHCI]) covering, in 2020, more than 12,000 journals from all fields of knowledge. While the bibliometric database offers a good coverage of the publication output for the research fields of health, natural sciences and engineering, this coverage is far less complete for the fields of social sciences, and even less for the humanities. To mitigate this limitation, two additional analyses were conducted, on a subset of publications for each agency. The first was an analysis of publications that fell only within the SSCI and SCIE Databases; that is, publications that were indexed in the AHCI were excluded as there was concern about the completeness of outputs indexed in this field. The second was an analysis of publications that were indexed in WoS only as belonging to the top three disciplines for each agency, in terms of the disciplines that were represented by the greatest number of publications. This analysis was undertaken in an attempt to “standardize” bibliometric analysis of outputs across the three agencies by including disciplines that were indexed in WoS the most consistently. These two sub-analyses were conducted for all metrics across all three agencies.

Case Studies

As part of the 2020 evaluation of the Banting PDF program, nine case studies of Banting PDF recipients from competition years 2012-13 and 2013-14 were conducted. Cases were selected based on achievement of a tenure-track academic position or a leadership position in an organization outside of academia, which is one of the intended intermediate outcomes of the Banting PDF program. The purpose of the case studies was to provide in-depth information about the experience of the Banting PDF and the role that recipients felt the Banting PDF played in their career path and outcomes. Each case study included the Banting PDF recipient (case), their Banting PDF supervisor, and one or two “stakeholders” in their research: a colleague, collaborator or trainee who could speak to the impact of the recipient’s research. Recipients were only eligible for inclusion in the case studies if their supervisor and several stakeholders were available to be interviewed; however, in two cases, supervisors were ultimately not accessible and thus an additional stakeholder was interviewed in place of the supervisor. Cases were balanced across Tri-agency (3-CIHR, 3-NSERC, 3-SSHRC), gender (5-women, 4-men), and two cases featured Banting PDF recipients who were working outside academia (provincial government and intergovernmental organization).

Barriers to Access and Participation Survey

A commitment outlined in the evaluation design report was the application of a SGBA+ lens to the evaluation. One line of evidence proposed in the evaluation as a means of exploring EDI considerations in greater depth as part of the SGBA+ was a journey map of equity-deserving groups. However, as the plan for this line of evidence was developed, which included consultation with CIHR’s EDI Strategy team, it became clear that a journey map (i.e., generation of “personas” to represent equity-deserving groups) was not an appropriate or accurate way to either collect or represent the experiences of these individuals. Thus, the journey map evolved into the barriers to access and participation survey. The barriers survey intended to reflect the experiences of all applicants who have experienced barriers to application to the Banting PDF program, with a particular focus on members of equity-deserving groups within the population of applicants, both funded and unfunded. In addition, barriers in the experience of the fellowship were represented, for recipients of the Banting PDF program. Specifically, these equity-deserving groups included: women, members of visible minority groups, Indigenous populations, members of the LGBTQ+ community, individuals with disabilities, and international applicants. All applicants, regardless of self-identification as members of equity-deserving groups, were invited to participate and identify whether they experienced barriers to application and during the fellowship (for recipients only).

Survey questions included Likert-type response scales (5-point) to indicate the extent to which the respondent had experienced a particular issue to be a barrier (extent scale: Not at all, Slight Extent, Moderate Extent, Great Extent, Very Great Extent), with an option for each item to follow up and provide an open-ended comment. Most findings reported in the evaluation refer to the proportion of respondents who identified an item as a barrier to a Moderate, Great, or Very Great Extent (i.e., at least a moderate barrier). The items included on the survey were those previously identified in the literature as potential barriers, across four stages of application: eligibility requirements and fellowship features (award value and duration), application process, institution nomination process, and decision and review process; plus elements of the fellowship itself were included (e.g., supervision, opportunities for collaboration).

The barriers survey sampling frame included applicants to the Banting PDF program (both funded and unfunded) from competition years 2017-21, as the focus was on the application stage and to a lesser extent the fellowship stage. Thus, recent cohorts were included to capitalize on the recency effect of application experience, including those who had been impacted by the covid-19 pandemic. The survey was conducted online, using the Voxco platform. The sampling frame included 2346 unfunded applicants and 350 recipients. The final sample consisted of 475 unfunded applicants (response rate: 20.2%) and 167 recipients (response rate: 47.7%). Recipients who responded to the barriers survey were most frequently awarded their Banting PDF through CIHR (37%, n = 61), followed by SSHRC (33%, n = 55) and NSERC (31%, n = 51), while a similar number of applicants who responded to the barriers survey had applied through NSERC (35%, n = 166) and CIHR (35%, n = 165), followed by SSHRC (30%, n = 144).

Key Informant Interviews

The interviews, conducted by Goss Gilroy Inc., provided insights concerning the relevance of the Banting PDF, in terms of its alignment with the mandates of the Tri-agencies and the alignment with Government of Canada programs and priorities. They also help evaluators assess the performance of the program, by assessing the achievement of the program’s expected outputs, and immediate and intermediate outcomes, and how the design and delivery of the program supports the achievement of the intended outputs and outcomes.

In total, 58 key informant interviews were conducted with 59 individuals from eight respondent groups:

The interviews were approximately 45-60 minutes long, fully confidential and semi-structured. Respondents received an interview guide prior to the interview, to allow them to consider the questions in advance. Interviews were conducted by contractors.

Focus Groups

One of the methods used for this evaluation was conducting Focus Groups with former recipients of agency-specific (SSHRC/NSERC/CIHR) PDFs who were eligible for the Banting PDF, but never submitted a Banting PDF application to the Tri-agencies. The purpose of these focus groups, conducted by Goss Gilroy Inc., was to learn more about their experiences with PDF programs in Canada and internationally, especially regarding the application and selection processes. In particular, the focus groups wanted to help identify why those who would have been eligible to submit a Banting PDF application did not, and what their career outcomes have been to date.

Three focus groups were conducted, one per agency, with 8 participants per focus group. Of the total participants across focus groups, 59% of those who participated identified as women, and 41% as men. (Other gender options were included but did not receive responses.) Almost a fifth of those who completed the survey identified as a visible minority/racialized person, 9% as LGBTQ+ persons, 9% as having a disability, and no one identified as Indigenous. The majority (64%) indicated that English was their first official language spoken. Almost a third (27%) indicated that their first official language was French, and 9% that it was both French and English. These proportions were not broken down further by agency due to small numbers.

Evaluation Limitations and Mitigation Strategies

Limitations Challenges and Mitigation Strategies
Limited counterfactual The recipient and unfunded applicant groups may differ in terms of certain important attributes that existed prior to their fellowship, as it was outside the scope of the evaluation to undertake a fulsome comparison between recipients and applicants prior to application. Thus, differences observed in the subsequent outcomes between past recipients and unfunded applicants could be the result of the Banting PDF or the result of these pre-existing group differences. This difference could also result from the difference between the time when end of award reports are administered to recipients and the time when equivalent surveys are administered to unfunded applicants. To mitigate this limitation, this evaluation of the Banting PDF program also examined the change in outputs (i.e., number of publications) since the application relative to the period prior to the application (pre-post comparison).
Contribution vs. attribution It would be difficult to attribute recipients’ current career status solely to the impact of the Banting PDF. The career trajectory of researchers is complex and young researchers compete to access support at different levels (e.g., doctoral, postdoctoral, Tier I and II Canada Research Chairs) from different funders within Canada and abroad. Therefore, outcomes may be attributable to more than one source of support. The evaluation therefore interprets any findings related to program participants’ outputs and outcomes in terms of the Banting PDF program’s relative contribution (rather than full attribution).

Data availability and inconsistency in reporting of data:

  • EDI data for Banting PDF recipients and applicants and agency PDF recipients are difficult to access due to privacy constraints, which limits analyses possible
  • Different systems for recording data within agencies (for NSERC and SSHRC) and across agencies make it challenging to conduct analyses that require linking of data across funding programs using a unique identifier (e.g., funding history)
  • Data at the institution level of application are unavailable, which limits the ability to measure true attraction and assessment of barriers at this stage of the application. Data measuring international awareness are also limited which impacts this assessment.
  • PM data, while comprehensive, present several limitations (difficulty linking data between PM tools and to other data sources with no PIN, name, or competition year identified; response options and coding of items in some cases are difficult to analyze and interpret)

EDI considerations were built into other lines of evidence (eg., interviews, case studies), and the barriers survey was undertaken with a focus on EDI issues, in order to provide other inputs

CIHR’s Funding Analytics and comparable data analysts at NSERC and SSHRC were consulted to ensure that we had the most accurate data possible and were interpreting the data appropriately. Manual checks of data were undertaken where there were concerns about accuracy and completeness of data.

Proxy data sources for attraction, institution-level data, and awareness were used wherever possible, and these limitations were noted.

The surveys of unfunded applicants replicated the BEAR and B5, which meant that these surveys had some of the same limitations although this approach allowed for the unfunded applicant data to be directly compared to the Banting PDF PM data using the same metrics.

Data from all lines of evidence (surveys, interviews, case studies) were triangulated to make conclusions about the program.
Performance results are based largely on self-reported data (surveys, end of award reports, and interviews), which is subject to potential biases and recall issues Multiple data sources were included to triangulate findings related to performance wherever possible, including more objective measures such as bibliometrics and funding history data.
Ill-defined concepts of “attraction”, “retention”, and “leadership” The concepts of attraction, retention, and leadership are hard to define and operationalize within the context of the program and the evaluation. Thus, measurements of these constructs are bound to be subjective perceptions. Further, these perceptions may be subject to their own, often unconscious biases. The program’s performance measurement strategy is the source of these constructs and the evaluation will assess the effectiveness of the PMS in supporting the planning and conduct of the evaluation. The Evaluation Unit will collaborate with program management and the performance measurement team to develop workable conceptual and operational definitions.
Biased (traditional) measures of research excellence Bibliometric analysis has been criticized as misleading due to the fact that publication rates and citation practices differ across disciplines and even among sub-fields within the same discipline. As a counterbalance to the use of bibliometrics and in line with the DORA recommendations, several qualitative lines of evidence such as key informant interviews, case studies and focus groups are also used to assess research impact.

References

Date modified: