Research evaluation

Evaluating the outcome and impact of the research they support is essential in helping foundations undertake and develop their activities.

Evaluation is necessary in ensuring that foundations support projects and initiatives of the highest quality. It also affords them the opportunity to learn from what they have done, to evaluate the relevance of their activities, and to review their policies and the focus of their programmes for the future.

The EFC Research Forum, with the support of the Lundbeckfonden in Denmark, has organised a series of workshops to discuss the advantages and disadvantages of current evaluation techniques for foundations engaged in research. The presentations from each of the workshops are available below.

One of the results of these discussions is the evaluation guidelines and case-studies below.

Evaluation guidelines and case-studies

Good decisions about what research to fund require firm evidence. A well designed and executed evaluation provides exactly this evidence. For this reason, the EFC Research Forum has supported the production of a series of 4 pamphlets aimed at foundations that are considering how to develop and strengthen their monitoring and evaluation activity. The pamphlets consist of a set of guidelines and case studies and are best seen as as an aid to foundations who are developing their overall evaluation and monitoring strategy. Detailed questions concerning implementation can be followed up using the information in the case studies and the list of suggested reading.

The first pamphlet (Why Evaluate Research?) provides a practical and accessible introduction to evaluation for programme managers and foundation staff, focussing on the rationale for evaluation. It presents some theoretical background and practical considerations to be taken into account before carrying out an evaluation.

The second pamphlet (What to Evaluate) describes the typical objects of an evaluation exercise.

The third pamplet (How to Evaluate) describes in greater detail the standard methodologies and tools that are commonly adopted to conduct evaluations.

The fourth and final pamphlet presents  case studies of research evaluations as currently practiced by foundations in Europe.

The material was developed with the help of senior staff at foundations and research think-tanks, and was validated and modified on the basis of feedack received from foundation staff and other experts at a workshop on 2 June 2010 (see below), a virtual working group and targeted online consultations.

Evaluation workshops

Evaluation: keeping researchers in mind - 9 November 2010, Oxford

 

 Whether aiming to fund the best research or to evaluate its outcomes, there is no substitute for the judgement of experts. However in recent years the ‘gold standard’ of peer review has come under increasing scrutiny as a result of high-profile cases of bias and fraud, accusations that it stifles innovation and risk, and complaints of ‘peer review fatigue’ due to the proliferation of funding programmes and journals.

How can Foundations avoid the pitfalls of peer review while aiming to fund high-risk and innovative research? How can the burden on researchers – both as applicants and peer-reviewers – be minimised while ensuring the credibility the system? What is the role of evidence in peer review and how is it best handled? How can Foundations balance the interests of researchers with the need for accountability to their other stakeholders?

These and other questions were addressed at a 1-day workshop hosted by the University of Oxford. The programme and presentations are available below.

The programme for the day is available below and a short report of the day's discussion can be downloaded here.

How to build an evaluation culture in Foundations: A beginner’s guide to research evaluation - 2 June 2010

Over the past 3 years, the European Forum on Philanthropy and Research Funding has supported a number of workshops designed to exchange practice and views on the evaluation of research. The upshot of this work is the production of a research evaluation information pack (guidelines and case studies) tailored to the needs of foundations that are have just begun their evaluation activities or who wish to embed an evaluation culture in their organisation. Drawing on expertise from within and outside the Foundation sector, this workshop aimed to validate this material and to agree on a plan for its dissemination.

Programme(113 KB)

Jack Spaapen, Building an Evaluation Culture (311 KB)

Tracking impact: looking forward and back - 4 November 2009

Tracking the impact of a research grant is an important element of a foundation’s evaluation activity. It affords foundations the opportunity to ‘measure’ their contribution to scientific progress, to evaluate the relevance of their activities, and to review their policies and the focus of their programmes for the future. It also provides important information useful for foundations to account to boards, donors and the wider public on their activities and how funds are being spent. This workshop looked at how foundations can track/measure impact of past research grants and prepare for tracking the impact of future projects.

Programme (110 KB)

Paula Adam, Evaluating biomedical research impact in Spain and the challenges encountered by the ISOR project  (226 KB) 

Thed van Leeuwen, Application of bibliometric analysis in research funding (140 KB) 

Ian Viney, The Medical Research Council’s new evaluation programme  (701 KB) 

Steven Wooding, RAISSing the game: developing a survey instrument to capture research impacts (.MP4 34 MB)

Foundations and researchers, partners in evaluation - 8 June 2009

Evaluation is a key activity of foundations whether it is undertaken ex-post, in-itinere or ex-ante. A key to the success of foundations’ evaluation activities is working in cooperation and maintaining a link with the researchers they support or have funded in the past. As a result, the focus of this first workshop of 2009 was on researchers. The workshop featured presentations and talks from experts. Time was also dedicated for foundations attending the workshops to share and debate their own practices and the difficulties they face.

Programme(36 KB)

Briony Rayfield,  Career tracking (464 KB) 

Evaluating research outcomes and impact - 5 November 2008

Evaluating the outcome and impact of the research they support is essential in helping foundations undertake and develop their activities. Evaluation is necessary in ensuring that foundations support projects and initiatives of the highest quality. It also affords them the opportunity to learn from what they have done, to evaluate the relevance of their activities, and to review their policies and the focus of their programmes for the future.

Programme (20 KB)

Foundations and Evaluation: Brainstorming Meeting - 17 April 2008

The first working group of the European Forum on Philanthropy and Research Funding focused on evaluating research outcomes and impact. The objectives of the group were to initiate debate between foundations and relevant partners to: identify practices, best practices, obstacles – and their solutions – to evaluation; and evaluate the potential for, and if deemed necessary develop a matrix of common evaluation tools for European research foundations.

Programme (66 Kb)

Liz Allen, Making a difference: approaches to assessment & evaluation (593 KB)

João Caraça, Evaluating Research - Outcomes and impacts by Foundations (583 KB)

Anne Marie Engel, Evaluating Research - Outcomes and Impact (832 KB) 

Marta Lazarowicz, Foundation for Polish Science evaluation activities (746 KB) 

Mats Rolén, Riksbankens Jubileumsfond evaluation activities (273 KB) 

Simon Sommer, Reinvent the Wheel? Making use of endogenous, research-immanent mechanisms of excellence and quality control (38 KB)

Robert Tijssen:

• What is ’’evaluation’’ of research achievements and impacts? (260 KB)
• My Evaluation - how would I go about an evaluation? (58 KB)