Evaluation of Research Proposals

As observed in Chapter 4, the documentation sent to applicants for research funding normally includes detailed guidance on the structure of the evaluation process, the criteria that the funding organization proposes to apply in assessing bids and the weights attached to particular criteria. The Manual of Proposal Evaluation Procedures produced as part of the EC's FP5 documentation covers every step in the Commission's project selection process and gives essential information about checks on eligibility, evaluation criteria, the marking and ranking of proposals, contract preparation and other matters. It can be downloaded from the EC Web site at www.cordis.lu/fp5/management/eval/r_evaluation.htm.

Evaluation procedures under FP6 are likely to show some measure of evolution in response to the objectives of the programme. Nonetheless the basic principles, with their emphasis on scientific and technological excellence, European added value and management performance, are well established within the Commission's research activity and are likely to remain broadly the same. So far as procedural details are concerned, the European Research Advisory Board has made the following recommendations:

• Proposals for funding under FP6 should be evaluated by expert panels, similar to the expert advisory groups in FP5, who will be responsible for assessing the response to calls for expressions of interest.

• High-risk proposals with the potential for high gain should be encouraged.

• Researchers should be able to indicate whether they wish their proposal to be evaluated as basic/exploratory or applied research.

• The difference between exploratory and applied research proposals is to be recognized by the application of different weightings to the evaluation criteria in each case.

• Policy objectives and the European dimension of the problem should be clearly spelt out in the work programme.

• Proposals should indicate why the project needs to be undertaken at a European level in order to succeed.

• Dissemination of results is essential. Where proposals claim to have potential applications, clear indications must be given as to how the target audience will be reached. For research proposals generally, the intention to publish in a high-level journal or to patent should be an absolutely necessary condition, not an evaluation criterion.

• Outreach activities to bring science to the public should be required for larger projects.

• Evaluation panels should have a balanced composition in terms of background (industry and academia), nationality and gender.

• The anonymity of evaluators should be dropped, to reinforce a sense of responsibility.

• Researchers should be able to indicate, with reasons, whom they do not want to evaluate their proposal. This is an option that many organizations in the wider research community allow. The grounds on which they are prepared to entertain a request normally involve conflicts of interest, where the person is a collaborator or supervisor or is otherwise related to the applicant; or personal concerns, when the applicant has reason to believe that the person would be unable to make a fair assessment because of unreasonable bias.

The principles central to EC-related procedures are common to proposal evaluation in most areas of research:

• external peer review and grading by independent experts;

• prioritization of proposals in terms of merit and value, with scientific and technical excellence and relevance to the objectives of the research programme as the prime criteria;

• emphasis on creativity and advancement of knowledge in exploratory research and on innovation in applied research;

• assessment of the bidders' experience and research achievement, plus the strength and credibility of research associations or partnerships;

• factors conducive to bidding success include:

• strong support for the proposal from referees;

• management competence, expressed in a realistic plan for progressing the research;

• sound methodology;

• clear identification of milestones and deliverables;

• exploitability of the research in terms of its downstream benefits;

• effective mechanisms for transmission and communication of project results;

• links with other research and development initiatives;

• access to secured complementary funding from other sources;

• added value and multiplier benefits that reinforce competencies and performance.

Research councils and most of the UK government departments and agencies that fund research programmes publish information on evaluation procedures as part of their guidance for bidders. In assessing quality, expert panels consider principally the relevance of the proposal to the strategic objectives and requirements of the research programme, as well as scientific and technical merit. They look for the factors listed above - in particular a well-defined and competitively priced work plan, proven management ability and a viable method of disseminating the results of the research. The expertise and research infrastructure available to the applicant is taken into account, alongside whatever experience the department may have of previous research work from the same source.

How to annoy a bid evaluator

• Put the bid together in a way that suits you. 'Instructions to tenderers' can safely be discarded. 'Instructions' are just suggestions, aren't they? Anyway, no one really bothers about formalities.

• Don't worry if you have lost any forms that came from the client. Set things out as you choose to - isn't individuality supposed to be important in a bid?

• Leave out items that will take too much time and effort to prepare.

• Don't take seriously anything the client may say about wanting CVs to be just two pages long.

• Don't hesitate to insert marketing brochures into the bid - the more the better, since it may be read by someone who has never heard of you before, and brochures add weight to the document.

• Assume that the people evaluating your bid can be left to comb the text for key information without any guidance on your part.

• Don't bother to number paragraphs, or give titles to tables or figures, and don't even think of cross-referencing them in the text.

• Don't make any effort to be consistent with the phrasing of the bid specification: for example, write about 'the feasibility analysis stage' of the work, when the client talks about 'Phase 1', or call someone the 'project manager' when the client uses the term 'team leader'. While you're about it, why not give the same item of work two or more different titles?

• Ensure that information in tables and figures does not match information in the text.

• Test the evaluator's alertness by building arithmetical errors into tables, and decorating figures with unexplained splashes of colour.

• Avoid committing yourself to a precise statement of deliverables, a clear work programme or an explicit schedule of staff inputs.

• When writing about your team, take care not to specify the role for which each person is nominated or what exactly he or she would do in the assignment.

• Make liberal use of your organization's own internal management jargon.

• Ignore misspellings, incorrect grammar and faulty sentence construction.

• Give the evaluator a much-appreciated opportunity to take exercise by having to rotate the bid 90 degrees to read a table or diagram.