ICES Guidelines for Benchmarks

  

o ICES uses a ‘benchmark process’ as a means to peer-review and incorporate new science for use in provision of all types of recurrent advice. The benchmark process is a critical element in ICES advice to ensure a sound scientific basis. 

o There are three types of benchmark process – Expert Group level, Review, and Full benchmark. The work flow for each type is illustrated in Annex 1.

§ Expert group level: Adequate for small changes to the method that are mainly technical in nature. For ecosystem services and effects advice, the Expert Group review will require coordination among all Expert Groups involved in generating the evidence base for advice.

§ Review: Adequate when addressing one or two larger issues such as changing/correcting an entire data series, or for more substantive revisions to model setting such as changes to age ranges or natural mortality assumptions. This process will, in general, require one year to complete; Expert Groups should carefully consider their workload and their ability to work within the time frame. Peer review will be done by a Review Group composed of members external to the Expert Group(s).

§ Full benchmark: A full benchmark process is a full review of methods, underlying conceptual assumptions, and data; it can also provide the technical basis for the provision of new advice. This review must include an evaluation of the appropriateness of the chosen method. Annex 3 provides guidance for category 1 stock assessments, but the content may be useful for other benchmarks as well. As the full benchmark is a multi-stage process and includes members from outside a specific Expert Group, it is not associated with an Expert Group meeting. It is vital that the process be finished far enough in advance of the Expert Group meeting to allow for full documentation to be ready for the Expert Group, and to allow time for review by the Benchmark Oversight Group (BOG) and ACOM. 

o To plan ahead and utilize network resources in the best way possible, a prioritization process is in place. It considers aspects such as benchmark preparedness, perceived risks, and time since last benchmark, among others. The same factors should be considered by the Expert Group(s) as well as their own resource capacity when embarking on an Expert Group level or review level process. A detailed prioritization scheme is presented in Annex 2.

o The prioritization and approval of full benchmark processes is done by ACOM following recommendations from BOG. A full benchmark process will, in general, require more than one year from proposal to completion and approval.

o As with all aspects of ICES advice, ACOM is ultimately responsible for the outcome of benchmarks and as such has the responsibility to decide which benchmarks are conducted and to review and approve benchmark outcomes. There is an important ownership role for Expert Groups, however, as they will be responsible for implementing the outcome of benchmarks and should be important contributors to benchmarks from start to finish.