“Calibrate with Confidence”, Integrated panel assessment

Event


Date:

Time: 6:30 pm

Please Note: Parking in the North East multi-storey car park is free after 4.30 pm. Access is via Pritchatts Road (Sat Nav B15 2SA)

University of Birmingham

Watson Building, Lecture Theatre A, Birmingham, England

Tuesday March 7, 2017 6:30 pm Tuesday March 7, 2017 6:30 pm Europe/London “Calibrate with Confidence”, Integrated panel assessment University of BirminghamWatson Building, Lecture Theatre ABirminghamEngland “Calibrate with Confidence”, Integrated panel assessment, a talk by Robert MacKay FRS (University of Warwick) Abstract Suppose your company advertises […] Parking in the North East multi-storey car park is free after 4.30 pm. Access is via Pritchatts Road (Sat Nav B15 2SA) Event Link: https://ima.org.uk/4956/calibrate-confidence-integrated-panel-assessment/

“Calibrate with Confidence”, Integrated panel assessment


,

“Calibrate with Confidence”, Integrated panel assessment, a talk by Robert MacKay FRS (University of Warwick)

Abstract

Suppose your company advertises for new employees and tasks a panel with scoring the applicants. Each panel member has time to assess only a fraction of the applicants but each applicant’s dossier is evaluated by at least two panel members. How should the panel combine their scores to remove the effects of differences in use of the scale and different matches between expertise of the assessor and skills of the applicant? Appointments are a big investment for a company.

Or suppose you are in charge of an examination board for which students are assessed on their selection of options. How do you combine the marks to take into account differences in marking standards and length between the options?

Or suppose a research assessment panel of 15 people evaluates 3,000 publications in applied mathematics. Each panel member can evaluate 400 publications so each publication gets two assessors. How should the panel combine their scores to remove the effects of differences in use of the scale and different matches between expertise of the assessor and subject of the publication? The 2016/7 payout to English universities for the results of the REF (for all subjects) was over £1.5B so one ought to try to get it right.

With an undergraduate project student and colleagues from Coventry University, we devised, implemented and tested a method to achieve the above, that we call “Calibrate with Confidence”. I will explain and demonstrate it.

Thanks to SpectraAnalytics, it is available in Excel, Python and GUI from www.calibratewithconfidence.co.uk

Our paper is available at Royal Society Open Science / CC BY 4.0

An article about it appeared in Times Higher Education

For directions, please visit the University of Birmingham website. Parking in the North East multi-storey car park is free after 4.30 pm, access is via Pritchatts Road (Sat Nav B15 2SA).

No charge is made to attend meetings; non-IMA members are welcome

Image credit: Soviet Caliper by Morpheu5 / Flickr / CC BY-NC-ND 2.0

Leave a Reply

Your email address will not be published. Required fields are marked *