“Calibrate with Confidence”, Integrated panel assessment, a talk by Robert MacKay FRS (University of Warwick)
Suppose your company advertises for new employees and tasks a panel with scoring the applicants. Each panel member has time to assess only a fraction of the applicants but each applicant’s dossier is evaluated by at least two panel members. How should the panel combine their scores to remove the effects of differences in use of the scale and different matches between expertise of the assessor and skills of the applicant? Appointments are a big investment for a company.
Or suppose you are in charge of an examination board for which students are assessed on their selection of options. How do you combine the marks to take into account differences in marking standards and length between the options?
Or suppose a research assessment panel of 15 people evaluates 3,000 publications in applied mathematics. Each panel member can evaluate 400 publications so each publication gets two assessors. How should the panel combine their scores to remove the effects of differences in use of the scale and different matches between expertise of the assessor and subject of the publication? The 2016/7 payout to English universities for the results of the REF (for all subjects) was over £1.5B so one ought to try to get it right.
With an undergraduate project student and colleagues from Coventry University, we devised, implemented and tested a method to achieve the above, that we call “Calibrate with Confidence”. I will explain and demonstrate it.
Thanks to SpectraAnalytics, it is available in Excel, Python and GUI from www.calibratewithconfidence.co.uk
An article about it appeared in Times Higher Education
For directions, please visit the University of Birmingham website. Parking in the North East multi-storey car park is free after 4.30 pm, access is via Pritchatts Road (Sat Nav B15 2SA).
No charge is made to attend meetings; non-IMA members are welcome