Articles on: Judging

Feedback on judges

Judging Feedback

After you have set your judges off judging, you will want to look at how they getting along in their judgements. This article explains how to review their progress.

At any point during the judging process, for a given task, select the Run judging session option, and then, just to make sure the judging statistics are up to date, press the 'Refresh Scores' button:

You will be able to check how the judging is progressing for your judges by examining the information in the Judges table:

Email - This is the email address of the judge.

Link - This is the unique link that can take you to the judging site for this judge for this task.

Infit - This statistic is an indication of the degree of consistency or inconsistency of this judge's decisions compared to those of the other judges. A low infit score indicates better consistency, a higher infit score indicates worse consistency. See this page for critical values of infit.

Local - the number of 'local' judgements the judge has made (see Local judgements and moderation judgements).

Mod - the number of 'moderation' judgements the judge has made (see Local judgements and moderation judgements).

Quota - the total number of judgements allocated to that judge. This information is useful if you need to remind judges to complete their judgements.

Median Time - the median time taken by a judge in their judgements on this task. Quite short median times might indicate that a judge is rushing through their judgements (it is informative to compare the median time to the infit score), and a long median time might indicate that the judge is facing some problems with the judgement process.

% Left Click - In the judging screen, scripts are presented randomly on the left or right hand side of the screen. Therefore, we would generally expect that the percentage of times that a judge clicks in favour of the 'Left' script would be around 50%. If the % Left Click value is much higher or lower than 50%, it might indicate that the judge is not thinking carefully about their judgements and just staying with the left or right hand side. Again it is informative to compare this value with the infit statistic and the median time taken.

Exclude - If there are real concerns about the standard of judging by a particular judge, you may wish to Exclude this judge's decisions from the final calculation of scores, so that they are more reliable. Only do this after you have tried to sort out any problems with the judge.

The Remove button (the 'bin' icon) completely removes the judge from the process, and we would normally suggest that you do not do that. Remember that in excluding or removing judges, this will have a knock-on effect on the number of judgements other judges need to make, and you may need to change the number of judgements assigned to the other judges. You can do this by altering the figures in the judging calculation table and pressing the 'Adjust' button, or selecting a judge's check box and pressing the 'Update Judge Quota' button.

You may also want to examine the different decisions made by your judges. You can do this by following this guide.

Decisions

If you want to download all the decisions click on the 'Decisions' button. The CSV file details each individual judgement made by each judge:

Along with the email address of the judge, it shows the Codes that correspond to the script that was chosen (won) and the script that wasn't chosen (lost), along with the time taken to make that judgement (in milliseconds) and the time and date the judgement was made.

Feedback on judges

Judging Feedback

Decisions

Further Reading