Exploring the judgements made by a given judge

Here is how to explore more deeply the judgements made by a particular judge

To explore the judgements that were made by a particular judge more deeply, we can look at the decisions made by that judge, whose scripts they were comparing, and the probability that the decision was 'correct'.

After the judging is completed, go to the Run judging session section and then highlight the judge you want to look at by clicking on the box next to them. Then click on the Decisions for Judge button (the button with the three lines).




You will then see this page:




Here are the important aspects of this Judge Decisions page:

Chosen: the 'winning' candidate's name - you can click on the name to view their script;
Chosen Score: the Scaled Score for the 'winning' candidate;
Not Chosen: the 'losing' candidate's name - you can click on the name to view their script;
Not Chosen Score: the Scaled Score for the 'losing' candidate;
Time Taken: the time taken for this one judgement;
Probability: based on the two candidates' Scaled Scores, we can calculate the probability that this judgement was 'correct' based on other judges' decisions on the task. Or rather, the higher this probability, the more likely the decision was in agreement with the other judges.

We can order the judgements by clicking any of the headings of the Decisions table.

In the above example, we can see that Mike Mango won against Belinda Berry, as we might have expected based on their Scaled Scores. Mike also won a decision against Tina Tomato - a decision which in all likelihood could have gone either way (the probability being 50% in this case).

By looking at the judgements made by a judge, and which candidates they judged for and against, we can get an idea of their judging preferences. We can also highlight decisions that we are concerned about, and the candidates whose scripts these decisions were made on. We can use this information to get a better understanding of the reasons behind the decisions made by a particular judge.

Updated on: 04/09/2024