Judging Week
Preparing for a judging session in school
During judging week you should set aside up to an hour for teachers to complete their judging. Exactly how long it will take depends on how many teachers you have to complete the judging. You can estimate the amount of time using this online calculator. The process of judging itself can be really good CPD if you organise it well. You can decide whether some or all of the judging will be done in a first get-together. If teachers are carrying out some judging together, make sure that computers are available for all of them. It may be that there is not time for all the judging to be done, in which case teachers can judge at another time of their convenience.
Before judging week
Firstly, you need to go to the judging page on your task and set the number of teachers who will be judging. The system will then automatically allocate the quotas for each teacher. Then copy the invite link and paste it into a browser to try out the judging yourself. You may then wish to send out the judging sign up link to your teachers in preparation for the judging session, although it is quite possible that after signing up they will start judging straight away.
During judging week
On the day, make sure everyone knows what they are looking for in the students' work. A very quick discussion and outline before you start can help reassure more nervous judges. To start the judging, each judge should click on the sign up link to receive their own unique judging link via email (a copy of their links can be viewed in the table on the Judges screen.)
Everyone should make their judgements independently. The independence of judging removes any bias and makes sure every pupil is considered fairly.
Encourage people to take a few notes while they judge, rather than discuss. A good discussion at the end of the process is great, but too much discussion during the process can slow people down.
Review the judging
You will be able to check how the judging is progressing for your judges by examining the information on the Judges page. In the particular task, select the Run judging session option, and then, just to make sure the judging statistics are up to date, press the Refresh Scores button. You can now see how many of the expected judgments have been carried out by your judges, and you can take a look at the reliability of your assessment.
For reliability, as a rule of thumb, we would suggest the following:
Low stakes assessment = Minimum reliability of 0.65
High stakes assessment = Minimum reliability of 0.80
At the end of judging week
If you have time at the end, it is really good to take a look at your results and have a discussion with your teachers. (Press the 'Refresh Scores' button first, to update the results!) Are you happy with the piece you have judged to be the best? The worst? How do you feel as you move down the rank order? Do you feel that you have rewarded what you feel is important in the work? Has the judging given you a feel for next steps in teaching and learning?
If you would like to read more about providing feedback to pupils, please read our blog series on whole-class feedback here.
A note on moderation
When you take part in a national judging session as part of our national projects, you judge your own scripts as well as judging a sample of other schools' scripts. The judging from other schools moderates your results and allows you to make comparisons between your school's performance and everyone else's performance.
For example, we might set it so that every 5th or 10th judgement you make is comparing two scripts from other schools, whereas all the other judgements will be made on two of your own pupils. You will never be asked to compare your pupils with other pupils, to avoid possible bias in any judgements.
The same thing will happen for all the other schools in the project. This means that your pupils included in the 'moderation pot' will be judged by teachers in other schools against pupils from other schools, therefore resulting in the moderation of these scripts.
For further information about moderation, please refer to this article: Local judgements and moderation judgements.
Looking back at the judging
As a coordinator, you may want to look back at how well your judges carried out their judging, and also possibly look at any candidates where you are concerned about the accuracy of their scores.
- To look overall at how your judges did, click on this guide.
- To look at the judgements made by a particular judge, see this guide.
- To look analyse in more depth the judgements made for a particular candidate, see this guide.
During judging week you should set aside up to an hour for teachers to complete their judging. Exactly how long it will take depends on how many teachers you have to complete the judging. You can estimate the amount of time using this online calculator. The process of judging itself can be really good CPD if you organise it well. You can decide whether some or all of the judging will be done in a first get-together. If teachers are carrying out some judging together, make sure that computers are available for all of them. It may be that there is not time for all the judging to be done, in which case teachers can judge at another time of their convenience.
Before judging week
Firstly, you need to go to the judging page on your task and set the number of teachers who will be judging. The system will then automatically allocate the quotas for each teacher. Then copy the invite link and paste it into a browser to try out the judging yourself. You may then wish to send out the judging sign up link to your teachers in preparation for the judging session, although it is quite possible that after signing up they will start judging straight away.
During judging week
On the day, make sure everyone knows what they are looking for in the students' work. A very quick discussion and outline before you start can help reassure more nervous judges. To start the judging, each judge should click on the sign up link to receive their own unique judging link via email (a copy of their links can be viewed in the table on the Judges screen.)
Everyone should make their judgements independently. The independence of judging removes any bias and makes sure every pupil is considered fairly.
Encourage people to take a few notes while they judge, rather than discuss. A good discussion at the end of the process is great, but too much discussion during the process can slow people down.
Review the judging
You will be able to check how the judging is progressing for your judges by examining the information on the Judges page. In the particular task, select the Run judging session option, and then, just to make sure the judging statistics are up to date, press the Refresh Scores button. You can now see how many of the expected judgments have been carried out by your judges, and you can take a look at the reliability of your assessment.
For reliability, as a rule of thumb, we would suggest the following:
Low stakes assessment = Minimum reliability of 0.65
High stakes assessment = Minimum reliability of 0.80
At the end of judging week
If you have time at the end, it is really good to take a look at your results and have a discussion with your teachers. (Press the 'Refresh Scores' button first, to update the results!) Are you happy with the piece you have judged to be the best? The worst? How do you feel as you move down the rank order? Do you feel that you have rewarded what you feel is important in the work? Has the judging given you a feel for next steps in teaching and learning?
If you would like to read more about providing feedback to pupils, please read our blog series on whole-class feedback here.
A note on moderation
When you take part in a national judging session as part of our national projects, you judge your own scripts as well as judging a sample of other schools' scripts. The judging from other schools moderates your results and allows you to make comparisons between your school's performance and everyone else's performance.
For example, we might set it so that every 5th or 10th judgement you make is comparing two scripts from other schools, whereas all the other judgements will be made on two of your own pupils. You will never be asked to compare your pupils with other pupils, to avoid possible bias in any judgements.
The same thing will happen for all the other schools in the project. This means that your pupils included in the 'moderation pot' will be judged by teachers in other schools against pupils from other schools, therefore resulting in the moderation of these scripts.
For further information about moderation, please refer to this article: Local judgements and moderation judgements.
Looking back at the judging
As a coordinator, you may want to look back at how well your judges carried out their judging, and also possibly look at any candidates where you are concerned about the accuracy of their scores.
- To look overall at how your judges did, click on this guide.
- To look at the judgements made by a particular judge, see this guide.
- To look analyse in more depth the judgements made for a particular candidate, see this guide.
Updated on: 26/08/2024
Thank you!