Articles on: Judging

Setting your human judging quota with AI tasks

Go to the Run judging session page and make sure the expected candidates in the judges panel is as you expect (see image below).

Use the slider to set the ratio of AI judgements that you want on the task. The default will be 90% AI judging.

Now set the expected human judges but set this value not to the number of actual human judges you have available but set it so that human judges will receive a reasonable number of judgements each. This is to prevent only a few judgements being allocated to each human judge. Start with 1 expected human judge, press Adjust and see the quota for each judge. If you think this is too many, change the expected human judges to 2, and repeat until a reasonable quota will be given to each judge.

Do not worry if you then allocate more human judges than the stated expected – that will just mean you do more judgements than we usually ask for but that is a good thing!

If you want to manually calculate how many expected human judges you would ideally like, use this formula:

Therefore, if you have 100 expected candidates, you want to use an AI ratio of 90%, but you want 15 judgements for each human judge:

So set your expected human judges to either 6 or 7.

Updated on: 26/01/2026

Was this article helpful?

Thank you!