What to do if you spot an AI error

There are two sources of AI judging error.

• The AI can mis-transcribe the student handwriting. This is the most common source of error. It is also more prevalent in younger years, where student handwriting can be very challenging to read. It will typically result in the transcription being better than the original handwritten piece, meaning the essay gets a higher mark than deserves.
• The AI can make the wrong judgement. This is relatively rare. 80-85% of the time, the AI agrees with the human. The 15-20% of disagreements are quite small, and involve legitimate disagreements. There are then about 0.5-1% of big disagreements which need an explanation. The majority of these disagreements are the result of human error, not AI error. However, a couple of these will be the result of AI error. Typically, these will occur when the AI is presented with responses that are outside its training. For example, if a pupil writes a very short response when the AI is expecting an essay it can become confused.

It is very challenging to prevent the anomalies, but there are two reports you should review to check if there are any issues.

• Check your AI-human agreement report. This lists all the human-AI disagreements in order of size. If the AI has got anything completely wrong, it should be on the first page of this report.
• Check every student's infit statistic. Infit is a measure of the extent to which judges agreed on a script. A script that provokes great disagreement will have a high infit. On the Results & Feedback, feedback page we highlight inconsistent (high infit) scores in red. You can see what this looks like below. We recommend that you click on each red script and review it. Please note that a high infit score may not just be the result of AI or human error. Often scripts that are uneven in quality will get a high infit because they are genuinely difficult to judge.

What should I do next?

In cases where the AI is clearly wrong we can remove the decisions for that pupil and you can judge them again - just get in touch at support@nomoremarking.com

Updated on: 05/03/2026