How Test Data Leakage Can Make an AI Look Better

A high score can answer the wrong question

An AI evaluation is meant to tell us something about performance beyond the examples used to shape the system. If the answers or closely related examples already influenced development, the resulting score may answer a narrower question: how well does the system handle material it has effectively encountered before? That can be useful for some purposes, but it is not the same evidence as an independent test.

NIST's AI Technology Evaluation program describes a sequestered testing environment designed to keep evaluation data from being used for training. NIST's guidance on preventing evaluation cheating also discusses preserving held-out material and documenting evaluation conditions. These practices help protect the meaning of comparisons. A score is more informative when readers understand what the system could access and what remained separate until assessment.