A Model Test Stops Being Unseen When Its Examples Leak into Training

Conceptual desk scene with a notebook, a pressed leaf and blank cards beside a laptop.
Conceptual illustration created with AI; not a research result or laboratory photograph.

A remarkably good model score deserves a question about the data split, not just congratulations. If the same examples appear in training and evaluation, the test may no longer represent genuinely unfamiliar cases.

Check the boundary between datasets

Google’s guide to dividing datasets explains the roles of training, validation, and test sets, including the problem of duplicated examples. Different filenames do not prove that the underlying records are different.

In a hypothetical image collection, two resized versions of one original photograph could land on opposite sides of a random split. A useful audit therefore considers the originating item, not only an exact file hash.

Keep evaluation separate from repeated tuning

Repeatedly changing the model in response to one test set can also make that set less independent of development decisions. Record which data informed training, which informed adjustments, and which were reserved for final evaluation.

The right split depends on the problem, including whether future observations or new people are the intended challenge. A strong score is easier to interpret when the report explains those boundaries. Checking leakage is part of understanding what the test demonstrates, rather than a reason to dismiss every successful model.