The report shows a near-perfect score, yet on the line the model misses defects or rejects good parts. The most common cause is not the model itself but the data: the test set was too similar to the training set. This is called leakage — information from training seeps into the test, and instead of measuring how the model handles a new part, the test measures its memory.
In How many images do you need to train an AI inspection? we wrote that the test set has to be independent. This article is about how that independence is lost quietly — even when images are formally separated and nobody did anything wrong.
Leak one: the same shot in two sets
The simplest case is the same file imported twice — say once from the line folder and again from a copy on a network drive, under a different name. One ends up in training, the other in the test, and the model "recognises" an image it has already seen. That is why Trainer computes a SHA-256 checksum for every file on import and skips duplicates, even when the files have different names.
Leak two: frames from one video and shots of one part
Far more common, and harder to spot, is a video split into frames. Neighbouring frames differ only slightly — in practice they are the same image. If a random split sends some of them to training and some to the test, the model is tested on a part it saw a fraction of a second earlier. The score looks great but says little about behaviour on a new part.
The same goes for several photos of one part: from different sides, with a different setting, in successive attempts. In Trainer, shots of the same part and frames of one video form product and session groups, and a whole group always goes into one set. The split happens at the level of parts and sessions, not individual files.
Leak three: a test from the same hour as training
Even without repeated parts, a test set can be too uniform: all images from one shift, one batch of raw material and one lighting setup. The model passes the test because the test looks exactly like training. On the line it will meet another batch, another operator and optics that have had a week to get dirty.
So the test set should not be put together at the end, "from whatever is left". It is better to plan from the start that it covers different batches, shifts and days than the training set. An extra check is a test on a video from the target camera: at least 10 minutes covering good products, every known defect, several products at once and partial occlusions — through the same inference path the station will use on the line.
A "defect-free" image has to be looked at too
It is easy to fill the defect-free set automatically: treat everything nobody labelled as good. That is a trap. Among unlabelled images there are almost always defects nobody noticed. The model then learns that such a defect is normal, and a test with the same labelling errors will not reveal it — both sets are wrong in the same way.
That is why in Trainer a defect-free image has to be deliberately confirmed as negative. The model does not learn from images nobody has looked at.
Checking the dataset before training starts
Some data problems are visible before the first hour of GPU time is spent. Trainer checks the dataset before training:
- class distribution — with a warning when a defect is under-represented,
- very small boxes, which usually mean an annotation mistake,
- images with no product in the frame,
- ready-made datasets in Pascal VOC, YOLO, COCO, LabelMe and VIA formats — on an error the import stops and shows the problem image instead of silently accepting bad boxes.
A frozen dataset: a result that does not change
The last kind of leak is organisational: the data changes after training. Someone fixes annotations, adds images, moves a file between sets — and last week's report still describes the model, even though it now refers to different data. Worse still is fixing things "to suit the test score" until the test looks good. At that point the test set is no longer a test; it becomes a second training set, just tuned by hand.
Training in Trainer always uses a frozen version of the dataset. Later fixes do not change results you have already seen — after changes you prepare a new version. And when building a package for the line, Trainer checks the dataset fingerprint: if images or annotations changed after training, the package will not be created.
Validation chooses, the test only reports
Automatic training runs successive attempts and keeps the best model — but it chooses it only on the validation set. The test set takes no part in any decision; it only reports the result on images the model has not seen. If the best attempt were chosen by the test, the test would become part of training: the same leak, one step further.
Only figures from the independent test — recall, precision, false accepts and false rejects — are fit for setting the decision threshold. How to do that is covered in False accept or false reject?.
Six questions before you trust a result
- Can the same part or the same video be in both training and test?
- Does the test set cover different batches, shifts and days than the training set?
- Has someone actually looked at every "defect-free" image?
- Does the result come from a frozen data version that has not changed since training?
- Was the best model chosen on validation, with the test only reporting?
- Was the result also checked on a video from the target camera?
These safeguards are built into AIVQC Trainer, but no tool replaces well-planned image collection. If you want to check whether your data will give a trustworthy result, start with a pilot: 30 days, one Trainer workstation, one station and up to three defect classes. Get in touch — we will help plan image collection so that the test tells the truth.