Spinning the strand
method
Spinning the strand
method
—
Seven stages, each with a measurement behind it. Where something did not work, it says so — a method section that only reports successes is advertising.
01
Three backbones — SigLIP-SO400M, EVA-02-L and DINOv2-L — were run once over all 101,000 images and their embeddings cached. That took about twenty hours and made every experiment afterwards finish in seconds instead of hours. The cache is the reason this project could afford to be rigorous: calibration, conformal sets, abstention and every ensemble combination were re-run dozens of times against it.
02
A gated fusion head with 3.97M parameters was trained for 21.9 minutes to learn how to weigh the backbones per image. A parameter-free average of their probabilities, computed in milliseconds, beat it — and by exact McNemar test the fusion head did not significantly outperform its own best single input. The average is what ships.
03
Raw softmax output is not a probability. Fitting one temperature on held-out data cut expected calibration error from 0.051 to 0.006 — an eightfold improvement — and moved mean confidence from 92.0% to 96.6% against 97.16% accuracy. Notably the model was under-confident, not over-confident, which is the less common direction and means the correction sharpens rather than softens.
04
A conformal prediction set covers the true class 99.56% of the time, measured on 25,250 held-out images, at an average of 1.54 candidates. That is a property of the procedure rather than a claim about any single image, and it holds without assuming anything about the model. Below 97% coverage the machinery is pointless here — top-1 accuracy alone already satisfies it.
05
Two signals decide whether to answer at all: the calibrated maximum probability, and the size of the conformal set. Neither is sufficient alone — a flat blue image scored 0.43 on the first and slipped through, while the second caught it at eight candidates. Together they abstain on 2.28% of real food and raise accuracy on what remains from 97.16% to 97.94%.
06
BM25 and a bi-encoder fail differently, so their ranks are fused rather than their scores, then reranked — by a local cross-encoder, or by Cohere where a key is configured. Crucially, the question is rewritten to name the dish the vision model already identified — real questions say "how much sodium is in this", and against 101 near-identical sodium documents no retriever can resolve that pronoun on its own.
07
Every quantity in a generated answer is checked against the retrieved sources, matched per unit so milligrams can never be supported by the same digits in grams. An answer that fails is withheld and a deterministic summary served instead. On the first live query the gate caught a figure that was arithmetically correct but appeared in no source — the fix was to put the number in the corpus, not to loosen the check.
—
Every one of these was found by measuring rather than assumed, and none of them is hidden in the interface.
Grad-CAM localises at 0.87× chance on this architecture — better than random but not sharply on the food. The probe never sees the image, only a globally pooled vector, so position is partly discarded before classification. The attention-pooling map is worse still, at 1.14×, which reproduces the known artefact where transformers park high attention on empty background patches.
60 of the 101 dishes have no single USDA record and were built from weighted ingredients. Those figures are estimates, and every surface says so rather than presenting them as measurements.
Anything outside Food-101 is refused rather than guessed at. That is a deliberate limit, not an oversight — a classifier restricted to 101 categories would otherwise return one of them for a photograph of a car.
It reports that the model is lost, which correlates with but is not the same as "this is not food". A real detector needs non-food negatives to train against.