Spinning the strand
benchmarks
Spinning the strand
benchmarks
—
Every figure on this page was written by an evaluation script, not typed in. The test split holds 25,250 images and was untouched until final evaluation; model selection used a 3,030-image slice carved out of train.
97.156%
test top-1
prob average (uniform)
99.644%
test top-5
98.301%
oracle ceiling
if a perfect combiner always picked the right member
99.56%
conformal coverage
average 1.544 candidates
01
Each probe trained on cached frozen features.
| Head | Params | Val top-1 | Test top-1 | Test top-5 | Train |
|---|---|---|---|---|---|
| Gated fusion head | 3.97 M | 95.78 | 96.97 | 99.48 | 21.9 min |
| SigLIP-SO400M | 1.93 M | 95.64 | 96.83 | 99.53 | 3.8 min |
| EVA-02-L | 1.73 M | 93.14 | 95.53 | 99.30 | 6.9 min |
| DINOv2-L | 1.73 M | 92.64 | 94.87 | 99.20 | 0.8 min |
02
Every subset of members, in probability and logit space, plus weights tuned on validation only.
| Members | Method | Test top-1 | Test top-5 |
|---|---|---|---|
| SigLIP-SO400M + EVA-02-L | prob average (uniform) | 97.156 | 99.644 |
| SigLIP-SO400M + EVA-02-L | prob average (val-tuned) | 97.141 | 99.659 |
| SigLIP-SO400M + EVA-02-L | logit average (uniform) | 97.105 | 99.640 |
| SigLIP-SO400M + EVA-02-L + DINOv2-L | prob average (uniform) | 97.093 | 99.675 |
| SigLIP-SO400M + EVA-02-L + DINOv2-L | prob average (val-tuned) | 97.081 | 99.671 |
| SigLIP-SO400M + EVA-02-L + DINOv2-L | logit average (uniform) | 97.065 | 99.707 |
| SigLIP-SO400M + DINOv2-L | prob average (val-tuned) | 97.018 | 99.655 |
| SigLIP-SO400M + DINOv2-L | logit average (uniform) | 96.947 | 99.651 |
| SigLIP-SO400M + DINOv2-L | prob average (uniform) | 96.907 | 99.659 |
| SigLIP-SO400M | solo | 96.828 | 99.533 |
| EVA-02-L + DINOv2-L | prob average (uniform) | 96.222 | 99.497 |
| EVA-02-L + DINOv2-L | logit average (uniform) | 96.222 | 99.537 |
| EVA-02-L + DINOv2-L | prob average (val-tuned) | 96.040 | 99.489 |
| EVA-02-L | solo | 95.529 | 99.295 |
| DINOv2-L | solo | 94.871 | 99.196 |
03
A few tenths of a point on 25,250 images is inside the range where two systems differ by luck. Exact McNemar tests on paired predictions decide it.
prob average (uniform) siglip_so400m+eva02_large vs best solo siglip_so400m
p = 0.000003
✓ significant
198 images only the first got right, 115 only the second.
prob average vs logit average (all members)
p = 0.546713
! not significant
53 images only the first got right, 46 only the second.
04
Ensembling only helps when members fail on different images. These mostly fail on the same ones.
SigLIP-SO400M vs EVA-02-L
65.2%
of the first model's errors are shared with the second
SigLIP-SO400M vs DINOv2-L
66.3%
of the first model's errors are shared with the second
EVA-02-L vs DINOv2-L
59.6%
of the first model's errors are shared with the second
05
EVA-02-L was fine-tuned end to end — six epochs at 224px then one at 448px, about nine GPU-hours. Alone it reaches 95.901%, below the frozen pair. The question is whether its errors are different enough to help in combination.
Frozen pair (deployed)
97.156%
two small heads over cached features
Three-way, uniform
97.236%
p = 0.214 — not significant
Three-way, val-tuned
97.267%
p = 0.052 — not significant
Ensembling only pays when members fail on different images. This is the number that decided it.
At 224px the fine-tune shared 65.5% of SigLIP's errors — no better than the frozen backbone. The single 448px epoch dropped that to 60.9% while adding just 0.2 points of validation accuracy. Its contribution was diversity, which no accuracy column shows.
Validation top-1 per epoch. The resolution change is the last step.
06
A 76-case gold set whose ground truth is known before the system is asked, including questions it is supposed to refuse.
| Category | n | R@1 | R@5 | Answerable@1 | Correct | Refused |
|---|---|---|---|---|---|---|
| nutrient lookup | 48 | 64.6 | 100.0 | 100.0 | 97.9 | 0 |
| ingredients | 10 | 100.0 | 100.0 | 100.0 | 100.0 | 0 |
| graph inversion | 8 | 75.0 | 87.5 | 75.0 | 100.0 | 0 |
| superlative | 4 | 100.0 | 100.0 | 100.0 | 100.0 | 0 |
| out of scope | 6 | correct behaviour is refusal | 100 | 100 | ||
98.57%
answered correctly
100%
refusal accuracy
0 false refusals
100%
grounded
every figure traced to a source
$0.0245
cost of the run
76 questions