◂▸
A field engineer photographs an equipment rating plate. CohereLabs North-Micro-Vision-Instruct, a 2.4B open vision model, reads the fields, then is asked to locate each value it read. Every field is tiered: green when it is located on the plate and passes its format check, amber when it is located but the value looks wrong for the field, and red when the model could not point at it, so the value is discarded and left for a person to type. The rule: a field the model cannot point at is a field the model invented. Nothing is saved without a person confirming it.
What we tried
- Asked the model for a JSON record of the plate's fields, then, for each value, where it appears in the image.
- Accepted only evidence inside the image: a box, or a single point. A line or a coordinate pulled out of prose doesn't count.
- Scored 32 seeded fictional plates with exact truth boxes from the renderer, through the same go/no-go harness the product uses. The plate shown is the set's median plate by green count, picked by rule.
What we measured
| Measure | Run 1: boxes only | Run 3: points accepted | Note |
|---|---|---|---|
| Present fields shown green | 7.3% | 97.6% | |
| Green values that were right | 94.4% | 97.1% | |
| Location lands on the right text | 4.0% | 96.4% | |
| Values discarded (red) | 93.3% | 10.0% | Most red fields after the fix are fields absent from the plate, correctly left blank |
| Photos rejected as too blurred | — | 2 of 32 | |
| Time per plate | — | 29 s | RTX 4070, shared with other GPU work during the run |
What went wrong
- First real run: 93% of values were discarded. The model answers "where is it?" with a point, not a box, and the parser only accepted boxes. The values were right; the evidence was being thrown away.
- The first fix let lines and numbers in sentences count as locations. An audit caught it before any number was published.
- Serial numbers: in 6 of 30 plates the model inserted an extra digit (HU484878923 read as HU4848789923) and the value still came out green, because it pointed at the right place and the format looked valid. Pointing at the right place isn't the same as reading it right; serials need a check digit or a second read.
- On one plate with no model number, it read the "3-PHASE INDUCTION MOTOR" subtitle as the model number.
What happens next
- Test on about 50 real site photographs, the go/no-go gate the product actually depends on.
- A stricter check for serial numbers.
- A hosted demo.
Built with
- CohereLabs North-Micro-Vision-Instruct Apache 2.0
- Hugging Face Transformers Apache 2.0
- PyTorch BSD-3-Clause
- Gradio Apache 2.0
- OpenCV Apache 2.0
- Pillow MIT-CMU (HPND)
Next experiment
Five streets in two cities, remembered in four dimensions →
Want this for your data?