Vision Tower Bench

Choosing a vision tower for a text-only LLM

We probed six frozen vision towers across stages. No tower wins everywhere, and spatial tasks peak before the last layer. Projectors preserve spatial info, with the size of the gain varying by dataset.

Finding 1No Tower wins every capabilitySee the evidence ↓Finding 2Spatial information often peaks before the final layerSee the evidence ↓Finding 3Projector effects depend on the datasetSee the evidence ↓

Paper Repo 🤗Towers

Version

Why this exists

Adding vision to a base LLM means choosing a tower, layer, and projector. These are choices full-model benchmarks like MMMU never isolate. We probe frozen features across stages, before any multimodal training, and publish Capability Profiles.

What the capability axes mean

Recognition

Can a probe identify the object, with plenty of labels and with almost none?

Geometry

Can it recover depth and surface orientation from a single image?

Cross-view

Can it find the same physical point again in a second photograph?

Occlusion

How much recognition survives when half the image is hidden?

Capability Profiles

Each column measures a different capability, so the results are not combined into an overall score. Some apparent differences are too small to distinguish on the current test set.

Tower	Few-label1% top-1 ↑	Full-label100% top-1 ↑	Depthd1 ↑	Normalsdeg ↓	Cross-viewNAVI@2cm ↑	Occlusion50% kept ↑
DINOv2	0.369	0.908	0.702	18.85	0.539	52%
SigLIP2	0.656	0.917	0.670	23.90	0.401	49%
Muse Glimmer	0.433	0.921	0.659	24.30	0.214	52%
Kimi K2.6	0.345	0.886	0.660	25.51	0.365	28%
Qwen3.8	0.245	0.879	0.674	24.24	0.378	30%
Kimi K3	0.219	0.837	0.655	27.03	0.333	20%

Muse Glimmer and SigLIP2 are unresolved at full-label recognition. Estimated difference: +0.004, paired 95% interval: [−0.004, +0.012].
↑ higher is better, ↓ lower is better.

Method, in short

Towers stay frozen. We tap eight Relative Depth points, plus merged and projected when present. Features cache once; only the readout trains. Recognition is a 1.6M attention pool on ImageNet-100, geometry is Probe3D decoders on DIODE, and correspondence is training-free matching.

Heads are capacity matched with frozen PCA so wider stages don't get bigger heads; the unmatched arm runs as a check. Learning rate is picked per cell on validation, three seeds, with paired bootstrap intervals on headlines.

Figure 1

How the bench taps a model

Results

Finding 1

No Tower wins every capability

DINOv2 sweeps depth, normals and cross-view but sits third at recognition; Muse Glimmer leads recognition and sits last on correspondence. On the attention readout, SigLIP2 leads by 0.22 at 1% labels.

Figure 2

Full-label accuracy is a poor selection criterion

Finding 2

Spatial information often peaks before the final layer

Geometry falls before the last layer in 23 of 24 arms while recognition rises to it in 19 of 24. Only Qwen3.8 declines into the last layer on recognition, and only in unmatched arms.

Figure 3

Geometry peaks early

Finding 3

Projector effects depend on the dataset

On DIODE, all 16 projected-against-tower depth and normal comparisons favor the Projector, so it preserves spatial information. However, on KITTI driving scenes none resolve in its favor and two reverse. Qwen3.8 also reverses on ScanNet while gaining on NAVI.

Figure 4

The Projector helps on DIODE

What v1 does not establish

—These are frozen-representation results, not evaluations of a working VLM.
—The full-token attention control was inconclusive, so cross-model attention rankings rest on pooled features.
—Geometry and robustness use small test sets, and cross-model geometry still mixes representation quality with native token geometry.

The paper sets out every limitation in full, along with the controls that failed.

Representation explorer

What the features look like

Qualitative PCA of frozen patch tokens. Read structure, not color.

source · 448²

DINOv2 · tower · 32 x 32

DINOv2 segments its subject

The eagle separates as one solid region. First on all three correspondence sets, third on recognition.

NAVI recall@2cm0.539 · rank 1 of 6

recognition top-10.9079 · rank 3 of 6

Changelog

Sept 2026

Vision Tower Bench v1 released, covering six Towers on geometry, semantic and correspondence.

Sept 2026

Vision Towers for Kimi K2.6, Kimi K3, Muse Glimmer and Qwen3.8 published on Hugging Face.