Segment Where You Look

Abstract

Vision-language models (VLMs) that segment from language must pass the location of the referent to a pixel decoder. The dominant interface compresses it into the hidden state of a single [SEG] token, a d-dimensional vector with no spatial index, and leaves the decoder, which never sees the expression, to recover the referent's extent. We show that the VLM already computes a better signal: the [SEG] token's attention over the image tokens, which is defined on the patch grid by construction. Probing a frozen segmentation VLM, we find that this attention yields 18 to 22 more gIoU than the [SEG] hidden state when read by the same probe, that it gains nothing from the image where the hidden state does, and that it is concentrated in the late layers. Building on these findings, we introduce AttnSeg, which replaces the [SEG] embedding with this attention as the decoder's prompt. A learned, signed combination of the attention heads forms a dense spatial prompt, and an attention-guidance objective supervises it directly. A two-stage recipe first trains the VLM to localize without the decoder and then trains the decoder to refine, within the budget of standard finetuning. Applied to Sa2VA, AttnSeg improves ReasonSeg by 6.4 gIoU and GS-Eval by 10.2 at 4B (4.2 and 9.6 over the same model finetuned on the same data), and by 8.8 and 11.3 at 14B. The gains reach 13 to 18 gIoU on object parts, multiple objects and single objects, where the referent must be singled out, and are negligible on stuff regions, which the decoder already recovers unaided.

AttnSeg in Action

Each demo follows one query through AttnSeg. The VLM answers with a [SEG] token; the [SEG] row of every layer and head is read over the image tokens; a learned 1×1 convolution emphasizes the heads on the target and subtracts those on distractors, sinks or diffuse regions; and SAM refines the resulting map into the mask.

Motivation

Segmenting an object from a natural-language expression requires deciding which region the expression refers to and where exactly its boundary lies. Promptable segmenters such as SAM have made the second the easier half of the problem: given a prompt that points at the right region, they return a high-quality mask. The first requires understanding the expression, which is the role of the VLM.

The VLM should be the load-bearing localizer, and the decoder should only refine.

Whether this holds depends on the interface through which the VLM passes the referent's location to the decoder. The dominant one, introduced by LISA and adopted by most later systems, prompts the decoder with the final hidden state of a [SEG] token: a single vector with no spatial index, handed to a decoder that never sees the expression.

What the VLM Already Knows

We freeze Sa2VA-4B entirely, discard its decoder, and train only a lightweight probe to predict the mask from a single internal signal: the [SEG] hidden state, which the dominant interface transmits, or the [SEG] token's attention over the image tokens, which lies on the patch grid by construction. Because the model is frozen, whatever the probe recovers is information the signal already carries. Each signal is read at two capacities: a single 3×3 convolution (≈5K parameters) and a U-Net (≈3.86M).

(a) Which signal, and what it carries

(b) Where in the network it resides

Method

Attention as the spatial interface

AttnSeg replaces the [SEG] embedding with the [SEG] token's attention as the interface between the VLM and the decoder. For each layer \(l\) and head \(k\), the attention row queried by the [SEG] token and restricted to the image tokens assigns a weight to every patch; each map \(\boldsymbol{a}^{(l,k)}\) is rescaled to spatial mean one, so a head is characterized by where it attends rather than by how much. A learned combination turns the \(LH\) maps into one map, and that map alone prompts the decoder:

• Dense. The map enters the decoder's mask-prompt channel and is never thresholded or sampled, so the decoder receives its graded confidence rather than a binary mask or a handful of points.
• Selective. The weights amount to a 1×1 convolution with only \(LH\) parameters, initialized at the uniform average and learned rather than fixed to the band the probes found.
• Signed. A softmax gate can only suppress a misleading head; a negative weight can subtract it. The learned weights emphasize heads on the referent and subtract heads locked onto distractors, attention sinks or diffuse regions.

Attention-guidance objective

We supervise the map directly against the ground-truth mask, area-pooled onto the patch grid as \(\boldsymbol{m}\). The map should delineate the referent as fully as a patch-level map allows, while recovering the boundary at full resolution remains the decoder's task. We adopt a weighted cosine, with foreground patches weighted \(\lambda_{\mathrm{fg}} = 3\) and background patches \(\lambda_{\mathrm{bg}} = 1\), and combine it across targets as

where \(\bar{m}_i\) is the fraction of the grid that target \(i\) covers and the gate \(g_i\) is zero once the peak of its map already falls inside the target.

Decoupled two-stage training

We train the two components separately, so that the VLM carries the localization at the granularity the expression demands and the decoder only refines it. Under joint training, the decoder's gradient into the map is roughly 49 times that of \(\mathcal{L}_{\mathrm{att}}\) and roughly orthogonal to it, so the map would be shaped mostly by the decoder.

Each stage takes half of one epoch of the Sa2VA training mixture, the same budget as finetuning the base model for one epoch.

Results

Main results

We build AttnSeg on two Sa2VA models, with InternVL2.5-4B and InternVL3-14B as their VLMs, and evaluate on ReasonSeg (test), which names the target by its function or role rather than its appearance, and on GS-Eval, which varies the granularity of the target. Neither benchmark's training data is part of our training mixture.

Where the gain comes from

Starting from the released Sa2VA-InternVL2.5-4B, we add one component at a time. The attention-guidance objective accounts for 61% of the total gain on ReasonSeg and 83% on GS-Eval.

Choice of the objective

The three objectives fail differently. The peak ratio points best but specifies almost no extent, which is the pointer of the [SEG] embedding, measured directly. Dice specifies extent but points worst. The weighted cosine is the only one with no severe weakness, and it yields the best masks.

Training makes the attention easier to read

We repeat the probing on the trained model, adding a gate with one weight per attention map (560 parameters) that can only select among the maps the model has already drawn. The gate gains 18.04 and 17.86 gIoU and now recovers close to four fifths of what a U-Net nearly seven thousand times larger recovers, against about half before training. The objective mainly reorganizes the attention so that a trivial readout can take the localization directly.

Qualitative Results

Because the map is the decoder's only prompt, the map and the mask can be read against each other directly: the attention map is the prediction, not a diagnostic. Click any figure to enlarge it.

BibTeX

@article{abdelrahman2026segment,
  title   = {Segment Where You Look: Attention as the Interface
             Between Language Models and Segmenters},
  author  = {Abdelrahman, Eslam and Alkaf, Hamza Omar and Li, Yuheng and
             Iyer, Siddharth Srinivasan and Zhao, Nanxuan and
             Shechtman, Eli and Singh, Krishna Kumar and Elhoseiny, Mohamed},
  journal = {arXiv preprint},
  year    = {2026}
}