Do Composed Image Retrieval Benchmarks Require Multimodal Composition?

TL;DR. Composed Image Retrieval (CIR) should require both a reference image and a modification text. Across the four standard CIR benchmarks — CIRR, FashionIQ, LaSCo, CIRCO — we audit 11 retrievers (9 open-source + 2 commercial APIs) and find that a large share of test queries can be solved with only the image or only the text. CIRCUS releases the audit, the resulting shortcut-free subset (23 228 queries), and a human-validated subset (1 689 queries) that survived both filters — plus a normalised Composition Gap metric to quantify how much a retriever actually composes.

By the numbers

How CIRCUS works

1. Encode each query three times for every retriever in the pool: with both modalities (the standard CIR setting), with only the text (image zeroed out), and with only the image (text dropped).
2. Record the rank of the ground-truth target under each modality.
3. Aggregate across the 11-retriever pool and assign each query one of four labels:
   • composition_required — at least one retriever solves it with both modalities; none solves it unimodally.
   • unresolved — no retriever solves it under any modality.
   • shortcut_solvable — at least one retriever solves it with only image or only text. Excluded.
   • shortcut_free — the union of composition_required and unresolved.
4. Human validation: re-show the surviving queries to multiple annotators using the rubric below. Queries are kept only when a majority of annotators marked them as well-formed CIR.

Validity rubric (used during human validation)

Inter-annotator agreement

The full write-up — including Krippendorff's α — is in annotations/agreement_report.md.

Shortcut audit (Table 1)

Per-benchmark split sizes after running the audit across all 11 retrievers at $K = 10$.

“shortcut-free” = composition_required ∪ unresolved.

Human-validated subset (Table 2)

For CIRR and CIRCO the full shortcut-free residue was audited; for FashionIQ and LaSCo a stratified sample of 1 000 + 1 000 was audited.

The Composition Gap

Once the subsets are built, we need to ask: does a retriever actually use both modalities? Following the paper, we define the normalised Composition Gap:

$\mathrm{CompGap}$ measures the fraction of ranking quality that cannot be recovered from either unimodal input. Larger values mean the retriever genuinely needs both modalities; values close to zero mean one modality is enough.

We use a normalised gap because absolute nDCG varies a lot across splits — the shortcut-free and validated subsets are harder, so the absolute multimodal score drops. Normalising by $\mathrm{MM}$ makes a fair comparison possible. An MRR variant $\mathrm{CompGap}_{\mathrm{MRR}} = 1 - \max(I,T)/\mathrm{MM}_{\mathrm{MRR}}$ follows the same trends and is reported in the appendix.

Recall@10 across the audited subsets

Recall@10 (%) on the original benchmark (Full), the shortcut-free subset (SF), and the validated subset (V). Image-only and text-only columns are omitted because, by construction, no unimodal configuration solves these queries within top-10. Best in column is bold, second-best underlined.

Retriever pool

9 open-source + 2 commercial APIs:

• E5-Omni · GME-Qwen2VL · LamRA · LamRA-Qwen2.5VL · MM-Embed
• Qwen3-VL-Embedding (2B and 8B) · Rzen-Embed · VLM2Vec-V2
• Gemini Embedding 2 · Voyage Multimodal 3.5

Reproduce in five stages

1. Envs — one conda env per retriever family (envs/create_*.sh).
2. Per-(dataset, retriever) retrieval — retrieval/run_generate_retrieval_data*.sh writes one JSON per pair with multimodal / text-only / image-only ranks.
3. Aggregate — retrieval/aggregate_retrieval_data.py turns ranks into the audit labels (already shipped under shortcut_audit/).
4. Validated subset — final_dataset/build_validated_subsets.py derives final_dataset/query_jsonl/ from annotations/users/.
5. Re-evaluation — subset_evaluation/run_subset_eval.sh writes Recall@10 (and the two ablations, when requested) per (dataset, subset, retriever) to subset_evaluation/logs/results.jsonl.

Step-by-step instructions are in the repository README.

Citation

@inproceedings{circus,
  title  = {Do Composed Image Retrieval Benchmarks Require Multimodal Composition?},
  author = {Attimonelli, Matteo and De Bellis, Alessandro and Gema, Aryo Pradipta and
            Saxena, Rohit and Sekoyan, Monica and Kwan, Wai-Chung and Pomo, Claudio and
            Suglia, Alessandro and Jannach, Dietmar and Di Noia, Tommaso and
            Minervini, Pasquale},
  year   = {2026},
  eprint = {2605.14787},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url    = {https://arxiv.org/abs/2605.14787},
}