Gorio Tech Blog search

Summary explanation: RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs

11 Sep 2026 | Paper Review Open-Vocabulary Scene Graph Generation Relation Prediction Positive-Unlabeled Learning

Contents

This article explains the key points of RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs.

  • 2026-09-11 (arXiv)
  • Neau, Maëlic.
  • Independent Researcher
  • Paper
  • Github

Read this article in Korean

Summary

  • RelateAnything predicts relations for arbitrary supplied regions and inference-time predicate strings without using object labels as network inputs. The paper contributes a 53M-parameter model, the RA-4M free-text relation corpus, and OV-SGG-Bench; the compiled deployment path is reported at 20 ms per frame on an A40, while the eager end-to-end comparison is 25.0 ms.

The overview links the corpus, label-free model interface, and six-axis protocol. It identifies 40.1 as the OVS composite for RelateAnything and 11.8 for OvSGTR, while clarifying that A1 uses a stronger available baseline.

Pipeline overview linking RA-4M, the relation model, and the OV-SGG-Bench evaluation axes.
Pipeline overview linking RA-4M, the relation model, and the OV-SGG-Bench evaluation axes.

The RelateAnything model

Given an image and arbitrary boxes or masks, the model scores ordered region pairs against predicate strings supplied at inference. Its 19,103-string bank combines 10,102 RA-4M predicates with 9,001 strings from other corpora; training treats absent annotations as positive-unlabeled rather than automatically negative.

Architecture

A fine-tuned DINOv3 visual path pools contextual features for subject, object, union, and contact regions, appends 19 geometry features, and refines sampled pairs with a Relation Transformer and deformable scene read. Two cosine-scoring branches are mixed by a gate computed from each predicate text embedding, so new predicate strings can replace the embedding bank without retraining.

The diagram shows external regions entering a DINOv3-based pair model while detector labels are discarded. It also depicts offline predicate embeddings, the predicate-conditioned mixing gate, pair sampling, and synonym-aware discounted-negative training.

Architecture showing the visual path, frozen text path, predicate gate, scoring function, and training-only InfoNCE loss.
Architecture showing the visual path, frozen text path, predicate gate, scoring function, and training-only InfoNCE loss.

Training with 10^4 predicates

The batch-local InfoNCE loss treats synonym groups as positives and discounts a negative predicate q when corpus co-annotation estimates that q is likely true given p; directional inverses retain full negative weight. A distilled 512-dimensional text encoder separates inverse predicates while retaining synonym structure: mean inverse cosine falls from 0.92 to 0.09, synonym cosine is 0.71, and synonym-versus-inverse AUC rises from 0.854 to 0.989.

The teacher text space places inverse spatial predicates close together, whereas the distilled student separates inverse clusters while retaining synonym groups. The visual supports the motivation for modifying text targets used by cosine-based relation scoring.

Teacher and distilled text-embedding geometry for spatial predicates, shown with cosine matrices and t-SNE plots.
Teacher and distilled text-embedding geometry for spatial predicates, shown with cosine matrices and t-SNE plots.

RA-4M: a verified free-text relation corpus

RA-4M contains 474,413 images, 4,282,531 relations, and 10,102 distinct free-text predicates. A 26B open-weight VLM generates semantic, grow, and spatial relations from numbered box markers; deterministic geometry checks reject 11.3% of raw candidates, repair compatible directional role errors, and leave geometry-unconstrained errors as stated residual risk. The paper reports no human-audited corpus-precision estimate.

OV-SGG-Bench: analysis of scene-graph evaluation and a cross-dataset protocol

OV-SGG-Bench argues that standard scene-graph recall is confounded by corpus overlap, object-category frequency priors, annotation propensity, duplicate-credit matching, and detector operating points. Its OVS composite is a harmonic mean over chance-corrected A1, A2, A4, A5, and A6 scores; A3 is reported separately because matcher choice can change comparisons substantially.

Varying confidence threshold and box cap with fixed detector weights changes the pair-recall ceiling substantially. This motivates reporting the detector operating point and ceiling alongside detection-mode relation results.

Pair-recall ceilings across detector operating points for VG150 and PSG.
Pair-recall ceilings across detector operating points for VG150 and PSG.

The six axes

The six axes respectively measure cross-dataset transfer, precision on adjudicated negatives, full-vocabulary prediction, shared-detector deployment, judge-accepted graph information, and adversarial spatial relations. A2 and A6 are designed to resist annotation-propensity and category-prior shortcuts, whereas A4 is explicitly bounded by detector pair recall.

Results

RelateAnything reports A1 F1@50 (mR@50) of 36.9 (28.2) on VG150, 34.7 (30.6) on PSG, 37.8 (29.5) on IndoorVG, and 18.7 (12.7) on zero-shot HICO-DET. It reports 72.6 mean fAP on Haystack, 20.0 PSG detection-mode wR@50 with shared YOLO-World boxes, 18.6 accepted-information bits per image at matched graph depth, and 69.0 SpatialSense macro AUC; the latter does not by itself establish visual spatial reasoning because SpatialSense’s boxes-only baseline reaches 68.8.

The table assembles results from the six axes and compares RelateAnything with the strongest available baseline per axis. Its baseline column is an envelope over systems for A1 and A3, whereas the 40.1 OVS composite is specifically computed against OvSGTR.

Six-axis benchmark results for RelateAnything and the best available baseline per axis.
Six-axis benchmark results for RelateAnything and the best available baseline per axis.

The table separates explicit-negative precision, full-vocabulary retrieval, and shared-detector deployment. It also records that the sampler covers 89.0% of labelled Haystack cells, compared with 100.0% for OvSGTR, while RelateAnything has higher fAP and P-AUC.

Detailed A2 precision, A3 open-vocabulary, and A4 detection-mode results.
Detailed A2 precision, A3 open-vocabulary, and A4 detection-mode results.

Analysis: decoupling object and relation prediction

A label-pair lookup reproduces 88–90% of OvSGTR predictions but 69–70% of RelateAnything predictions, despite the latter receiving no labels. Lesion and variance analyses attribute 87–93% of the semantic-logit variance to pair context and about 0.1% and 0.0% to subject and object identity features; under this recipe, relation fine-tuning sharpens class selectivity without producing strong relational dense features.

Fine-tuning shifts patch similarity around the hand-object interaction toward the hand and bird. The associated retrieval probe nevertheless finds that this feature identifies the handled object more reliably than the verb, limiting the decoupling conclusion to the tested recipe.

Patch-similarity maps before and after relation fine-tuning for hand and bird queries.
Patch-similarity maps before and after relation fine-tuning for hand and bird queries.

Analysis: in-domain versus cross-dataset measurement

Doubling the corpus improves in-domain R@50 and mR@50 by 15% and 56%, but improves zero-shot transfer by only 2.5–3.8% and 9.9–11.9%, respectively; the authors estimate that in-domain slopes overstate transfer by about 5×. Adding a sparse Open Images V6 relation source was negative on every measured axis in the tested setting, consistent with unannotated true relations becoming damaging false negatives under the objective.

The figure controls gradient steps when comparing full and half corpora, then contrasts large in-domain improvements with substantially smaller zero-shot gains. It supplies the basis for the approximate 5× gap between in-domain and transfer scaling slopes.

Effects of doubling the corpus measured in-domain and under zero-shot transfer.
Effects of doubling the corpus measured in-domain and under zero-shot transfer.

Discussion and limitations

The reported speed comparison is configuration-dependent: eager RelateAnything plus YOLO-World takes 25.0 ms on an A40 versus 194.0 ms for OvSGTR Swin-T, but box budget, resize, precision, and pair scoring differ. Limitations include unheld benchmark predicate strings in most transfer tests, several single-seed conclusions, no human audit of RA-4M precision, imperfect A5 judge controls, and detector pair recall as the principal deployment bottleneck.

The latency comparison reports lower eager A40 end-to-end latency for RelateAnything plus YOLO-World, but documents non-controlled differences in box count, resize, numerical precision, and pair scoring. It therefore supports an efficiency comparison of deployed configurations, not an architecture-only speed claim.

End-to-end latency and parameter comparison with OvSGTR under the reported deployment configurations.
End-to-end latency and parameter comparison with OvSGTR under the reported deployment configurations.

Appendix

  • The appendix specifies architecture and training details, geometric verification, text-space objectives, evaluation and calibration protocols, representation probes, scaling, ablations, generation prompts, and latency measurements. It reports that ViT-S/16+ was selected because ViT-B/16 adds 114% parameters yet changes the released-family OVS-F1 by −0.1%, while scoring the full 19,103-string vocabulary adds 0.7–0.8 ms per A40 frame at batch 1.

Brief Thoughts

The paper’s main contribution is a combined argument that relation prediction should be evaluated independently of corpus overlap, label priors, sparse annotations, and unstated detector choices. Its protocol analysis and ablations are extensive, but conclusions about corpus precision and visual spatial understanding remain appropriately conditional on future human audits and broader held-out-concept evaluations.

Related Posts