Detecting Twenty-thousand Classes using Image-level Supervision | Summary

This article explains the key points of Detecting Twenty-thousand Classes using Image-level Supervision.

• 2022-01-07 (arXiv)
• Zhou, Xingyi, Girdhar, Rohit, Joulin, Armand, Krähenbühl, Philipp, Misra, Ishan.
• Meta AI, The University of Texas at Austin
• Paper, version 3 (revised 2022-07-29; this review uses the revised version)

Read this article in Korean

Summary

• Object detectors are limited by the number of categories with costly bounding-box annotations. On open-vocabulary LVIS, a benchmark that tests classes without training box labels, Detic improves mask mean average precision (mAP), the average precision of instance masks across classes and overlap thresholds, by 2.4 points overall and 8.3 points on novel classes over a box-supervised baseline.
• Detic trains on both box-labeled images and image-level class labels, applying the latter to the largest candidate region. With all 21K ImageNet classes, it transfers without finetuning to Objects365 and OpenImages, although it remains below models trained with target-dataset box labels.

1 Introduction

Detection combines localization, finding an object’s box, with classification, naming it. Detic separates these tasks because region proposal networks, which generate candidate object boxes, can already localize many classes absent from box-labeled training data; classification data can then expand the detector’s vocabulary.

The examples show missed or mislabeled common objects, while the class-frequency curves illustrate why image classification data can supply a wider vocabulary than detection annotations.

2 Related Work

Weakly supervised object detection uses image-level labels, while semi-supervised variants also use box labels. Prior methods often assign image labels to candidate boxes using predicted class scores; Detic instead uses a fixed assignment and co-trains on image-labeled data.

3 Preliminaries

The detection training set provides boxes and classes, whereas the classification set provides image-level classes; their vocabularies and the test vocabulary may differ. For open-vocabulary detection, the paper uses fixed class-name embeddings from CLIP, a vision-language model, as classifier weights [17, 42].

4 Detic: Detector with Image Classes

For box-labeled images, Detic applies standard proposal, box regression, and classification losses. For image-labeled images, it applies binary cross-entropy, a classification loss, to the features of the largest proposed box, shares classification parameters across datasets, and weights this loss by λ = 0.1; whole-image and maximum-objectness proposals are alternatives.

The two training branches show that localization losses require box labels, while image labels update classification through the largest candidate region.

5 Experiments

LVIS contains 1,203 classes and 100K images; the open-vocabulary split removes annotations for 337 rare classes and retains 866 base classes. Most experiments use IN-L, the 1.2M-image ImageNet subset covering 997 LVIS classes, or Conceptual Captions (CC), a caption dataset filtered to 1.5M images covering 992 classes; LVIS results use mask mAP.

5.1 Implementation details

The main baseline uses the CenterNet2 detector with a ResNet50 backbone, ImageNet-21K pretraining, a CLIP classifier, 640×640 detection inputs, and a 4× schedule of approximately 48 LVIS epochs. Joint training samples detection and classification mini-batches in a 1:1 ratio, uses 320×320 image-labeled inputs, and finetunes a converged baseline for another 4× schedule.

5.2 Prediction-based vs non-prediction-based methods

Across three runs on open-vocabulary LVIS, max-size achieves 32.4 overall and 24.6 novel mask mAP with IN-L, versus 31.2 and 20.4 for the strongest prediction-based alternative. With CC it achieves 30.9 and 19.5, compared with 29.7 and 16.9 for the best prediction-based novel-class result.

The three-run comparison shows max-size gaining 8.3 novel-class points over the baseline with IN-L, while remaining slightly below the all-class box-supervised model on novel classes.

On an annotated IN-L subset, the largest proposal has a cover rate of 92.8%, versus 69.0% for prediction-based assignment. Cover rate counts objects whose ground-truth box overlaps the assigned proposal by more than half of the ground-truth area. The selected region can still be loose or miss small objects, and coverage alone does not fully explain final performance.

The selected max-size boxes remain more stable across training snapshots than the prediction-selected boxes, illustrating the proposed assignment mechanism.

5.3 Comparison with a fully-supervised detector

With IN-L, Detic reaches 32.4 overall and 24.6 novel mask mAP without novel-class box labels, compared with 31.1 and 25.5 for the detector trained with box labels for all classes. Novel-class performance remains 0.9 points below full supervision in this comparison.

5.4 Comparison with the state-of-the-art

Under ViLD’s Mask R-CNN-ResNet50 setup, Detic obtains 26.8 overall and 17.8 novel mask mAP versus 25.5 and 16.6 for ViLD-ensemble [17]. On open-vocabulary COCO, using OVR-CNN’s architecture and data, Detic obtains 45.0 overall and 27.8 novel box mAP at an intersection-over-union threshold of 0.5, versus 39.9 and 22.8 for OVR-CNN [72]; intersection over union measures predicted and ground-truth box overlap.

With the same architecture and training data as OVR-CNN, Detic improves novel-class box mAP50 from 22.8 to 27.8.

5.5 Detecting 21K classes across datasets without finetuning

The 21K-class Swin-B model samples 50 classes per iteration for the classification loss. Without target-dataset finetuning, it reaches 21.5 box mAP on Objects365 and 55.2 box mAP50 on OpenImages, versus 19.1 and 46.2 for the box-supervised baseline; dataset-specific oracles reach 31.2 and 69.9.

Full ImageNet-21K supervision improves transfer over the smaller IN-L subset, but both target-dataset oracle models remain ahead.

5.6 Ablation studies

Detic improves all tested classifiers: with a trained classifier, novel mask mAP rises from 0 to 17.4, while CLIP reaches 24.9. Image-level co-training also improves novel mAP by 8.1 points with ImageNet-1K pretraining and 8.5 points with ImageNet-21K pretraining.

5.7 The standard LVIS benchmark

On standard LVIS, with box and mask labels for all classes and additional IN-L image labels, the Swin-B Detic model reaches 41.7 overall and 41.7 rare-class mask mAP. Its matched baseline scores 40.7 overall and 35.9 on rare classes; frequent-class mAP decreases from 43.1 to 42.6.

The matched Swin-B comparison shows that the overall gain is concentrated in rare classes, while frequent-class performance falls slightly.

6 Limitations and Conclusions

The method assigns all image labels to the same region and does not use overall dataset statistics. The authors also state that open-vocabulary generalization is not guaranteed in extreme domains.

Appendix

• A Region proposal quality: At 1,000 proposals and intersection-over-union 0.5, rare-class recall is 78.5 without rare-class training annotations versus 79.7 with them.
• B Direct captions supervision: Combining image-label and caption losses yields 21.3 novel mask mAP on LVIS and 27.8 novel box mAP50 on COCO, exceeding either loss alone.
• C LVIS baseline details: Federated Loss, CenterNet2, augmentation, CLIP classification, and ImageNet-21K pretraining contribute to a 31.5 mask mAP baseline, versus 22.4 for the detectron2 baseline.
• D Resolution change for classification data: Reducing image input size from 640 to 320 improves IN-L results by roughly one point, with little effect for CC.
• E Prediction-based losses implementation details: The comparisons include weighted proposal scores, the highest predicted class score, and DLWL*, a simplified clustered loss without the original bootstrapping stage [3, 44, 45].
• F More comparison between prediction-based and non-prediction-based methods: Better backbones and rare-class annotations narrow the max-size advantage; coverage and assignment consistency do not alone predict final accuracy.
• G ViLD baseline details: The reproduced Mask R-CNN-ResNet50-FPN setup trains from scratch at 1024×1024, followed by ImageNet finetuning with max-size loss.
• H Open-vocabulary COCO benchmark details: The split uses 48 base and 17 novel classes; evaluation includes all COCO validation images, and the matched Faster R-CNN setup uses 800×1333 inputs.
• I Compare to MosaicOS [73]: Under its baseline setup, Detic with a trained classifier reaches 25.1 overall and 18.6 rare mask mAP, compared with MosaicOS at 24.5 and 18.3.
• J Generalization to Deformable-DETR.: Applying image supervision to the largest predicted query improves box mAP from 31.7 to 32.5 and rare-class box mAP from 21.4 to 26.2.
• K Improvements breakdown to classes: Most gains occur in classes with image labels; with IN-L, mask mAP for classes without those labels rises from 27.6 to 28.1.
• L mAPᶠⁱˣᵉᵈ evaluation: Under the alternative metric [8], the displayed overall scores rise from 31.2 to 33.4 and novel scores from 18.2 to 26.7; the table prints gains of 2.3 and 8.5 points, respectively.
• M Image Attributions: The OpenImages examples in Figure 5 credit The Local People Photo Archive, Michael Gil, Alden Chadwick, and Paxson Woelber under CC BY 2.0.

Brief Thoughts

The comparison with prediction-based assignments shows gains on both object-centric IN-L and CC images, which can contain multiple objects. The cross-dataset oracle gap and the shared region for multiple labels delimit the evidence.