VOS: Learning What You Don't Know by Virtual Outlier Synthesis | Summary

This article explains the key points of VOS: Learning What You Don't Know by Virtual Outlier Synthesis.

• 2022-02-02 (arXiv), ICLR 2022
• Du, Xuefeng, Wang, Zhaoning, Cai, Mu, Li, Yixuan.
• Department of Computer Sciences, University of Wisconsin - Madison
• Paper (Reviewed version: v4 (2022-05-09))

Read this article in Korean

Summary

• Neural networks can confidently misclassify unfamiliar objects because training provides little supervision for out-of-distribution (OOD) inputs, whose categories fall outside the training labels. Against the best listed ResNet-50 object-detection baseline trained without real outlier data, Virtual Outlier Synthesis (VOS) reduces FPR95 by up to 9.36 percentage points.
• VOS samples low-likelihood feature vectors from estimated class-conditional Gaussian distributions and trains an uncertainty branch to distinguish them from in-distribution (ID), or known-class, examples. It uses no auxiliary real outlier dataset and applies to object detection and image classification. For object detection, see Detecting Twenty-thousand Classes using Image-level Supervision | Summary.
• On PASCAL-VOC with MS-COCO as OOD data, VOS with ResNet-50 achieves 47.53% FPR95, the false-positive rate on OOD samples at 95% true-positive rate on ID samples, and 48.9% mean average precision (mAP), an object-detection accuracy measure. The energy-score baseline obtains 56.89% FPR95 and 48.7% mAP; a separate real-outlier exposure comparison achieves a higher area under the receiver operating characteristic curve (AUROC), a measure of ID–OOD separation, than VOS.

1 Introduction

Models trained only on known categories can assign high confidence to unfamiliar objects, as illustrated by a detector labeling a moose as a pedestrian. The paper seeks a more compact boundary around ID features by generating training signals for unknowns.

The synthetic example shows high ID scores extending far from training clusters without outlier training, while VOS concentrates high scores near the clusters; the moose example illustrates the corresponding detection failure.

2 Problem Setup

The object-level task assigns each predicted bounding box an ID or OOD decision, allowing different regions of the same image to be treated differently. A detected object is OOD when its semantic category lies outside the training label set; image classification is the special case in which the box covers the image.

3 Method

VOS combines feature-space outlier synthesis, an uncertainty regularizer trained alongside the recognition task, and thresholded uncertainty scores at inference.

The diagram connects class-conditional feature sampling to an uncertainty branch trained jointly with classification and localization, showing how VOS fits into an object detector.

3.1 VOS: VIRTUAL OUTLIER SYNTHESIS

The method models penultimate-layer object features as class-conditional multivariate Gaussians with class-specific means and a shared covariance, estimated online from per-class queues. It samples from each Gaussian and selects vectors below a likelihood threshold, then passes these virtual outliers through the classifier’s final linear layer.

The displayed two-dimensional feature projection shows distinct clusters for a subset of PASCAL-VOC classes, motivating class-conditional modeling without establishing that the original features are exactly Gaussian.

3.2 Unknown-Aware Training Objective

A binary sigmoid loss trains a nonlinear multilayer perceptron (MLP), a small neural network, on energy scores derived from classifier outputs to give ID features high ID probability and virtual outliers low ID probability. For object detection, the energy uses learnable class weights, and the uncertainty loss is added to classification and box-regression losses with weight β.

3.3 Inference-time OOD Detection

At inference, the detector scores each predicted box with the learned uncertainty branch and compares its ID probability with a threshold γ. The threshold can be chosen to retain 95% of ID examples.

4 EXPERIMENTAL RESULTS

The experiments evaluate object-level OOD detection and image-level classification, measuring OOD separation alongside performance on the original ID task.

4.1 Evaluation on Object Detection

The detector experiments train on PASCAL-VOC or Berkeley DeepDrive (BDD-100k) and use manually screened subsets of MS-COCO and OpenImages as OOD data. With ResNet-50 and MS-COCO OOD, VOS obtains 47.53% FPR95 on PASCAL-VOC and 44.27% on BDD-100k; the corresponding best listed baseline values are 56.89% and 47.10%.

VOS-ResNet50 improves OOD separation over the listed ResNet-50 baselines while maintaining similar ID mAP; VOS-RegX4.0 uses a different backbone, so its gains also reflect an architecture change.

Under the PASCAL-VOC and MS-COCO setup, virtual feature outliers attain 88.70% AUROC versus 85.98% for Gaussian noise and 83.67% for images generated by a generative adversarial network (GAN). On BDD-100k, the proposed uncertainty loss yields 44.27% FPR95 versus 54.29% for the squared hinge-loss variant; Outlier Exposure, which trains with real outliers, achieves 90.18% AUROC versus VOS’s 88.70% on PASCAL-VOC and COCO.

On the same PASCAL-VOC and COCO evaluation, VOS exceeds the listed image synthesis, noise and negative-proposal alternatives in AUROC while retaining 48.9% mAP.

The proposed loss has lower FPR95 than hinge loss, constant energy weights and an additional outlier class for both ID datasets.

4.2 Evaluation on Image Classification

With CIFAR-10 as ID data, results averaged over Textures, SVHN, Places365, LSUN-C, LSUN-Resize and iSUN give VOS FPR95 of 24.87% on WideResNet-40 and 22.47% on DenseNet-101. WideResNet ID accuracy is 94.68% with VOS versus 94.84% for the pretrained model.

Averaged over six OOD datasets, VOS has the lowest reported FPR95 and highest AUROC on both WideResNet-40 and DenseNet-101.

4.3 Qualitative Analysis

The qualitative comparison shows VOS marking several unfamiliar MS-COCO objects as OOD where a BDD-100k-trained Faster-RCNN assigns known-class labels. Some false positives remain, including a truck prediction with lower confidence under VOS.

The paired predictions illustrate that VOS can reject unfamiliar objects that the baseline assigns to known BDD-100k classes, while some known-class predictions persist.

5 Related Work

The paper situates VOS among post-hoc uncertainty scores, regularization using real or generated outliers, and object-level uncertainty methods. Its class-conditional Gaussian estimates are updated during training, whereas the compared Mahalanobis approach uses feature distributions for post-hoc detection (Lee et al., 2018b).

6 Conclusion

The authors conclude that synthesizing low-likelihood features can improve OOD separation while preserving ID task performance across the evaluated detectors and classifiers.

Reproducibility Statement

The paper specifies public datasets, baseline settings, training details and pseudocode, and provides the implementation at https://github.com/deeplearning-wisc/vos. Object detection training uses Detectron2: https://github.com/facebookresearch/detectron2.

Ethics Statement

The authors frame the work as improving reliability in applications such as autonomous driving and state that the study did not involve human subjects.

Appendix

• A EXPERIMENTAL DETAILS and B SOFTWARE AND HARDWARE: PASCAL-VOC and BDD-100k models train for 18,000 and 90,000 iterations, respectively; uncertainty regularization begins after two-thirds of training with β = 0.1. Experiments use Python 3.8.5, PyTorch 1.7.0 and NVIDIA GeForce RTX 2080Ti GPUs.
• C EFFECT OF HYPERPARAMETERS: The authors use one lowest-likelihood sample from 10,000 candidates per class and queues of 1,000 for PASCAL and 300 for BDD-100k. Sensitivity tests use speckle-noised PASCAL-VOC validation data rather than the main OOD test datasets; β = 0.5 reduces mAP to 36.4 and raises FPR95 to 99.33%.
• D ADDITIONAL VISUALIZATION RESULTS and E BASELINES provide further examples and implementation settings. For ODIN and Mahalanobis, input noise is set to zero because region cropping and ROIAlign, the detector’s region feature extraction operation, prevent end-to-end differentiation.
• F VIRTUAL OUTLIER SYNTHESIS USING EARLIER LAYER finds that the penultimate layer outperforms the preceding layer: on BDD-100k with COCO OOD, FPR95 is 44.27% versus 49.66%. G relates learned energy weights to class frequencies, and H uses a two-dimensional UMAP feature projection to visualize virtual outliers around the Person feature cluster.
• I Discussion on the detected, rejected and ignored OOD objects during inference emphasizes that VOS targets the classification of detected boxes. Under the same default thresholds on BDD-100k with a ResNet backbone, the authors report 25% more detected OOD boxes and 63% more rejected OOD samples than vanilla Faster-RCNN.

Brief Thoughts

The matched-backbone comparisons and synthesis ablations support the value of feature-space outliers, but performance remains dataset-dependent: the PASCAL-VOC ResNet-50 result has 47.53% FPR95 on COCO. The Gaussian assumption and dependence on detector proposals limit what these experiments establish about unfamiliar objects that are not proposed or that differ from the tested OOD sets.