A ConvNet for the 2020s | Summary

This article explains the key points of A ConvNet for the 2020s.

• 2022-01-10 (arXiv)
• Liu, Zhuang, Mao, Hanzi, Wu, Chao-Yuan, Feichtenhofer, Christoph, Darrell, Trevor, Xie, Saining.
• Facebook AI Research (FAIR), UC Berkeley
• Paper, version 2 (revised 2022-03-02; this review uses the revised version)

Read this article in Korean

Summary

• This paper asks whether modernized convolutional neural networks (ConvNets), which process images using learned local filters, can match vision Transformers on visual recognition. At 384×384 resolution after ImageNet-1K training, ConvNeXt-B reaches 85.1% top-1 accuracy versus 84.5% for the similarly sized Swin-B, with 95.7 versus 85.1 images per second on a V100 GPU.
• The authors progressively adapt a ResNet’s training recipe and architecture to obtain ConvNeXt, using per-channel spatial convolutions, larger filters and simplified blocks without attention. With ImageNet-22K pre-training, ConvNeXt-XL reaches 87.8% ImageNet-1K top-1 accuracy; ConvNeXt also matches or exceeds comparable Swin models on the reported COCO and ADE20K evaluations.

1. Introduction

Vision Transformers process image patches with attention, but global attention becomes costly at high resolution. The paper investigates whether training and architectural differences confound comparisons with hierarchical Transformers such as Swin, which use local windows and multiple feature resolutions.

The plotted model families show ConvNeXt reaching accuracy levels comparable to hierarchical Transformers across the displayed compute scales; bubble area represents FLOPs, and the pre-training and image-resolution conditions differ across the two halves.

2. Modernizing a ConvNet: a Roadmap

The modernization sequence starts from ResNet-50 and ResNet-200 in regimes of approximately 4.5G and 15.0G floating-point operations (FLOPs), respectively. The authors hold training settings fixed across the architectural sequence and approximately control computation, although intermediate models vary in FLOPs.

The progression exposes both gains and temporary regressions: depthwise convolution and moving it earlier reduce accuracy before width and kernel changes recover it, while the final model exceeds Swin-T in this comparison.

2.1. Training Techniques

A 300-epoch recipe uses the AdamW optimizer, image augmentation and regularization techniques. It raises ResNet-50 accuracy from 76.1% to 78.8%; subsequent ResNet-50-regime results are averages over three random seeds.

2.2. Macro Design

Changing the four stage block counts from (3, 4, 6, 3) to (3, 3, 9, 3) raises accuracy from 78.8% to 79.4%. Replacing the stem, the network’s initial image-processing layers, with a non-overlapping 4×4, stride-4 convolution increases it to 79.5%.

2.3. ResNeXt-ify

Depthwise convolution applies a spatial filter separately to each channel, separating spatial mixing from the channel mixing performed by 1×1 convolutions. After this substitution reduces accuracy and computation, increasing the initial width from 64 to 96 channels yields 80.5% accuracy at 5.3G FLOPs.

2.4. Inverted Bottleneck

An inverted bottleneck expands the block’s hidden channel dimension to four times its input dimension before reducing it again. This change improves accuracy from 80.5% to 80.6% and reduces total computation to 4.6G FLOPs; in the larger regime it improves 81.9% to 82.6%.

The diagram shows how the spatial convolution moves from the narrow middle of a conventional bottleneck to the expanded middle, then ahead of expansion, reducing the channels on which spatial processing operates.

2.5. Large Kernel Sizes

Moving the depthwise convolution before the channel expansion reduces computation but temporarily lowers accuracy to 79.9%. Increasing its kernel, the spatial filter, from 3×3 to 7×7 recovers accuracy to 80.6% at roughly constant FLOPs, with no further gain beyond 7×7 in the tested regimes.

2.6. Micro Design

Replacing the rectified linear unit (ReLU) with the smoother Gaussian error linear unit (GELU) activation leaves accuracy at 80.6%, while retaining only one activation per block raises it to 81.3%. Removing normalization layers and replacing BatchNorm with LayerNorm reaches 81.5%; separate downsampling convolutions with normalization at resolution changes stabilize training and yield 82.0%, versus Swin-T’s 81.3% in the modernization comparison.

ConvNeXt combines one depthwise spatial convolution, two pointwise channel projections, one normalization and one activation in a residual block; the illustrated Swin block has two residual connections and specialized attention operations.

3. Empirical Evaluations on ImageNet

ConvNeXt-T/S/B/L are constructed at complexities similar to Swin-T/S/B/L, and ConvNeXt-XL tests further scaling. Variants change the number of channels and blocks per stage, with channels doubling at each new stage.

3.1. Settings

ImageNet-1K has 1,000 classes and 1.2M training images; ImageNet-22K has 21,841 classes and approximately 14M images. ImageNet-1K training uses 300 epochs, batch size 4096 and learning rate 4e-3; ImageNet-22K pre-training uses 90 epochs, followed by 30 epochs of ImageNet-1K fine-tuning at learning rate 5e-5 and batch size 512.

3.2. Results

At 224×224 after ImageNet-1K training, ConvNeXt-T scores 82.1% versus Swin-T’s 81.3%, both at 4.5G FLOPs. After ImageNet-22K pre-training, ConvNeXt-L at 384×384 scores 87.5% versus Swin-L’s 87.3%, while the larger ConvNeXt-XL reaches 87.8%.

The matched 384×384 ImageNet-1K-trained B models provide an accuracy and V100 throughput comparison, while the lower portion shows results after larger-data pre-training.

3.3. Isotropic ConvNeXt vs. ViT

An isotropic architecture keeps the same feature-map resolution at every depth. At 224×224, isotropic ConvNeXt-S/B/L obtain 79.7%/82.0%/82.6%, compared with 79.8%/81.8%/82.6% for vision Transformer (ViT) S/B/L models at similar parameter counts and FLOPs.

The three isotropic comparisons show accuracy differences of at most 0.2 percentage points, with lower reported training memory for ConvNeXt at each size under the stated V100 batch condition.

4. Empirical Evaluation on Downstream Tasks

On COCO, object detection locates objects and instance segmentation predicts their masks; with ImageNet-22K pre-training and Cascade Mask R-CNN, ConvNeXt-B achieves 54.0 box average precision (AP) and 46.9 mask AP versus Swin-B’s 53.0 and 45.8. On ADE20K semantic segmentation, which assigns a class to each pixel, the same pre-training gives ConvNeXt-B 53.1 mean intersection over union (mIoU), versus 51.7 for Swin-B; UperNet is trained for 160K iterations with batch size 16 and evaluated at multiple scales.

The table distinguishes detection and mask scores and shows that gains vary by model size and detector: the T and S Cascade results tie Swin on the main box and mask AP values, while the pre-trained B comparison favors ConvNeXt.

With ImageNet-22K pre-training and 640×640 crops, ConvNeXt-B improves mIoU by 1.4 points over Swin-B, while the L comparison differs by 0.2 points.

5. Related Work

The paper situates ConvNeXt among hybrid convolution-attention models and recent convolution-based approaches, including work replacing Swin attention with depthwise convolution [25]. Its contribution includes a modernization trajectory for a standard ResNet; the authors note that the individual design choices had been studied previously.

6. Conclusions

The authors conclude that a pure ConvNet assembled from previously studied components can compete with hierarchical vision Transformers across the reported benchmarks. The code is available at https://github.com/facebookresearch/ConvNeXt.

Appendix

• A. Experimental Settings specifies training, fine-tuning and downstream hyperparameters: modernization experiments disable exponential moving average (EMA), an average of model weights, and downstream experiments initialize from final pre-training weights. Fine-tuning generally starts from final rather than EMA weights; ImageNet-1K-pre-trained ConvNeXt-L is an exception because of overfitting.
• B. Robustness Evaluation reports that ImageNet-22K-pre-trained ConvNeXt-XL achieves 69.3%, 68.2% and 55.0% on ImageNet-A, ImageNet-R and ImageNet-Sketch without specialized modules or additional fine-tuning. C. Modernizing ResNets: detailed results and D. Detailed Architectures provide both modernization regimes and stage-level specifications; the ResNet-50 results include three-seed standard deviations.
• E. Benchmarking on A100 GPUs reports up to 49% higher throughput for ConvNeXt than comparable Swin models using TensorFloat32 and the channel-last memory layout. F. Limitations notes that the evaluated vision tasks do not cover all applications and that cross-attention may be useful for multimodal interactions; G. Societal Impact discusses computing demands, carbon emissions and potential dataset biases.

Brief Thoughts

The training-recipe baseline makes the architectural comparison more informative: ResNet-50 gains 2.7 percentage points before architectural changes, and several later changes have modest or negative intermediate effects. The sequence is path-dependent, so its incremental results do not establish that each component would have the same effect in isolation or on other tasks.