Toward Training at ImageNet Scale with Differential Privacy | Summary

This article explains the key points of Toward Training at ImageNet Scale with Differential Privacy.

• 2022-01-28 (arXiv; v2 revised 2022-02-09)
• Kurakin, Alexey, Song, Shuang, Chien, Steve, Geambasu, Roxana, Terzis, Andreas, Thakurta, Abhradeep.
• Google Research, Columbia University
• Paper

Read this article in Korean

Summary

• The paper studies ImageNet classification with differential privacy (DP), a guarantee that limits how much a model’s output can reveal about any one training example. A Resnet-18 pretrained on Places365 reaches 47.9% top-1 accuracy at ε = 10 and δ = 10⁻⁶, compared with a few percent for naive DP training at the same ε; the paper’s non-private 90-epoch results are 70.8% for Resnet-18 and 75.3% for Resnet-50.
• The authors combine private stochastic gradient descent, large batches, model and hyperparameter choices, and public-data pretraining. Their JAX implementation reduces the cost of computing the per-example gradients required for private training.

1 Introduction

Models can expose training examples through membership inference, which tests whether an example was used for training, or reconstruction of sensitive data. The authors identify accuracy loss and computational overhead as obstacles to applying DP to ImageNet and release code at https://github.com/google-research/dp-imagenet.

2 Background

The background introduces the privacy guarantee and private optimization algorithm, then examines the cost of their per-example operations.

2.1 Privacy Notions and DP-SGD

For datasets differing by one entry, (ε, δ)-DP bounds the probability of an output on one dataset by e^ε times its probability on the other, plus δ; ε and δ specify the privacy guarantee. Differentially private stochastic gradient descent (DP-SGD) clips each example’s gradient to an ℓ₂ norm C, sums the clipped gradients, and adds Gaussian noise with standard deviation σC; sampling and composition determine the resulting privacy bound.

2.2 Automatically Fast DP Training with JAX

JAX, a computational library, uses automatic vectorization and just-in-time compilation to parallelize per-example gradient computations. On one V100 GPU, private CIFAR-10 training took 1.47 seconds per epoch with JAX, versus 2.64 with Opacus and 11.80 with TF-Privacy; on eight V100 GPUs, ImageNet Resnet-18 took 555.05 seconds privately versus 275.5 seconds without DP.

JAX has the shortest private epoch time for both small-convolutional-network benchmarks, although its non-private MNIST time exceeds Opacus.

The private ImageNet epochs take roughly twice the non-private time for Resnet-18 and about 1.5 times for Resnet-50 on the measured hardware.

3 Effective Training Settings for DP

The experiments combine previously known techniques and assess architecture choice, training duration, tuning, batch size, and public-data pretraining on ImageNet. The authors present these as initial observations rather than an exhaustive search.

3.1 Obs. 1: Choice of model matters

The authors replace batch normalization, which uses information across examples, with group normalization for DP-SGD analysis. At 10 epochs, batch size 1024, ε = 4.6 and δ = 10⁻⁶, Resnet-18 achieves 3.7% top-1 accuracy versus 2.4% for Resnet-50; without DP, the larger model performs better across the reported settings.

Resnet-50 outperforms Resnet-18 at every listed non-private training duration, establishing the comparison for the private experiments.

At ε values 4.6, 13.2 and 71, Resnet-18 leads; at very large ε, Resnet-50 eventually regains the advantage.

3.2 Obs. 2: More epochs is better than lower noise

At a fixed privacy bound, longer training with correspondingly more noise improves accuracy in the reported experiments over fewer, less noisy epochs. For ε = 4.57, Resnet-18 rises from 3.6% at 10 epochs to 7.6% at 90 epochs; the curve at ε = 9.7 × 10⁶ becomes nearly flat much earlier.

Accuracy increases with training duration at the two smaller privacy bounds, while the very large ε curve becomes nearly flat after early epochs.

3.3 Obs. 3: Hyperparameter tuning makes a big difference

The authors recommend jointly tuning clipping norm and learning rate: find a non-private learning rate, identify a clipping threshold with zero noise, select noise for the privacy budget, and sweep the learning rate. Below the threshold, decreasing C by a factor k while increasing the learning rate by k can preserve similar accuracy, although extreme settings can be unstable; the proposed tuning procedure itself does not preserve privacy.

3.4 Obs. 4: Large batch size improves the privacy-utility tradeoff

Large batches can be simulated with virtual steps, which accumulate gradients before an update. With batch size 16 × 1024 and 10 epochs, accuracy is 5.8% at ε = 2.2 × 10⁵; with batch size 1024 × 1024 and 640 epochs, it is 6.2% at approximately ε = 72, illustrating the tradeoff at low accuracy.

Scaling batch size and epochs together keeps accuracy near 56–58% while the reported ε decreases, but all listed privacy bounds are too large for meaningful privacy according to the authors.

The plotted privacy bound falls sharply as batch size increases when the noise-to-batch ratio and number of updates are fixed.

3.5 Obs. 5: Transfer learning from public data significantly boosts accuracy

Places365 supplies 1.8 million public scene images across 365 labels; the authors pretrain Resnet-18, replace its final layer with a 1,000-class ImageNet layer, and privately fine-tune it. At ε = 10, batch size 4 × 1024 and 70 epochs, training from scratch reaches 20.6%, while pretraining and freezing three block groups reaches 40.7%; larger batches eventually hurt in the tested fine-tuning runs.

Pretraining improves every reported result over training from scratch; at 70 epochs, freezing three groups gives the best result in this comparison.

The best observed result is 47.9% with three frozen groups, 70 epochs and batch size 64 × 1024; the largest batch performs worse at the tested durations.

3.6 Combining everything together

The best run freezes three block groups and fine-tunes for 70 epochs with batch size 64 × 1024. It uses a one-epoch warmup and cosine learning-rate decay with maximum 7.68, Nesterov momentum 0.9, weight decay 10⁻⁴, clipping norm C = 1, and noise chosen for ε = 10.

4 Conclusion

The combined configuration establishes a 47.9% ImageNet baseline under the reported privacy parameters, while leaving a substantial accuracy gap to non-private training. The improvement comes from combining training choices using DP-SGD.

Appendix

• A Resnet training without DP and B Resnet training with DP-SGD describe the optimizer and hardware settings. The private implementation adds independent noise on eight replicas, and its Rényi differential privacy accounting, used to calculate the final privacy bound, may overestimate ε, especially at large ε.
• C Experiments with fixed epsilon and different number of epochs uses a binary search for the noise multiplier and reports the underlying accuracy values; D Hyperparameter tuning for DP-SGD presents clipping and learning-rate sweeps, mostly limited to 10 epochs by the compute budget.
• E Details of large batch training sweep reports batch and epoch combinations; F Pre-training on Places365 dataset reports 54.96% accuracy after 80 epochs. G DP-SGD finetuning experiments defines frozen block groups and compares private and non-private fine-tuning.
• H Experiments on CIFAR-10 tests simpleVGG convolutional architectures for 100 epochs at batch size 500 and three noise levels; simpler models can perform better as noise grows. I Values of ε at different δ shows that the same approximately 48%-accuracy run has a higher ε when δ is reduced.

Brief Thoughts

At ε = 10, the reported comparison shows that pretrained fine-tuning substantially exceeds training from scratch under the same batch size and epoch count. Interpretation is limited by non-private hyperparameter tuning, reliance on public pretraining data, and the remaining accuracy gap; Table 3 reports 70.8% for Resnet-18 and 75.3% for Resnet-50 at 90 epochs, so the abstract’s approximately 75% same-network comparison should be treated cautiously.