Gorio Tech Blog search

MART: Improving LLM Safety with Multi-round Automatic Red-Teaming | Summary

한국어로 읽기

13 Nov 2023 | Paper Review LLM Safety Adversarial Learning Self-Training
Contents

This article explains the key points of MART: Improving LLM Safety with Multi-round Automatic Red-Teaming.

  • 2023-11-13 (arXiv)
  • Ge, Suyu, Zhou, Chunting, Hou, Rui, Khabsa, Madian, Wang, Yi-Chia, Wang, Qifan, Han, Jiawei, Mao, Yuning.
  • GenAI, Meta, University of Illinois Urbana-Champaign
  • Paper

Read this article in Korean

Summary

  • Finding and correcting unsafe responses in large language models (LLMs) is costly when people must repeatedly write test prompts and responses. After four rounds of Multi-round Automatic Red-Teaming (MART), the model-based violation rate on the authors’ adversarial prompt set, SafeEval, fell by 84.7% relative to an instruction-tuned baseline with limited safety alignment.
  • MART trains an adversarial model to generate prompts that elicit unsafe responses and a target model on its own responses selected for safety and helpfulness. Human evaluation found a 53.7% relative reduction in SafeEval violations, though MART remained behind ChatGPT, GPT4, and Llama 2-Chat-70b on the reported safety measures.
  • Safety improved on SafeEval and the separate Anthropic Harmless dataset, while helpfulness scores changed relatively little. The method relies on reward models, which score response quality, and generated attacks became less effective as the target improved.

1 Introduction

Manual red-teaming, the practice of probing a model for harmful behavior, requires people to repeatedly write adversarial prompts and responses. MART couples automatic attack generation with safety fine-tuning so the attack model can adapt as the target changes.

2 Approach

Each round generates prompts from previous successful attacks, queries the target, and uses feedback to select successful attacks and safe, helpful responses. The selected examples update the adversarial and target models respectively.

The two branches show how evaluator feedback routes successful attacks to adversarial training and safe responses to target-model training.

Successful attacks train the adversarial model, while safe responses support target-model safety alignment.
Successful attacks train the adversarial model, while safe responses support target-model safety alignment.

2.1 Initialization

Both models start from LLaMA-65B fine-tuned on LIMA and Open Assistant instruction data. The authors curate approximately 2,400 red-teaming prompts without responses, described here as 1,700 training and 700 evaluation prompts, and use existing safety and helpfulness reward models for feedback.

2.2 Jailbreaking with Adversarial LLM

The adversarial model first learns to produce similar prompts from seed pairs sharing a violation category, the unsafe topic, and an attack style, the way the prompt is expressed. In later rounds, prompts that elicit low-safety responses are paired with their preceding input prompts to train the adversary, alongside general instruction data.

The examples show variation in generated prompts conditioned on examples from two named violation categories.

Examples of adversarial-model generations for two violation categories.
Examples of adversarial-model generations for two violation categories.

2.3 Feedback Guided Safety Finetuning

The target is fine-tuned on its own responses that exceed both safety and helpfulness thresholds. In the first iteration, context distillation generates additional candidates using a safety preprompt; in the last iteration, rejection sampling generates multiple answers per prompt and selects among those passing the thresholds.

A response with safety 0.91 and helpfulness 0.27 is excluded, while one scoring 0.91 and 0.85 is selected, illustrating why both criteria matter.

Examples of safety and helpfulness reward-model scores and response selection.
Examples of safety and helpfulness reward-model scores and response selection.

2.4 Iteratively Training Madv and Mtgt

The two models are updated in an iterative cycle: successful attacks guide the next adversarial generation, and selected responses train the target. This design aims to keep attacks relevant as target vulnerabilities change.

3 Experiment

The experiments assess safety, helpfulness, attack generation, and training-data selection across four iterations. Comparisons include the instruction-tuned Vanilla baseline, other chat models, and alternative attack-generation methods.

3.1 Experimental Setting

The authors use 2,852 cleaned Open Assistant samples and 1,000 LIMA samples, or 986 cleaned LIMA samples for the adversarial model. Most models use learning rate 1e-5 decaying to 9e-6, weight decay 0.1, batch size 8, and dropout 0.1; generation uses temperature 0.7 and nucleus sampling, which samples from a probability-limited token set, with p = 0.9.

  • The paper reports 752 SafeEval prompts and 480 non-adversarial HelpEval prompts, plus 2,312 Anthropic Harmless prompts and 805 AlpacaEval prompts. Its earlier approximate seed split states 700 evaluation prompts.
  • A safety reward-model score below 0.5 counts as a violation; human annotators also flag unsafe responses.

3.2 Helpfulness and Safety Performance

On SafeEval, Vanilla and MART have reward-model violation rates of 31.4% and 4.8%, and human rates of 17.2% and 8.0%, respectively. On Anthropic Harmless, the corresponding rates fall from 26.7% to 6.9% by reward model and from 12.1% to 4.9% by human evaluation.

  • The reward-model and human rankings agree, but the same reward models also guide training, creating a potential optimization bias.
  • HelpEval helpfulness at iteration 4 is reported as 3%–4% lower than at iteration 1; AlpacaEval scores change little.

Safety scores rise most visibly at lower percentiles on both safety sets, whereas helpfulness curves remain comparatively stable across iterations.

Safety and helpfulness reward-model score percentiles across iterations on in-domain and out-of-domain sets.
Safety and helpfulness reward-model score percentiles across iterations on in-domain and out-of-domain sets.

Both evaluation methods rank MART safer than Vanilla on the two datasets, but assign lower violation rates to the three comparison chat models.

Reward-model and human violation rates for MART and comparison models.
Reward-model and human violation rates for MART and comparison models.

3.3 Adversarial Performance for Red-teaming

MART-1shot uses one example attack to prompt the adversary and achieves the lowest final SafeEval violation rate among the tested methods. Greedy Coordinate Gradient (GCG), which uses greedy and gradient-based search for an adversarial suffix, triggers more unsafe responses initially but loses effectiveness after the first safety update.

  • The adversarial-generation violation rate falls from 29.74% at iteration 1 to 10.21% at iteration 4, making further collection of successful attacks difficult.

SafeEval and Anthropic Harmless violation rates decrease each round, while the rate on generated adversarial prompts first increases and then falls to 10.21%.

Violation rates across iterations on evaluation sets and generated attacks.
Violation rates across iterations on evaluation sets and generated attacks.

GCG variants lead in attack success in the first iteration, while MART-1shot sustains stronger attacks in intermediate rounds and achieves the lowest final SafeEval violation rate.

Attack-generation effectiveness and SafeEval violation rates for adversarial-generation methods.
Attack-generation effectiveness and SafeEval violation rates for adversarial-generation methods.

3.4 Impact of Safety Data Scaling

The authors vary the safety selection threshold from 0.4 to 0.9 and the helpfulness threshold from 0.0 to 0.6. They report the best tested combination as safety = 0.8 and helpfulness = 0.4; performance does not improve monotonically with the number of selected samples.

Increasing thresholds reduces the selected sample count substantially, yet plotted safety and helpfulness scores remain relatively stable and do not track sample count monotonically.

Selected sample counts and model scores under different safety and helpfulness thresholds.
Selected sample counts and model scores under different safety and helpfulness thresholds.

The paper situates MART within automatic adversarial attacks and iterative safety alignment. Its stated contribution is to use generated attacks to improve an evolving target model.

4.1 Adversarial Attack Towards LLM

Prior attack methods include prompt injection, which alters instructions to induce unwanted behavior, transferable adversarial prompts, and models trained to generate malicious inputs. MART also uses target feedback and incorporates generated prompts into repeated target-model safety training.

4.2 LLM Safety Alignment

Prior safety alignment uses supervised fine-tuning or reinforcement learning from human feedback and often relies on extensive human red-teaming. The paper cites 42k prompts for initial Claude training and more than 350 people involved in Llama 2-Chat red-teaming as examples of that effort.

5 Conclusion and Future Work

The authors propose reinforcement learning and multi-turn adversarial dialogue as future extensions. Their experiments focus on instruction fine-tuning and rejection sampling.

6 Ethical Statement

The ethical statement identifies dual-use concerns and calls for system design, dataset curation, human oversight, and continuous monitoring alongside safety improvements.

Appendix

  • The provided paper contains no appendix.

Brief Thoughts

Human evaluation complements reward-model evaluation, but both show MART behind the evaluated chat-model baselines. Generalization evidence is limited to the reported datasets and model setup, and declining attack success constrains further rounds.

Related Posts