Contrastive Chain-of-Thought Prompting | Summary

This article explains the key points of Contrastive Chain-of-Thought Prompting.

• 2023-11-15 (arXiv)
• Chia, Yew Ken, Chen, Guizhen, Tuan, Luu Anh, Poria, Soujanya, Bing, Lidong.
• DAMO Academy, Alibaba Group, Singapore, Singapore University of Technology and Design, Nanyang Technological University, Singapore
• Paper
• Github

Read this article in Korean

Summary

• The paper addresses errors in language-model reasoning by adding incorrect reasoning examples to prompts that already contain correct examples. With GPT-3.5-Turbo (0301) and four demonstration examples, contrastive prompting improves scores over conventional chain-of-thought prompting by 9.8 points on GSM8K and 16.0 points on Bamboogle.
• Chain-of-thought prompting supplies intermediate reasoning steps in example solutions. The proposed method automatically creates incorrect examples by identifying and shuffling objects such as numbers and names in existing correct explanations.
• The method outperforms the conventional chain-of-thought baseline on all seven tested datasets, both with and without self-consistency, a decoding strategy that combines multiple reasoning paths. The evidence is limited to one model version, four-shot prompts, and at most 500 test examples per dataset.

1 Introduction

The authors motivate contrastive demonstrations by observing that mistakes in intermediate steps can compound, while earlier work found that invalid demonstrations sometimes perform similarly to valid ones (Wang et al., 2023). Their approach pairs positive and negative examples to show the model useful steps and faults to avoid.

The demonstration pairs a correct sequence of arithmetic operations with a wrong explanation that rearranges those operations. The subsequent model output computes 15 fixed teeth out of 30, yielding 50%.

2 Preliminary Study: Effect of Different Types of Contrastive Demonstrations

The preliminary study asks whether invalid reasoning demonstrations can improve chain-of-thought prompting when presented alongside valid rationales, or step-by-step explanations.

2.1 Components of Chain of Thought

The paper separates a rationale into bridging objects, the symbolic items traversed during reasoning, and language templates, the textual hints that derive and contextualize those objects. Numbers and equations are examples of objects in arithmetic, while entity names serve that role in factual questions.

2.2 What is Invalid Chain of Thought?

Invalid rationales are classified by coherence, the ordering of reasoning steps, and relevance, the correspondence of a rationale to the question. Applying these properties to objects and language yields four categories; a fifth, invalid reasoning, contains logical errors without necessarily being incoherent or irrelevant.

Arithmetic and factual examples show how invalid reasoning, incoherent objects or language, and irrelevant objects or language differ. The color coding distinguishes language, objects, and invalid components.

2.3 Experimental Setup

The preliminary experiments use GSM8K for arithmetic and Bamboogle for factual reasoning, GPT-3.5-Turbo (0301), and four manually constructed demonstrations per dataset. They compare five categories of invalid rationales in a standardized prompt format.

2.4 Preliminary Results

All five contrastive categories improve on conventional chain-of-thought in the preliminary comparison. Incoherent objects achieve the highest two-dataset average, 66.6 versus 55.0 for conventional chain-of-thought, with scores of 79.6 on GSM8K and 53.6 on Bamboogle.

3 Contrastive Chain of Thought

Each contrastive demonstration contains a question, correct reasoning and answer, and incorrect reasoning and answer. To generate the incorrect reasoning, an entity recognition model extracts object spans from a valid rationale and their positions are randomly shuffled; the model then answers a new question using the demonstrations.

For the illustrated dental problem, standard prompting produces 37.5%, conventional chain-of-thought produces 36.67%, and the contrastive example produces the correct 50%. This is an illustrative example rather than an aggregate comparison.

All five invalid-rationale categories raise the average above the conventional chain-of-thought score of 55.0; incoherent objects lead at 66.6.

4 Experiments

The main evaluation compares standard prompting, conventional chain-of-thought, and contrastive chain-of-thought, each also evaluated with self-consistency.

4.1 Experimental Setup

The arithmetic datasets are GSM8K, AQuA, GSM-Hard, SVAMP, and ASDIV; the factual question-answering datasets are Bamboogle and StrategyQA. Each uses four demonstrations, with 254 AQuA and 125 Bamboogle test examples and 500 randomly sampled examples for each other dataset.

4.2 Main Results

Without self-consistency, contrastive prompting scores 79.0 versus 69.2 on GSM8K and 56.8 versus 40.8 on Bamboogle; the other five gains range from 3.9 to 14.4 points as reported in Table 2. With self-consistency, it scores 86.2 versus 71.0 on GSM8K and 58.4 versus 40.8 on Bamboogle; on AQuA, the displayed scores are 71.7 versus 55.9.

Contrastive chain-of-thought exceeds its corresponding conventional baseline on all seven datasets in both decoding settings. The displayed AQuA values 57.5 and 53.5 differ by 4.0 points, although the table labels the gain +3.9.

All datasets use four demonstration examples; AQuA and Bamboogle have fewer than 500 test examples, while the other five have 500 each.

5 Related Work

The authors relate their work to prompting without additional model training, earlier chain-of-thought methods, and learning from negative examples. Their method uses positive and negative reasoning demonstrations in the prompt; the related approaches discussed include contrastive representation learning and reward modeling from human preferences.

6 Conclusions

The authors conclude that valid and invalid demonstrations improve reasoning in their experiments and that automatic construction removes the need to manually annotate invalid rationales.

Appendix

• The supplied paper has no appendix. Its code link is https://github.com/DAMO-NLP-SG/contrastive-cot.

Brief Thoughts

The comparison across five invalid-rationale categories supports the choice of shuffled objects, and the seven-dataset results show consistent gains under the reported setup. Generalization to other model versions, demonstration choices, and larger test sets remains untested in this paper; its longer contrastive prompts also make a matched prompt-length comparison useful.