Chain of Thought Prompting

Chain of Thought (CoT) prompting (@wei2022chain) is a recently developed prompting method, which encourages the LLM to explain its reasoning. The below image(@wei2022chain) shows a %%few shot standard prompt|few shot standard prompt%% (left) compared to a chain of thought prompt (right).

Regular Prompting vs CoT (Wei et al.)

The main idea of CoT is that by showing the LLM some few shot %%exemplars|exemplars%% where the reasoning process is explained in the exemplars, the LLM will also show the reasoning process when answering the prompt. This explanation of reasoning often leads to more accurate results.

Example

Here are a few demos. The first shows GPT-3 (davinci-003) failing to solve a simple word problem. The second shows GPT-3 (davinci-003) succesfully solving the same problem, by using CoT prompting.

Incorrect

Correct

Results

CoT has been shown to be effective in improving results on tasks like arithmetic, commonsense, and symbolic reasoning tasks (@wei2022chain). In particular, prompted PaLM 540B(@chowdhery2022palm) achieves 57% solve rate accuracy on GSM8K(@cobbe2021training) (SOTA at the time).

Comparison of models on the GSM8K benchmark (Wei et al.)

Limitations

Importantly, according to Wei et al., "CoT only yields performance gains when used with models of ∼100B parameters". Smaller models wrote illogical chains of thought, which led to worse accuracy than standard prompting. Models usually get performance boosts from CoT prompting in a manner proportional to the size of the model.

Notes

No language models were ~~hurt~~ finetuned in the process of writing this chapter 😊.