Paper link:

https://arxiv.org/pdf/2404.15758

1. Motivation
2. Related work
3. Methodology
   1. Tasks created
   2. Experiments
4. Results
   1. 3SUM
   2. 2SUM-Transform
   3. Additional Insight
   4. Food for thought
5. Summary

1. Motivation

How do Large Language Models (LLMs) think?

Chain-of-thought (CoT) reasoning [1, 2, 3] shows that by allowing LLMs to think out loud before answering, their performance actually improves considerably when compared to direct answering techniques without CoT.

This provides some intuition as to how LLMs reason through their tasks.

Recent work [3, 4] suggests that for CoT reasoning, LLMs answers could be unfaithful to the intermediate reasoning steps (simply put, the LLM answers do not tally with their “workings”).

Some guiding questions:

• Do LLMs need to think out loud or are they able to think “internally” like humans?
• Are there specific tasks to show that LLMs are not simply relying on the semantic information found in the additional tokens?