Explain Chain-of-Thought Prompting and When to Use It

Why This Is Asked

Chain-of-thought (CoT) is one of the most impactful prompting techniques. Interviewers ask this to verify you understand not just what it is but when to apply it — and critically, when it's unnecessary or counterproductive.

Key Concepts to Cover

• What CoT is — instructing the model to reason step-by-step before answering
• Zero-shot CoT — "Let's think step by step" appended to the prompt
• Few-shot CoT — examples of reasoning chains included in the prompt
• Why it works — decomposing complex problems reduces errors
• When to use it — multi-step reasoning, math, logic, planning
• When NOT to use it — simple retrieval, classification, latency-sensitive tasks

How to Approach This

1. What Is Chain-of-Thought?

Standard prompt:

"If Alice has 5 apples and gives 2 to Bob, how many does she have?" → "3"

CoT prompt:

"...Let's think step by step." → "Alice starts with 5. She gives 2 to Bob. 5 - 2 = 3. Alice has 3 apples."

The intermediate reasoning improves accuracy on complex tasks.

2. Zero-Shot vs. Few-Shot CoT

Zero-shot CoT: Add "Let's think step by step" to your prompt. Works well with capable frontier models — with current models (Claude, GPT-4o, Gemini), simply asking for reasoning often performs comparably to few-shot CoT for standard tasks.

Few-shot CoT: Provide 2-5 complete reasoning chain examples. More reliable for highly domain-specific patterns where the format or reasoning style needs to be precisely controlled, or when working with smaller/older models where zero-shot CoT is less reliable.

3. When to Use CoT

• Multi-step arithmetic or logic
• Planning a sequence of actions
• Debugging or root cause analysis
• Synthesizing multiple facts

4. When NOT to Use CoT

• Simple lookup/retrieval tasks
• Classification with clear categories
• Latency-sensitive applications
• Structured output tasks (CoT can interfere with clean JSON output)

5. Extracting the Final Answer

Add explicit delimiters: "After your reasoning, output the final answer on its own line after '### Answer:'"

Common Follow-ups

1. "What is self-consistency and how does it relate to CoT?" Generate multiple reasoning chains and take a majority vote. Improves accuracy at higher token cost.

2. "Does CoT always help? Can it hurt?" Yes, it can hurt. On simple factual tasks, CoT can introduce hallucinated reasoning steps leading to wrong conclusions.

3. "How does CoT relate to dedicated reasoning models?" Dedicated reasoning models (like OpenAI o1/o3 or Claude's extended thinking) generate an explicit chain-of-thought scratchpad as a sequence of tokens before producing their final answer — the reasoning is a real generated token sequence, not a hidden internal process. This built-in reasoning typically outperforms manually prompted CoT on complex tasks. The tradeoff: reasoning models cost more, have higher latency, and the scratchpad may be partially hidden in some UIs. For most tasks, start with a capable frontier model + CoT prompting; upgrade to a dedicated reasoning model when the task genuinely requires multi-step search or complex planning.