gpt2-implicit-cot-multiplication-20-digits

GPT-2 multiplication with implicit chain of thought

This GPT-2 checkpoint predicts products without generating written intermediate calculation steps. It uses stepwise internalization from From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step: explicit reasoning steps are gradually removed during fine-tuning. At inference, the answer digits are still generated autoregressively.

• Interactive demo: compare this checkpoint with explicit-CoT and no-CoT models.
• Training code and data-format notes
• Paper record and citation downloads
• Implicit chain-of-thought reading list · Internalized reasoning

Input and output

Reverse each number's decimal digits and separate the digits with spaces. Use * between the numbers and = at the end. For example, 12 × 34 becomes:

2 1 * 4 3 =

The generated product is also digit-reversed and space-separated: 8 0 4 represents 408. Remove spaces and reverse the generated digits to read the answer. Do not pass ordinary prose or a chat prompt to this checkpoint.

The demo accepts two non-negative whole numbers of up to 20 digits each. This is an input limit, not a guarantee of correctness. The authors' repository describes mixed digit-length training for this demo, unlike the separate fixed-length experiments in the paper. The demo also uses = as its input separator and omits the #### answer marker used in the research code.

Example

Install PyTorch and Transformers. This example follows the demo's prompt format and greedy decoding.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "yuntian-deng/gpt2-implicit-cot-multiplication-20-digits"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).to(device).eval()

def digits(number):
    return " ".join(str(number)[::-1])

prompt = f"{digits(12)} * {digits(34)} ="
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )
generated = outputs[0, inputs["input_ids"].shape[-1]:]
raw_answer = tokenizer.decode(generated, skip_special_tokens=True)
prediction = "".join(raw_answer.split())[::-1]
print(prediction)

Limitations

This is a research demonstration, not a reliable calculator or a general-purpose assistant. Predictions can be wrong, including for inputs within the demo's accepted range. Check answers with exact arithmetic. Individual examples do not establish benchmark accuracy or generalization to longer numbers. The original paper's fixed-length results should not be treated as an evaluation of this mixed-length demo checkpoint.

Citation

The method used by this checkpoint is stepwise internalization:

@misc{deng2024internalize,
  title = {{From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step}},
  author = {Yuntian Deng and Yejin Choi and Stuart Shieber},
  year = {2024},
  eprint = {2405.14838},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2405.14838}
}

Related earlier work, Implicit Chain of Thought Reasoning via Knowledge Distillation, studies a different training approach based on hidden-state distillation. See its paper record for citation downloads.

@misc{deng2023implicit,
  title = {{Implicit Chain of Thought Reasoning via Knowledge Distillation}},
  author = {Yuntian Deng and Kiran Prasad and Roland Fernandez and Paul Smolensky and Vishrav Chaudhary and Stuart Shieber},
  year = {2023},
  eprint = {2311.01460},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2311.01460}
}