litm-original-reproduction

Lost in the Middle — Original Paper Reproduction

Exact reproduction of the experiments from Liu et al. (2023) "Lost in the Middle: How Language Models Use Long Contexts" (arXiv:2307.03172).

Original Repo: https://github.com/nelson-liu/lost-in-the-middle

What's Included

All files fetched directly from the official GitHub repository via raw.githubusercontent.com.

Core Library

• src/prompting.py — Prompt construction (exact from src/lost_in_the_middle/prompting.py)
• src/metrics.py — Scoring/normalization (exact from src/lost_in_the_middle/metrics.py)

Prompt Templates

• prompts/qa.prompt
• prompts/kv_retrieval.prompt
• prompts/closedbook_qa.prompt
• prompts/qa_with_query_aware_contextualization.prompt
• prompts/kv_retrieval_with_query_aware_contextualization.prompt

Data Generation

• scripts/make_qa_data_from_retrieval_results.py — Multi-document QA data creation
• scripts/create_kv_pairs.py — KV retrieval synthetic data generation

Experiment Runners

• scripts/get_qa_responses_from_mpt.py — Multi-document QA runner (MPT-30B original)
• scripts/get_kv_responses_from_mpt.py — KV retrieval runner (MPT-30B original)

Mistral-7B Adapters

• scripts/run_qa_mistral.py — Multi-document QA on Mistral-7B-Instruct
• scripts/run_kv_mistral.py — KV retrieval on Mistral-7B-Instruct

Requirements

pip install transformers torch tqdm regex pydantic xopen

Quick Start: KV Retrieval on Mistral 7B

# 1. Generate synthetic KV data
python scripts/create_kv_pairs.py \
    --num-keys 100 \
    --num-examples 50 \
    --output data/kv_100.jsonl.gz

# 2. Run evaluation
python scripts/run_kv_mistral.py \
    --input-path data/kv_100.jsonl.gz \
    --model mistralai/Mistral-7B-Instruct-v0.3 \
    --gold-index 50 \
    --output results/kv_100_gold50.jsonl.gz

Paper

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023).
Lost in the Middle: How Language Models Use Long Contexts.
arXiv:2307.03172.