openlm-3b-202407-retrieval-sft

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

openlm_3b_202407_retrieval_sft_full

This model is a full-parameter fine-tuned version of dogtooth/open-lm-3b-202407 on the mattwang123/chrononauts-retrieval-sft-202407-500 dataset.

Model description

OpenLM-3B (custom OpenLMForCausalLM; load with trust_remote_code=True) with a July 2024 (202407) knowledge cutoff, fine-tuned for retrieval-grounded question answering with explicit reasoning. Given numbered context documents and a question, it emits a <think>...</think> trace that cites documents by bracket number, followed by a concise answer.

Use the default chat template: Human: {prompt}<|endoftext|>\nAssistant:.

Intended uses & limitations

Proof-of-concept retrieval-SFT (500 examples) for the Chrononauts time-bounded-LM project. It reliably produces the <think> format and grounds simple answers in the provided context, but accuracy on hard multi-hop questions with distractors is limited by the small training set, and there is no abstention training (it does not reliably say when the context lacks the answer).

Training and evaluation data

Fine-tuned on mattwang123/chrononauts-retrieval-sft-202407-500: 300 HotpotQA (distractor), 100 SQuAD v1, 100 2WikiMultiHopQA, with gpt-5 teacher reasoning traces; all sources predate the 202407 cutoff.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• total_eval_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 5.0

Training results

Framework versions

• Transformers 4.57.1
• Pytorch 2.6.0+cu124
• Datasets 4.0.0
• Tokenizers 0.22.2