rho-math-1b-v0.1-GGUF

QuantFactory/rho-math-1b-v0.1-GGUF

This is quantized version of microsoft/rho-math-1b-v0.1 created using llama.cpp

Model Description

<h1 align="center"> Rho-1: Not All Tokens Are What You Need </h1> <p align="center"> <a href="https://arxiv.org/abs/2404.07965"><b>[📜 Arxiv]</b></a> • <a href="https://huggingface.co/papers/2404.07965"><b>[💬 HF Paper]</b></a> • <a href="https://huggingface.co/microsoft/rho-math-1b-v0.1"><b>[🤗 Models]</b></a> • <a href="https://github.com/microsoft/rho"><b>[🐱 GitHub]</b></a> </p> <p align="center"> <img src="https://github.com/microsoft/rho/blob/main/docs/static/images/acc_vs_tokens_1b_7b.png?raw=true" width="1000"> <br> <em>Figure 1: Rho-1 is pre-trained with Selective Language Modeling (SLM). SLM improves average few-shot accuracy on GSM8k and MATH by over 16%, achieving the baseline performance 5-10x faster.</em> </p>

🔥 News

• [2024/04/12] 🔥🔥🔥 Rho-Math-v0.1 models released at 🤗 HuggingFace!
  • Rho-Math-1B and Rho-Math-7B achieve 15.6% and 31.0% few-shot accuracy on MATH dataset, respectively — matching DeepSeekMath with only 3% of the pretraining tokens.
  • Rho-Math-1B-Interpreter is the first 1B LLM that achieves over 40% accuracy on MATH.
  • Rho-Math-7B-Interpreter achieves 52% on MATH dataset, using only 69k samples for fine-tuning.
• [2024/04/11] Rho-1 paper and repo released.

💡 Introduction

Rho-1 base models employ Selective Language Modeling (SLM) for pretraining, which selectively trains on clean and useful tokens that aligned with the desired distribution.

Selective Lanugage Modeling (SLM)

<p align="center"> <img src="https://github.com/microsoft/rho/blob/main/docs/static/images/example.png?raw=true" width="1000"> <br> <em>Figure 2: <b>Upper:</b> Even an extensively filtered pretraining corpus contains token-level noise. <b>Left:</b> Previous Causal Language Modeling (CLM) trains on all tokens. <b>Right:</b> Our proposed Selective Language Modeling (SLM) selectively applies loss on those useful and clean tokens.</em> </p> <p align="center"> <img src="https://github.com/microsoft/rho/blob/main/docs/static/images/pipeline.png?raw=true" width="1000"> <br> <em>Figure 3: <b>The pipeline of Selective Language Modeling.</b> SLM optimizes language model performance by concentrating on valuable, clean tokens during pre-training. It involves three steps: (Step 1) Initially, train a reference model on high-quality data. (Step 2) Then, score each token's loss in a corpus using the reference model. (Step 3) Finally, train the language model selectively on tokens that show higher excess loss compared to the reference loss.</em> </p> <!-- results: -->

Evaluation Results

Base models (Few-shot CoT):

Tool-integrated reasoning (Code Interpreter):

🚀 Quick Start

Evaluation

git clone [email protected]:microsoft/rho.git
cd rho-1/math-evaluation-harness

Base model few-shot evaluation:

bash scripts/run_eval.sh cot microsoft/rho-math-7b-v0.1

SFT model (code-interpreter) evaluation:

bash scripts/run_eval.sh tora microsoft/rho-math-7b-interpreter-v0.1

Our reproduced outputs are provided in rho-1/outputs.zip.