quant-repair-llama-3

LoRAs for improving the quality of quantized Llama 3 models.

LoRAs available

<table> <tr> <th></th> <th colspan="3">WikiText</th> </tr> <tr> <th>Model</th> <th>PPL</th> <th>KL-Div</th> <th>Same top p</th> </tr> <tr> <td>Llama 3 70B Instruct (baseline)</td> <td style="text-align: end">5.282</td> <td style="text-align: end">0</td> <td style="text-align: end">100%</td> </tr> <tr> <td> <a href="https://huggingface.co/simsvml/quant-repair-llama-3/tree/main/70b_instruct_iq2xxs"> Llama 3 70B Instruct IQ2_XXS</a> <!-- ckpt20 --> </td> <td style="text-align: end">7.691</td> <td style="text-align: end">3.340 &times; 10^-1</td> <td style="text-align: end">79.3%</td> </tr> <tr> <td>Llama 3 70B Instruct IQ2_M (without LoRA)</td> <td style="text-align: end">7.765</td> <td style="text-align: end">3.430 &times; 10^-1</td> <td style="text-align: end">81.4%</td> </tr> <tr> <td>Llama 3 70B Instruct IQ2_XS (without LoRA)</td> <td style="text-align: end">9.320</td> <td style="text-align: end">5.502 &times; 10^-1</td> <td style="text-align: end">77.2%</td> </tr> <tr> <td>Llama 3 70B Instruct IQ2_XXS (without LoRA)</td> <td style="text-align: end">10.554</td> <td style="text-align: end">6.767 &times; 10^-1</td> <td style="text-align: end">73.8%</td> </tr> </table>

How to use this

Each subdirectory in this repo has a LoRA for a specific model and quant. Each LoRA works only with the exact quantized GGUF file it was trained on. See the README in each directory for details.

1. Choose a subdirectory and download its LoRA GGUF and the matching quantized model GGUF.
2. Download the quant-repair scripts: https://github.com/simsvml/quant-repair
3. Apply the LoRA to the model:

   cd quant-repair
   # Install dependencies if needed:
   pip3 install numpy
   python3 combine_gguf.py lora.gguf model.gguf -o out.gguf
   

   For combine_gguf.py, the LoRA GGUF must be the first input because the script copies metadata entries from the first input.
4. Build the llama-with-lora branch of llama.cpp: https://github.com/simsvml/llama.cpp/tree/llama-with-lora
5. Run llama.cpp as usual using the combined out.gguf file.

Training from a checkpoint

Some subdirectories contain LoRAs that haven't finished training. These have filenames like lora_ckpt17.gguf and come with a corresponding training checkpoint lora_ckpt17.pt. If you have 24GB VRAM, you can continue the training yourself:

1. Download the training checkpoint and quantized model GGUF.

2. Download the original model in safetensors format. This is used as a reference during training.

3. In the quant-repair repo, set up symlinks to the original and quantized models. Check the config file for the checkpoint to find the correct paths. The path given for orig_weights_safetensors_dir should be a symlink to the directory containing the original model's safetensors file, and the path given for quant_weights_gguf_path should be a symlink to the quantized GGUF.

   Alternatively, reconfigure the checkpoint with paths that are correct for your system. See the quant-repair README for instructions.

4. Install dependencies for the quant-repair training scripts:

   cd quant-repair
   pip3 install -r requirements.txt
   cd ../llama.cpp/gguf-py
   pip3 install .
   

5. Train:

   cd quant-repair
   python3 train_repair_lora2.py train lora.pt