airoboros-2.2.1-y34b-3.0bpw-h6-exl2

airoboros-2.2.1-y34b

Unofficial training of Jon Durbin's powerful airoboros 2.2.1 dataset on Charles Goddard's Llama-fied Yi 34B model, aiming to bring the instruction-following capabilities of the airoboros dataset to the new Yi 34B foundational model.

As a 34B model with grouped-query attention, users will be able to conduct inference on the model with 4bit quantization on a single 24gb consumer GPU.

This Yi model is "Llama-fied", meaning the keys are renamed to match those used in Llama models, eliminating the need for remote code and ensuring compatibility with existing training and inference repositories. Architecturally this is similar to a Llama 2 34B model with an expanded vocab size of 64000.

This model is retrained thanks to compute provided by alpin with a monkeypatch to the trainer to resolve EOS token issues in the prompter. A smaller batch size and learning rate were used and training was extended by one epoch. 8-bit lora was also used instead of qlora.

Usage:

The intended prompt format is the modified Vicuna 1.1 instruction format used by airoboros v2:

A chat.
USER: {prompt}
ASSISTANT:

Training Details:

The model was trained using axolotl as a lora adapter on 1x A100 80gb GPU for 4 epochs, before being fused to the base model with PEFT.

License:

This model is built on the Yi 34B base model, which has its own custom license included in this repository.

Please refer to the airoboros 2.2.1 dataset card regarding the usage of gpt-4 API calls in creating the dataset.