Downloads · 30 days
0
RichardErkhov/NickyNicky_-_gemma-2b-it_chatML_distilabel-intel-orca-dpo-pairs_v1-exl2
NickyNicky_-_gemma-2b-it_chatML_distilabel-intel-orca-dpo-pairs_v1-exl2 is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
0
Access
Public
Updated Jan 18, 2025
Repo size
—
Likes
0
Public
Click a slice to open those files.
.json1 MB · 99%
From the Hugging Face model README
Quantization made by Richard Erkhov.
gemma-2b-it_chatML_distilabel-intel-orca-dpo-pairs_v1 - EXL2
| Branch | Bits | Description |
|---|---|---|
| 8_0 | 8.0 | Maximum quality that ExLlamaV2 can produce, near unquantized performance. |
| 6_5 | 6.5 | Very similar to 8.0, good tradeoff of size vs performance, recommended. |
| 5_0 | 5.0 | Slightly lower quality vs 6.5, but usable |
| 4_25 | 4.25 | GPTQ equivalent bits per weight, slightly higher quality. |
| 3_5 | 3.5 | Lower quality, only use if you have to. |
With git:
git clone --single-branch --branch 6_5 https://huggingface.co/NickyNicky_-_gemma-2b-it_chatML_distilabel-intel-orca-dpo-pairs_v1-exl2 gemma-2b-it_chatML_distilabel-intel-orca-dpo-pairs_v1-6_5
With huggingface hub:
pip3 install huggingface-hub
To download a specific branch, use the --revision parameter. For example, to download the 6.5 bpw branch:
Linux:
huggingface-cli download NickyNicky_-_gemma-2b-it_chatML_distilabel-intel-orca-dpo-pairs_v1-exl2 --revision 6_5 --local-dir gemma-2b-it_chatML_distilabel-intel-orca-dpo-pairs_v1-6_5 --local-dir-use-symlinks False
Windows (which apparently doesn't like _ in folders sometimes?):
huggingface-cli download NickyNicky_-_gemma-2b-it_chatML_distilabel-intel-orca-dpo-pairs_v1-exl2 --revision 6_5 --local-dir gemma-2b-it_chatML_distilabel-intel-orca-dpo-pairs_v1-6.5 --local-dir-use-symlinks False
license: apache-2.0 datasets:
library_name: transformers
widget:

reference data model:
datasets:
link: https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs
link dataset format gemma chatml: https://huggingface.co/datasets/NickyNicky/distilabel-intel-orca-dpo-pairs_gemma_chatml
model:
- google/gemma-2b-it
Link base:
https://huggingface.co/google/gemma-2b-it
Link fine-tune:
https://huggingface.co/NickyNicky/gemma-2b-it_oasst2_chatML_Cluster_2_V1
Epoch: 1
Future experts: 4
Eval model:
- link:
soon
!python -m pip install --upgrade pip
!pip install "torch>=2.1.1" -U
# !pip install torchaudio==2.2.0
!pip install -q datasets trl peft bitsandbytes sentencepiece wandb
!pip install -q accelerate safetensors deepspeed
!pip install -q scipy ninja -U
!pip install -q -U transformers==4.38.0
import torch
torch.__version__
#OUTPUTS: ('2.2.0+cu121' )
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
HfArgumentParser,
TrainingArguments,
pipeline,
logging,
GenerationConfig,
TextIteratorStreamer,
)
from transformers import StoppingCriteria, StoppingCriteriaList
import torch
model_id='NickyNicky/gemma-2b-it_chatML_distilabel-intel-orca-dpo-pairs_v1'
model = AutoModelForCausalLM.from_pretrained(model_id,
device_map="auto",
trust_remote_code=True,
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
# load_in_4bit=True,
# low_cpu_mem_usage= True,
)
max_length=2155
print("max_length",max_length)
tokenizer = AutoTokenizer.from_pretrained(model_id,
# use_fast = False,
max_length=max_length,)
class ListOfTokensStoppingCriteria(StoppingCriteria):
"""
Clase para definir un criterio de parada basado en una lista de tokens específicos.
"""
def __init__(self, tokenizer, stop_tokens):
self.tokenizer = tokenizer
# Codifica cada token de parada y guarda sus IDs en una lista
self.stop_token_ids_list = [tokenizer.encode(stop_token, add_special_tokens=False) for stop_token in stop_tokens]
def __call__(self, input_ids, scores, **kwargs):
# Verifica si los últimos tokens generados coinciden con alguno de los conjuntos de tokens de parada
for stop_token_ids in self.stop_token_ids_list:
len_stop_tokens = len(stop_token_ids)
if len(input_ids[0]) >= len_stop_tokens:
if input_ids[0, -len_stop_tokens:].tolist() == stop_token_ids:
return True
return False
# Uso del criterio de parada personalizado
stop_tokens = ["<end_of_turn>"] # Lista de tokens de parada
# Inicializa tu criterio de parada con el tokenizer y la lista de tokens de parada
stopping_criteria = ListOfTokensStoppingCriteria(tokenizer, stop_tokens)
# Añade tu criterio de parada a una StoppingCriteriaList
stopping_criteria_list = StoppingCriteriaList([stopping_criteria])
#EXAMPLE #1
txt="""<bos><start_of_turn>system
You are a helpful AI assistant.<end_of_turn>
<start_of_turn>user
Me dices los diferentes tipos de reciclaje que suelen existir en las ciudades europeas<end_of_turn>
<start_of_turn>model
"""
#EXAMPLE #2
txt="""<bos><start_of_turn>system
You are a helpful AI assistant.<end_of_turn>
<start_of_turn>user
What is the meaning of life in the current time?<end_of_turn>
<start_of_turn>model
"""
inputs = tokenizer.encode(txt,
return_tensors="pt",
add_special_tokens=False).to("cuda:0")
max_new_tokens=1000
generation_config = GenerationConfig(
max_new_tokens=max_new_tokens,
temperature=0.1, # .82 .2
#top_p=0.9,
top_k=50,
repetition_penalty=1.1,
do_sample=True,
)
outputs = model.generate(generation_config=generation_config,
input_ids=inputs,
stopping_criteria=stopping_criteria_list,)
tokenizer.decode(outputs[0], skip_special_tokens=False) #True