Downloads · 30 days
55
2% of all-time downloads
RichardErkhov/nie3e_-_Qra-1b-dolly-instruction-0.1-gguf
nie3e_-_Qra-1b-dolly-instruction-0.1-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
55
2% of all-time downloads
All-time downloads
2.4K
Public
Repo size
11.2 GB
Likes
1
Public
Click a slice to open those files.
.gguf13.2 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
Qra-1b-dolly-instruction-0.1 - GGUF
| Name | Quant method | Size |
|---|---|---|
| Qra-1b-dolly-instruction-0.1.Q2_K.gguf | Q2_K | 0.4GB |
| Qra-1b-dolly-instruction-0.1.Q3_K_S.gguf | Q3_K_S | 0.47GB |
| Qra-1b-dolly-instruction-0.1.Q3_K.gguf | Q3_K | 0.51GB |
| Qra-1b-dolly-instruction-0.1.Q3_K_M.gguf | Q3_K_M | 0.51GB |
| Qra-1b-dolly-instruction-0.1.Q3_K_L.gguf | Q3_K_L | 0.55GB |
| Qra-1b-dolly-instruction-0.1.IQ4_XS.gguf | IQ4_XS | 0.57GB |
| Qra-1b-dolly-instruction-0.1.Q4_0.gguf | Q4_0 | 0.59GB |
| Qra-1b-dolly-instruction-0.1.IQ4_NL.gguf | IQ4_NL | 0.6GB |
| Qra-1b-dolly-instruction-0.1.Q4_K_S.gguf | Q4_K_S | 0.6GB |
| Qra-1b-dolly-instruction-0.1.Q4_K.gguf | Q4_K | 0.62GB |
| Qra-1b-dolly-instruction-0.1.Q4_K_M.gguf | Q4_K_M | 0.62GB |
| Qra-1b-dolly-instruction-0.1.Q4_1.gguf | Q4_1 | 0.65GB |
| Qra-1b-dolly-instruction-0.1.Q5_0.gguf | Q5_0 | 0.71GB |
| Qra-1b-dolly-instruction-0.1.Q5_K_S.gguf | Q5_K_S | 0.71GB |
| Qra-1b-dolly-instruction-0.1.Q5_K.gguf | Q5_K | 0.73GB |
| Qra-1b-dolly-instruction-0.1.Q5_K_M.gguf | Q5_K_M | 0.73GB |
| Qra-1b-dolly-instruction-0.1.Q5_1.gguf | Q5_1 | 0.77GB |
| Qra-1b-dolly-instruction-0.1.Q6_K.gguf | Q6_K | 0.84GB |
| Qra-1b-dolly-instruction-0.1.Q8_0.gguf | Q8_0 | 1.09GB |
tags:
This model if a fine-tuned version of OPI-PG/Qra-1b on the s3nh/alpaca-dolly-instruction-only-polish dataset.
Trained from OPI-PG/Qra-1b
This model has been fine-tuned for question-answering task. It is possible to use it as a chat, but it doesn't work well because the dataset did not contain conversations.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
model_id = "nie3e/Qra-1b-dolly-instruction-0.1"
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
pipe = pipeline(
"text-generation", model=model, tokenizer=tokenizer, device=device
)
def get_answer(system_prompt: str, user_prompt: str) -> str:
input_msg = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt}
]
prompt = pipe.tokenizer.apply_chat_template(
input_msg, tokenize=False,
add_generation_prompt=True
)
outputs = pipe(
prompt, max_new_tokens=512, do_sample=False, temperature=0.1, top_k=50,
top_p=0.1, eos_token_id=pipe.tokenizer.eos_token_id,
pad_token_id=pipe.tokenizer.pad_token_id
)
return outputs[0]['generated_text'][len(prompt):].strip()
print(
get_answer(
system_prompt="Jesteś przyjaznym chatbotem",
user_prompt="Napisz czym jest dokument architectural decision record."
)
)
Dataset: s3nh/alpaca-dolly-instruction-only-polish
Each row has been converted into conversation using this function:
system_message = """Jesteś przyjaznym chatbotem"""
def create_conversation(sample) -> dict:
strip_characters = "\"'"
return {
"messages": [
{"role": "system", "content": system_message},
{"role": "user",
"content": f"{sample['instruction'].strip(strip_characters)} "
f"{sample['input'].strip(strip_characters)}"},
{"role": "assistant",
"content": f"{sample['output'].strip(strip_characters)}"}
]
}
Train/test split: 90%/10%
GPU: 2x RTX 4060Ti 16GB Training time: ~1 hour
Using accelerate + deepspeed with config:
compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
gradient_accumulation_steps: 2
zero3_init_flag: false
zero_stage: 1
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
Lora config:
peft_config = LoraConfig(
lora_alpha=128,
lora_dropout=0.05,
r=256,
bias="none",
target_modules="all-linear",
task_type="CAUSAL_LM"
)
Training arguments:
args = TrainingArguments(
output_dir="Qra-1b-dolly-instruction-0.1",
num_train_epochs=3,
per_device_train_batch_size=3,
gradient_accumulation_steps=2,
gradient_checkpointing=True,
optim="adamw_torch_fused",
logging_steps=10,
save_strategy="epoch",
learning_rate=2e-4,
bf16=True,
tf32=True,
max_grad_norm=0.3,
warmup_ratio=0.03,
lr_scheduler_type="constant",
push_to_hub=False,
report_to=["tensorboard"],
)