Downloads · 30 days
1
6% of all-time downloads
Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch
unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch is a machine learning model from Petermoyano. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft.
This is a LoRA (Low-Rank Adaptation) fine-tuned model based on the unsloth/gemma-7b-bnb-4bit base model. It has been adapted for a tipification analysis task similar to the Llama-3.2-3B-Instruct LoRA fine-tuning, wher…
Downloads · 30 days
1
6% of all-time downloads
All-time downloads
16
Public
Repo size
239 MB
Likes
0
Public
Click a slice to open those files.
.safetensors200 MB · 84%
From the Hugging Face model README
This is a LoRA (Low-Rank Adaptation) fine-tuned model based on the unsloth/gemma-7b-bnb-4bit base model. It has been adapted for a tipification analysis task similar to the Llama-3.2-3B-Instruct LoRA fine-tuning, where the model classifies text into categories such as "ESTAFA," "ROBO," "HURTO," and their "TENTATIVA DE" variations.
During fine-tuning, only specific adapter layers were trained (~50 million parameters), while the rest of the base model was frozen. This approach allows parameter-efficient training, significantly reducing computational costs.
unsloth/gemma-7b-bnb-4bitCAUSAL_LM)r: 16lora_alpha: 16lora_dropout: 0.0gate_proj, up_proj, down_proj, k_proj, q_proj, o_proj, v_projThis model was fine-tuned on the same dataset as the Llama-3.2-3B-Instruct LoRA version, with the following category distribution:
| Category | Count | Percentage |
|---|---|---|
| ESTAFA | 4610 | 47.3% |
| ROBO | 2307 | 23.7% |
| HURTO | 2141 | 22.0% |
| TENTATIVA DE ESTAFA | 306 | 3.1% |
| TENTATIVA DE ROBO | 272 | 2.8% |
| TENTATIVA DE HURTO | 113 | 1.2% |
| Total | 9749 | 100% |
Although the dataset has nearly 10K examples in this summary table, the fine-tuning run used an extended version (~15K examples) for this particular training session.
Below is a snapshot of how training and validation loss evolved during the single epoch (117 steps):
| Step | Training Loss | Validation Loss |
|---|---|---|
| 10 | 2.974900 | 4.242294 |
| 20 | 5.451000 | 4.526450 |
| 30 | 4.150400 | 3.632928 |
| 40 | 3.036100 | 2.615031 |
| 50 | 2.492900 | 2.178700 |
| 60 | 2.095400 | 1.886430 |
| 70 | 2.099200 | 1.548187 |
| 80 | 1.983100 | 2.104600 |
| 90 | 2.020900 | 1.526225 |
| 100 | 1.727700 | 1.699223 |
| 110 | 1.868300 | 1.716561 |
| ... | ... | ... |
Final training concluded at step 117.
We observe a steady decrease in both training and validation losses, indicating the model was converging throughout the single epoch.
You can use this LoRA fine-tuned model with the Hugging Face Transformers library. Below is an example of how to load and run the model for text generation or classification-like tasks:
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch")
model = AutoModelForCausalLM.from_pretrained("Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch")
input_text = "TENTATIVA DE ESTAFA:"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))