llama_vlsi_finetuned

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

llama_vlsi_finetuned

This model is a fine-tuned version of unsloth/llama-3.2-3b-unsloth-bnb-4bit on a extracted VLSI dataset. It achieves the following results on the evaluation set:

• Loss: 4.9901

Model description

As it is prototype version of the main projecvt, we have limited the data extraction to fixed number of research papers, PDFs and YouTube videos. The data pipeline then seamlessly integrates alll the extracted data into a single strutctured dataset of json format. Using Hugging face , we have imported Llama 3.2 - 3B and fine-tuned the model using PEFT (LoRA adapters). The main goal of this prototype is reduce the Loss value of the fine-tuned model to minimal value like 4.9901. Unsloth fastens the fine-tuning process.

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 10
• mixed_precision_training: Native AMP

Training results

Framework versions

• PEFT 0.14.0
• Transformers 4.50.3
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1