solidity-codellama-lora-r64-final

CodeLlama-7B Solidity — LoRA Adapter (final, 5 epochs)

This repository contains a LoRA / PEFT adapter (not a full model) fine-tuned on top of AlfredPros/CodeLlama-7b-Instruct-Solidity.

This is the final checkpoint of the run, saved at global step 1,485,045 — the full 5 epochs (max_steps reached, should_training_stop = true). Training is complete; this is the adapter to use for inference. Optimizer / scheduler / RNG state are also included if you wish to continue training beyond 5 epochs.

Model Details

Final training metrics

Files in this repository

Note: The ~13 GB base model weights are not included here — they are pulled separately from AlfredPros/CodeLlama-7b-Instruct-Solidity. The training dataset and training script are also not included.

How to use (inference)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "AlfredPros/CodeLlama-7b-Instruct-Solidity"
ADAPTER = "Mukesh0606/solidity-codellama-lora-r64-final"   # or a local path to this folder

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
base_model = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto")
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

prompt = "// Write a secure ERC20 token contract in Solidity\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Merge the adapter into a standalone model (optional)

merged = model.merge_and_unload()   # folds LoRA weights into the base
merged.save_pretrained("codellama-7b-solidity-merged")
tokenizer.save_pretrained("codellama-7b-solidity-merged")

Continue training beyond 5 epochs (optional)

This run finished at its planned max_steps. To train further you would raise num_train_epochs / max_steps in your TrainingArguments and resume:

from transformers import Trainer

# Re-create the SAME base model, LoRA config, tokenizer, dataset, and updated TrainingArguments.
trainer = Trainer(model=peft_model, args=training_args, train_dataset=..., ...)
trainer.train(resume_from_checkpoint="path/to/checkpoint-1485045")

Requirements:

• Same base model (AlfredPros/CodeLlama-7b-Instruct-Solidity).
• Same LoRA configuration (already in adapter_config.json).
• The original training dataset and preprocessing (not stored here).
• A GPU with enough memory for 7B + LoRA training (original run used train_batch_size=1).

If you load the adapter manually via PeftModel.from_pretrained for further training rather than via resume_from_checkpoint, pass is_trainable=True. adapter_config.json has inference_mode: true, which is correct for inference; the Trainer resume path switches back to training mode automatically.

Hardware notes

• Inference: 7B runs on ~16 GB GPU in fp16, or ~6 GB with 4-bit quantization (bitsandbytes).
• Training/resume: needs a capable GPU; original run used batch size 1.

Framework versions

• PEFT 0.14.0
• Transformers (Hugging Face Trainer)
• Base: CodeLlama-7B-Instruct (Solidity-tuned)

License

Inherits the base model's license (Llama 2 Community License via CodeLlama). Review the base model card before commercial use.