llava-7b-sft

LLaVA 7B — Supervised Fine-Tuning (SFT) on Synthetic QA

Model type: Vision-Language Causal Model (text-finetuned LLaVA-1.5)
Base model: llava-hf/llava-1.5-7b-hf
License: Llama 2 Community License
Framework: Axolotl + DeepSpeed ZeRO-1 (PyTorch 2.5.1 + CUDA 12.1)

---

Overview

llava-7b-sft is a supervised fine-tuned version of LLaVA 1.5 7B, trained on a synthetic instruction-following dataset of question–answer pairs to enhance text understanding and reasoning.
Although derived from a multimodal base, this SFT run fine-tunes the language model component using LoRA adapters which were later merged into the full model weights.

This model therefore supports text-only generation natively (without PEFT) and retains compatibility with the multimodal processor and vision configuration from LLaVA.

Training was conducted on the Leonardo EuroHPC system using Axolotl and DeepSpeed ZeRO-1.

---

Training Setup

---

Dataset

Name: axolotl_deduplicated_synthetic_qa.jsonl
Type: Instruction-following synthetic QA dataset (Alpaca-style)

Each record contains a single-turn question and a high-quality generated answer.
This SFT data improves the model’s reasoning, language coherence, and conversational QA quality.

---

Hyperparameters

---

Tokenizer & Processor

The processor configuration allows image or text inputs; however, this release focuses on text-based supervised tuning.

---

Files Included

This repository contains the fully merged model weights and all required configs for direct use with transformers:

• config.json
• model-*.safetensors
• tokenizer.json
• tokenizer_config.json
• tokenizer.model
• special_tokens_map.json
• processor_config.json
• preprocessor_config.json
• vision_config.json
• image_processor_config.json
• README.md

---

Usage Example

To run text-based generation with this model:

import torch
from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "ubitech-edg/llava-7b-sft"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = "USER: Explain the principle of energy conservation.\nASSISTANT:"
inputs = processor(text=prompt, return_tensors="pt").to("cuda")

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9)

print(processor.decode(outputs[0], skip_special_tokens=True))