Downloads · 30 days
13
13% of all-time downloads
Qwen3-VL-8B-lora
samuellimabraz/Qwen3-VL-8B-lora
Qwen3-VL-8B-lora is a image-text-to-text model from samuellimabraz. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
All-time downloads
97
Public
Parameters
8.8B
17.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors17.5 GB · 100%
At a glance
- Task
- Image-Text-to-Text
- Library
- transformers
- License
- apache-2.0
- Model type
- qwen3_vl
- Access
- Public
- Created
- Dec 8, 2025
- Updated
- Aug 13, 2026
- SHA
- f31fb8dd
Try a prompt
Base models
Datasets
Model card
From the Hugging Face model README
Quantum Assistant: Specialization of Multimodal Models for Quantum Computing
<div align="center">
The first multimodal Vision-Language Model specialized for quantum computing with Qiskit
</div>Model Description
This model is a fine-tuned version of Qwen3-VL-8B-Instruct specialized for quantum computing tasks using Qiskit 2.0. This model can interpret visual representations of quantum computing: circuit diagrams, Bloch spheres, and measurement histograms.
The model was trained using vanilla LoRA (without rank stabilization) with rank 8 for 1 epoch on the Quantum Assistant Dataset. This model was part of the Phase 1 PEFT comparison experiments and was not evaluated on external benchmarks.
Key Capabilities
- Code Generation: Generate complete Qiskit code from natural language descriptions
- Function Completion: Complete function bodies from signatures and docstrings
- Visual Understanding: Interpret quantum circuit diagrams, Bloch spheres, and histograms
- Conceptual Explanations: Answer questions about quantum computing theory
- Qiskit 2.0 Compliant: Uses modern APIs (SamplerV2, EstimatorV2, generate_preset_pass_manager)
Training Evaluation
This model was part of the Phase 1 PEFT variant comparison experiments. It was not evaluated on external benchmarks (Qiskit HumanEval, Qiskit HumanEval Hard, or synthetic test set).
Internal Validation Metrics
| Metric | Value | Step |
|---|---|---|
| Eval Loss | 0.646 | 183 (final) |
| Eval Token Accuracy | 0.812 | 183 (final) |
| Train Loss | 0.628 | 183 (final) |
| Train Token Accuracy | 0.823 | 183 (final) |
| Training Runtime | 1,056 seconds (~17.6 min) | 1 epoch |
PEFT Comparison Analysis
Vanilla LoRA performed worse than rsLoRA (Eval Loss 0.646 vs 0.622) in the Phase 1 experiments:
- The lack of rank stabilization (α/√r scaling) led to slower convergence
- Final validation loss was 3.9% higher than rsLoRA
- Training time was similar (~1,056s vs 1,060s for rsLoRA)
Conclusion: rsLoRA's rank stabilization provides measurably better performance without additional computational cost, making vanilla LoRA suboptimal for this task.
Training Strategy
The experimental strategy was organized in two phases: PEFT technique selection and hyperparameter optimization.
Phase 1: PEFT Variant Comparison
Five LoRA variants were compared with controlled configuration (r=16, α=32, 1 epoch):
| Variant | Eval Loss ↓ | Eval Accuracy ↑ | Runtime (s) |
|---|---|---|---|
| rsLoRA | 0.622 | 0.818 | 1,060 |
| DoRA | 0.622 | 0.818 | 2,307 |
| rsLoRA (frozen aligner) | 0.623 | 0.817 | 1,057 |
| LoRA (vanilla) | 0.646 | 0.812 | 1,056 |
| PiSSA | 0.657 | 0.812 | 1,172 |
| OLoRA | 0.742 | 0.794 | 1,067 |
Key findings:
- rsLoRA and DoRA achieved equivalent performance (Eval Loss 0.622)
- DoRA has 2.18× computational overhead (2,307s vs 1,060s) due to magnitude-direction decomposition
- rsLoRA selected for optimal performance-efficiency trade-off
Phase 2: Rank and Epoch Optimization
With rsLoRA selected, the impact of adapter rank and training duration was investigated:
| Configuration | Eval Loss ↓ | Eval Accuracy ↑ | Notes |
|---|---|---|---|
| r=32, 1 epoch | 0.607 | 0.821 | Optimal trade-off |
| r=64, 1 epoch | 0.609 | 0.822 | Marginal improvement |
| r=16, 1 epoch | 0.622 | 0.818 | Baseline rsLoRA |
| r=32, 2 epochs | 0.638 | 0.825 | Slight overfitting |
| r=128, 3 epochs | 0.789 | 0.822 | Severe overfitting |
Conclusions: rsLoRA with r=32 and 1-2 epochs maximizes generalization while avoiding memorization of the synthetic dataset.
Model Collection
This model is part of the Quantum Assistant collection. All models are merged versions ready for inference:
| Model | Configuration | Description |
|---|---|---|
| Qwen3-VL-8B-rslora-r32-2 | rsLoRA r=32, 2 epochs | Best overall performance |
| Qwen3-VL-8B-rslora-r32 | rsLoRA r=32, 1 epoch | Best generalization |
| Qwen3-VL-8B-rslora-r64 | rsLoRA r=64, 1 epoch | Higher capacity |
| Qwen3-VL-8B-rslora-r128 | rsLoRA r=128, 1 epoch | Maximum capacity |
| Qwen3-VL-8B-lora | LoRA r=8, 1 epoch | Vanilla LoRA (baseline) |
| Qwen3-VL-8B-dora | DoRA r=16, 1 epoch | Magnitude-direction decomposition |
| Qwen3-VL-8B-pissa | PiSSA r=16, 1 epoch | SVD initialization |
| Qwen3-VL-8B-olora | OLoRA r=16, 1 epoch | QR orthonormal initialization |
| Qwen3-VL-8B-rslora-frozen | rsLoRA r=16, frozen aligner | Ablation study |
| Qwen3-VL-8B-rslora | rsLoRA r=16, 1 epoch | Baseline rsLoRA |
Usage
With vLLM
python -m vllm.entrypoints.openai.api_server \
--host 0.0.0.0 \
--port 8000 \
--model samuellimabraz/Qwen3-VL-8B-lora \
--gpu-memory-utilization 0.92 \
--max-model-len 12288 \
--max-num-seqs 16 \
--max-num-batched-tokens 49152 \
--enable-chunked-prefill \
--enable-prefix-caching
With Transformers
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
model = Qwen3VLForConditionalGeneration.from_pretrained(
"samuellimabraz/Qwen3-VL-8B-lora",
torch_dtype="auto",
device_map="auto"
)
processor = AutoProcessor.from_pretrained("samuellimabraz/Qwen3-VL-8B-lora")
messages = [
{"role": "system", "content": "You are a quantum computing expert assistant specializing in Qiskit."},
{"role": "user", "content": "Create a function that builds a 3-qubit GHZ state and returns the circuit."}
]
messages_with_image = [
{"role": "system", "content": "You are a quantum computing expert assistant specializing in Qiskit."},
{"role": "user", "content": [
{"type": "image", "image": "path/to/circuit.png"},
{"type": "text", "text": "Implement the quantum circuit shown in the image."}
]}
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt"
).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output = processor.batch_decode(
generated_ids[:, inputs.input_ids.shape[1]:],
skip_special_tokens=True
)[0]
print(output)
Training Details
Dataset
- Training Data: Quantum Assistant Dataset
- Train Samples: 5,837 (45.1% multimodal)
- Validation Samples: 1,239 (45.2% multimodal)
- Task Distribution: 30% function completion, 32% code generation, 38% QA
- Categories: 7 quantum computing domains
Training Configuration
| Parameter | Value |
|---|---|
| Base Model | Qwen/Qwen3-VL-8B-Instruct |
| PEFT Method | LoRA (vanilla, no rank stabilization) |
| Rank (r) | 8 |
| Alpha (α) | 32 |
| Dropout | 0.05 |
| Target Modules | all-linear |
| Learning Rate | 2e-4 |
| LR Scheduler | Cosine |
| Weight Decay | 0.01 |
| Warmup Steps | 10 |
| Epochs | 1 |
| Batch Size | 16 |
| Precision | bfloat16 |
| Framework | ms-swift |
Freezing Strategy
| Component | Status |
|---|---|
| Vision Encoder (ViT) | ❄️ Frozen |
| Vision-Language Aligner | 🔥 Trainable |
| Language Model (LLM) | 🔥 Trainable |
Training Infrastructure
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (96GB VRAM)
- Training Time: ~17.6 minutes (1 epoch)
- Tracking: Weights & Biases | TensorBoard
System Prompt
You are a quantum computing expert assistant specializing in Qiskit.
Provide accurate, clear, and well-structured responses about quantum computing concepts,
algorithms, and code implementation. Use Qiskit 2.0 best practices.
Intended Uses & Limitations
Intended Uses
- Educational assistance: Learning quantum computing concepts with Qiskit
- Code generation: Creating Qiskit circuits from descriptions or diagrams
- Documentation: Understanding quantum circuit visualizations
- Research prototyping: Rapid development of quantum algorithms
Limitations
- Domain specificity: Optimized for Qiskit 2.0; may generate deprecated APIs for older versions
- Dataset size: Trained on 5,837 samples; may underperform on rare edge cases
- Category imbalance: Better performance on
circuits_and_gatesthanprimitives_and_execution - Hardware specifics: Limited coverage of IBM Quantum hardware-specific optimizations
- Execution: Generated code requires verification before running on real quantum hardware
Bias and Risks
- Model may perpetuate patterns from training data
- Visual understanding limited to common diagram styles in Qiskit documentation
- May generate syntactically correct but logically incorrect quantum algorithms
- Should not be used for production quantum computing without human review
Citation
If you use this model in your research, please cite:
@article{braz2026quantumassistant,
title = {Quantum Assistant: Specialization of Multimodal Vision-Language Models for Quantum Computing},
author = {Braz, Samuel Lima and Leite, Jo{\~a}o Paulo Reus Rodrigues},
journal = {Expert Systems with Applications},
year = {2026},
issn = {0957-4174},
doi = {10.1016/j.eswa.2026.133931},
url = {https://doi.org/10.1016/j.eswa.2026.133931},
publisher = {Elsevier}
}
Related Resources
- Paper: doi:10.1016/j.eswa.2026.133931
- Dataset: samuellimabraz/quantum-assistant
- Model Collection: Quantum Assistant Models
- Demo: Quantum Assistant Space
- Code: GitHub Repository
Acknowledgments
- IBM Quantum and Qiskit team for open-source documentation
- Qwen Team for the base model
- UNIFEI (Universidade Federal de Itajubá) for academic support
- Advisor: Prof. João Paulo Reus Rodrigues Leite
License
This model is released under the Apache 2.0 License.
- Task
- Image-Text-to-Text
- Library
- transformers
- Type
- qwen3_vl
- License
- apache-2.0
- Languages
- en
- Created
- Dec 8, 2025
- Updated
- Aug 13, 2026