Downloads · 30 days
34
26% of all-time downloads
Qwen3-VL-8B-olora
samuellimabraz/Qwen3-VL-8B-olora
Qwen3-VL-8B-olora is a image-text-to-text model from samuellimabraz. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
All-time downloads
131
Public
Parameters
8.8B
17.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors17.5 GB · 100%
At a glance
- Task
- Image-Text-to-Text
- Library
- transformers
- License
- apache-2.0
- Model type
- qwen3_vl
- Access
- Public
- Created
- Dec 8, 2025
- Updated
- Aug 13, 2026
- SHA
- 6d6cc63d
Try a prompt
Base models
Datasets
Model card
From the Hugging Face model README
Quantum Assistant: Specialization of Multimodal Models for Quantum Computing
<div align="center">
The first multimodal Vision-Language Model specialized for quantum computing with Qiskit
</div>Model Description
This model is a fine-tuned version of Qwen3-VL-8B-Instruct specialized for quantum computing tasks using Qiskit 2.0. This model can interpret visual representations of quantum computing: circuit diagrams, Bloch spheres, and measurement histograms.
The model was trained using OLoRA (Orthonormal Low-Rank Adaptation) with rank 16 for 1 epoch on the Quantum Assistant Dataset. OLoRA uses QR decomposition for orthonormal initialization. This model was part of the Phase 1 PEFT comparison experiments and was not evaluated on external benchmarks.
Key Capabilities
- Code Generation: Generate complete Qiskit code from natural language descriptions
- Function Completion: Complete function bodies from signatures and docstrings
- Visual Understanding: Interpret quantum circuit diagrams, Bloch spheres, and histograms
- Conceptual Explanations: Answer questions about quantum computing theory
- Qiskit 2.0 Compliant: Uses modern APIs (SamplerV2, EstimatorV2, generate_preset_pass_manager)
Training Evaluation
This model was part of the Phase 1 PEFT variant comparison experiments. It was not evaluated on external benchmarks (Qiskit HumanEval, Qiskit HumanEval Hard, or synthetic test set).
Internal Validation Metrics
| Metric | Value | Step |
|---|---|---|
| Eval Loss | 0.742 | 183 (final) |
| Eval Token Accuracy | 0.794 | 183 (final) |
| Train Loss | 0.779 | 183 (final) |
| Train Token Accuracy | 0.793 | 183 (final) |
| Training Runtime | 1,067 seconds (~17.8 min) | 1 epoch |
PEFT Comparison Analysis
OLoRA showed significantly worse performance than other PEFT variants in Phase 1 experiments:
- Highest validation loss: 0.742 vs 0.622 for rsLoRA (19.3% higher)
- Lowest token accuracy: 0.794 vs 0.818 for rsLoRA
- QR orthonormal initialization appears unstable for this quantum computing domain
- Training and validation metrics nearly identical, suggesting poor learning
Conclusion: Orthonormal initialization (OLoRA) was the worst performing PEFT method in this comparison, likely due to instability with the multimodal architecture or domain characteristics.
Training Strategy
The experimental strategy was organized in two phases: PEFT technique selection and hyperparameter optimization.
Phase 1: PEFT Variant Comparison
Five LoRA variants were compared with controlled configuration (r=16, α=32, 1 epoch):
| Variant | Eval Loss ↓ | Eval Accuracy ↑ | Runtime (s) |
|---|---|---|---|
| rsLoRA | 0.622 | 0.818 | 1,060 |
| DoRA | 0.622 | 0.818 | 2,307 |
| rsLoRA (frozen aligner) | 0.623 | 0.817 | 1,057 |
| LoRA (vanilla) | 0.646 | 0.812 | 1,056 |
| PiSSA | 0.657 | 0.812 | 1,172 |
| OLoRA | 0.742 | 0.794 | 1,067 |
Key findings:
- rsLoRA and DoRA achieved equivalent performance (Eval Loss 0.622)
- DoRA has 2.18× computational overhead (2,307s vs 1,060s) due to magnitude-direction decomposition
- rsLoRA selected for optimal performance-efficiency trade-off
Phase 2: Rank and Epoch Optimization
With rsLoRA selected, the impact of adapter rank and training duration was investigated:
| Configuration | Eval Loss ↓ | Eval Accuracy ↑ | Notes |
|---|---|---|---|
| r=32, 1 epoch | 0.607 | 0.821 | Optimal trade-off |
| r=64, 1 epoch | 0.609 | 0.822 | Marginal improvement |
| r=16, 1 epoch | 0.622 | 0.818 | Baseline rsLoRA |
| r=32, 2 epochs | 0.638 | 0.825 | Slight overfitting |
| r=128, 3 epochs | 0.789 | 0.822 | Severe overfitting |
Conclusions: rsLoRA with r=32 and 1-2 epochs maximizes generalization while avoiding memorization of the synthetic dataset.
Model Collection
This model is part of the Quantum Assistant collection. All models are merged versions ready for inference:
| Model | Configuration | Description |
|---|---|---|
| Qwen3-VL-8B-rslora-r32-2 | rsLoRA r=32, 2 epochs | Best overall performance |
| Qwen3-VL-8B-rslora-r32 | rsLoRA r=32, 1 epoch | Best generalization |
| Qwen3-VL-8B-rslora-r64 | rsLoRA r=64, 1 epoch | Higher capacity |
| Qwen3-VL-8B-rslora-r128 | rsLoRA r=128, 1 epoch | Maximum capacity |
| Qwen3-VL-8B-lora | LoRA r=16, 1 epoch | Vanilla LoRA |
| Qwen3-VL-8B-dora | DoRA r=16, 1 epoch | Magnitude-direction decomposition |
| Qwen3-VL-8B-pissa | PiSSA r=16, 1 epoch | SVD initialization |
| Qwen3-VL-8B-olora | OLoRA r=16, 1 epoch | QR orthonormal initialization |
| Qwen3-VL-8B-rslora-frozen | rsLoRA r=16, frozen aligner | Ablation study |
| Qwen3-VL-8B-rslora | rsLoRA r=16, 1 epoch | Baseline rsLoRA |
Usage
With vLLM
python -m vllm.entrypoints.openai.api_server \
--host 0.0.0.0 \
--port 8000 \
--model samuellimabraz/Qwen3-VL-8B-olora \
--gpu-memory-utilization 0.92 \
--max-model-len 12288 \
--max-num-seqs 16 \
--max-num-batched-tokens 49152 \
--enable-chunked-prefill \
--enable-prefix-caching
With Transformers
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
model = Qwen3VLForConditionalGeneration.from_pretrained(
"samuellimabraz/Qwen3-VL-8B-olora",
torch_dtype="auto",
device_map="auto"
)
processor = AutoProcessor.from_pretrained("samuellimabraz/Qwen3-VL-8B-olora")
messages = [
{"role": "system", "content": "You are a quantum computing expert assistant specializing in Qiskit."},
{"role": "user", "content": "Create a function that builds a 3-qubit GHZ state and returns the circuit."}
]
messages_with_image = [
{"role": "system", "content": "You are a quantum computing expert assistant specializing in Qiskit."},
{"role": "user", "content": [
{"type": "image", "image": "path/to/circuit.png"},
{"type": "text", "text": "Implement the quantum circuit shown in the image."}
]}
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt"
).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=1024)
output = processor.batch_decode(
generated_ids[:, inputs.input_ids.shape[1]:],
skip_special_tokens=True
)[0]
print(output)
Training Details
Dataset
- Training Data: Quantum Assistant Dataset
- Train Samples: 5,837 (45.1% multimodal)
- Validation Samples: 1,239 (45.2% multimodal)
- Task Distribution: 30% function completion, 32% code generation, 38% QA
- Categories: 7 quantum computing domains
Training Configuration
| Parameter | Value |
|---|---|
| Base Model | Qwen/Qwen3-VL-8B-Instruct |
| PEFT Method | OLoRA (QR orthonormal initialization) |
| Rank (r) | 16 |
| Alpha (α) | 32 |
| Dropout | 0.05 |
| Target Modules | all-linear |
| Learning Rate | 2e-4 |
| LR Scheduler | Cosine |
| Weight Decay | 0.01 |
| Warmup Steps | 10 |
| Epochs | 1 |
| Batch Size | 16 |
| Precision | bfloat16 |
| Framework | ms-swift |
Freezing Strategy
| Component | Status |
|---|---|
| Vision Encoder (ViT) | ❄️ Frozen |
| Vision-Language Aligner | 🔥 Trainable |
| Language Model (LLM) | 🔥 Trainable |
Training Infrastructure
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (96GB VRAM)
- Training Time: ~17.8 minutes (1 epoch)
- Tracking: Weights & Biases | TensorBoard
System Prompt
You are a quantum computing expert assistant specializing in Qiskit.
Provide accurate, clear, and well-structured responses about quantum computing concepts,
algorithms, and code implementation. Use Qiskit 2.0 best practices.
Intended Uses & Limitations
Intended Uses
- Educational assistance: Learning quantum computing concepts with Qiskit
- Code generation: Creating Qiskit circuits from descriptions or diagrams
- Documentation: Understanding quantum circuit visualizations
- Research prototyping: Rapid development of quantum algorithms
Limitations
- Domain specificity: Optimized for Qiskit 2.0; may generate deprecated APIs for older versions
- Dataset size: Trained on 5,837 samples; may underperform on rare edge cases
- Category imbalance: Better performance on
circuits_and_gatesthanprimitives_and_execution - Hardware specifics: Limited coverage of IBM Quantum hardware-specific optimizations
- Execution: Generated code requires verification before running on real quantum hardware
Bias and Risks
- Model may perpetuate patterns from training data
- Visual understanding limited to common diagram styles in Qiskit documentation
- May generate syntactically correct but logically incorrect quantum algorithms
- Should not be used for production quantum computing without human review
Citation
If you use this model in your research, please cite:
@article{braz2026quantumassistant,
title = {Quantum Assistant: Specialization of Multimodal Vision-Language Models for Quantum Computing},
author = {Braz, Samuel Lima and Leite, Jo{\~a}o Paulo Reus Rodrigues},
journal = {Expert Systems with Applications},
year = {2026},
issn = {0957-4174},
doi = {10.1016/j.eswa.2026.133931},
url = {https://doi.org/10.1016/j.eswa.2026.133931},
publisher = {Elsevier}
}
Related Resources
- Paper: doi:10.1016/j.eswa.2026.133931
- Dataset: samuellimabraz/quantum-assistant
- Model Collection: Quantum Assistant Models
- Demo: Quantum Assistant Space
- Code: GitHub Repository
Acknowledgments
- IBM Quantum and Qiskit team for open-source documentation
- Qwen Team for the base model
- UNIFEI (Universidade Federal de Itajubá) for academic support
- Advisor: Prof. João Paulo Reus Rodrigues Leite
License
This model is released under the Apache 2.0 License.
- Task
- Image-Text-to-Text
- Library
- transformers
- Type
- qwen3_vl
- License
- apache-2.0
- Languages
- en
- Created
- Dec 8, 2025
- Updated
- Aug 13, 2026