Downloads · 30 days
5
19% of all-time downloads
NghonseFiassap/Qwen3.5-0.8B-pallet-manifest-lora
Qwen3.5-0.8B-pallet-manifest-lora is a image-text-to-text model from NghonseFiassap. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
LoRA adapter for a compact generative Vision-Language Model (unsloth/Qwen3.5-0.8B) that extracts Pallet Load Manifests from document images directly into a strict JSON schema — without a classical OCR pipeline. Traine…
Downloads · 30 days
5
19% of all-time downloads
All-time downloads
26
Public
Repo size
72.8 MB
Likes
0
Public
Click a slice to open those files.
.safetensors52.8 MB · 73%
From the Hugging Face model README
LoRA adapter for a compact generative Vision-Language Model (unsloth/Qwen3.5-0.8B) that extracts Pallet Load Manifests from document images directly into a strict JSON schema — without a classical OCR pipeline. Trained as part of a Master's thesis at Fachhochschule Südwestfalen.
Kurzfassung (DE): Überwacht (LoRA-SFT) feinjustierter, kompakter VLM-Adapter zur schemakonformen Extraktion von Paletten-Manifesten aus Dokumentenbildern. Ende-zu-Ende (Bild + Prompt → JSON), ohne klassische Texterkennung (OCR).
unsloth/Qwen3.5-0.8B (4-bit NF4 quantised)r = 16, alpha = 16, dropout 0image-text-to-text)Comparison of the frozen base model vs. this LoRA adapter, same prompt and evaluation protocol for both.
| Metric | Base (zero-shot) | LoRA (this adapter) |
|---|---|---|
| WES (Weighted Extraction Score) | 0.049 | 0.996 |
| Schema compliance | 0 % | 100 % |
| CER (full JSON, lower=better) | 0.672 | 0.013 |
| Exact match | 0 / 10 | 9 / 10 |
| Field accuracy | 0.000 | 0.996 |
| Products list-F1 | 0.00 | 1.00 |
| ROUGE-L-F1 | 0.58 | 0.99 |
| BERTScore-F1 | 0.815 | 0.996 |
| LLM-judge (GPT-4o+4.1 ensemble, 1–5) | 1.36 | 4.96 |
The main effect is structural: the base model already reads part of the field values but serialises them into an unusable JSON envelope (schema compliance 0 %). The adapter fixes the serialisation into the binding schema — schema compliance rises to 100 % and the character error rate drops by ~50×.
Note on scope: All figures are point estimates over a small test split (n = 10) on a synthetically homogeneous layout. They characterise this domain; transfer to other layouts, real photographs or foreign schemas has not been evaluated.
shipment_id, asn_number, dock, route, products[], totals{}).Multi-page documents, layouts/schemas outside the training domain, and any use where a wrong field silently propagates without human/downstream validation.
from unsloth import FastVisionModel
from peft import PeftModel
model, processor = FastVisionModel.from_pretrained(
"unsloth/Qwen3.5-0.8B", load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "NghonseFiassap/Qwen3.5-0.8B-pallet-manifest-lora")
FastVisionModel.for_inference(model)
# messages = [{"role": "user", "content": [
# {"type": "image", "image": image},
# {"type": "text", "text": prompt},
# ]}]
# inputs = processor.apply_chat_template(messages, add_generation_prompt=True,
# tokenize=True, return_tensors="pt").to("cuda")
# out = model.generate(**inputs, max_new_tokens=1024)
The recommended production prompt is the schema-guided variant (P2): it is the strongest prompt on both models and is robust after fine-tuning (WES spread across five prompt variants drops from ≈ 0.85 to ≈ 0.002).
SFTTraineradamw_8bit (QLoRA), learning rate 1e-4, linear schedule, warmup 20max_steps = 70 (< 1 epoch equivalent, deliberate short run against memorisation)max_length = 2048A six-configuration sweep (identical rank, steps and protocol) localises the gain in the language layers: language-only LoRA replicates the full run to within 0.002–0.004 WES points, while vision-only configurations stay at schema 0 % — one vision variant even regresses the CER (defect mode only visible through schema-aware metrics).
Master's thesis, Fachhochschule Südwestfalen (2026): Strukturierte Informationsextraktion aus Dokumentenbildern mit kompakten generativen Vision-Language-Modellen.
Adapter released under apache-2.0. Please also observe the license of the base
model unsloth/Qwen3.5-0.8B and of the training dataset (CC BY 4.0).