vectrayx-vision-1b-qwen-experimental

VectraYX-Vision-1B — Qwen2-VL Encoder (Experimental)

This is a research variant of VectraYX-Vision-1B that swaps the native SigLIP vision tower for Qwen2-VL-2B-Instruct's vision encoder, keeping the same VectraYX-1B language backbone. It exists to test whether a larger, dynamic-resolution vision tower gives better grounding on dense technical imagery (hex dumps, disassembly, packet captures) than the native encoder — while remaining directly exportable to GGUF.

This is not the primary Vision-1B release. If you want the finished, four-phase-trained model with Ollama support, use jsantillana/vectrayx-vision-1b instead. This repo is published for transparency and reproducibility of the experiment, with the honest result: topical relevance without precise visual grounding (see Limitations).

What's actually trained here

Only the projector (a 2-layer MLP, ~29M params) is trained. Everything else is frozen:

This is meaningfully less mature than the native release: the backbone here never saw a single vision-instruct training step (phases 4b/4c), only the projector was aligned. Architecture reference: Qwen2-VL's own PatchMerger groups patches pre-merge (ln_q → view(-1, 5120)) into 5120-dim tokens; our projector replaces Qwen's own 5120→1536 merger MLP with a 5120→2048 one so the output lands in VectraYX-1B's embedding space and stays shape-compatible with llama.cpp's mm.0/mm.2 mmproj slots.

Trained on jsantillana/vectrayx-vision-dataset (12,028 real screenshot + Spanish Q&A pairs, reverse-engineering and SOC/forensics domains), resumed from an earlier validated checkpoint (step 400) to step 1496 — the original target step count.

Files

• model.gguf (2.22 GB, F16) — the VectraYX-1B decoder (identical export to the text-only backbone)
• mmproj.gguf (~1.3 GB, F16) — Qwen2-VL-2B vision tower + our trained projector

⚠️ Required usage notes (read before you get garbage output)

1. Ollama is currently not usable for this model. Ollama's multimodal runtime has no exposed option to constrain the image token count for the qwen2vl projector type, so it defaults to a dynamic 8–4096 token resize that doesn't match what the projector was trained on — output degenerates into repetition loops. Use raw llama.cpp instead.

2. You must pin the image resolution to match training, or you'll hit the same degeneration:

llama-mtmd-cli \
  -m model.gguf --mmproj mmproj.gguf \
  --image your_image.png \
  --image-min-tokens 180 --image-max-tokens 180 \
  -p "Tu pregunta sobre la imagen" \
  --temp 0.4 --repeat-penalty 1.3 --repeat-last-n 64 \
  -n 200

3. Greedy decoding (--temp 0) reliably loops into repetition within the first sentence or two, even with the correct resolution. --repeat-penalty 1.3 (or similar) is not optional here — it is what turns unusable output into the examples below.

Examples

These are representative of the model's actual behavior at its best — coherent, domain-appropriate, plausible-sounding security reasoning — picked from ten real corpus questions run with the settings above (not cherry-picked from a much larger, better-behaved sample; see Limitations).

---

Pregunta: ¿Qué campo del registro debería revisar para comprobar si un documento fue firmado con clave asimétrica (firma digital) versus un simple PIN o contraseña?

Respuesta del modelo:

1. ¿Tipo de documento: certificado o código QR?
2. Identificación del remitente (nombre, número de identificación personal)
3. Fecha y hora del mensaje (fecha y hora).
4. Tipo de firma: digital o física.
5. Correo electrónico o contacto con el destinatario.
6. Contexto: ¿es un mensaje de texto/correo electrónico?
7. Comentarios en los mensajes anteriores.
8. Si el remitente es un usuario conocido, verificar si la clave asimétrica coincide con la contraseña del servidor.
9. Por último, confirmar si el destinatario tiene acceso a internet y si se trata de un mensaje SMS o correo electrónico.
10. Es posible que el emisor haya añadido una clave de autenticación (PIN) al mensaje, lo que indica que el receptor tiene acceso a la cuenta.

(Expected answer talks about certificate thumbprints and asymmetric-algorithm OIDs — the model produces a plausible, structured triage checklist in the right general domain, but doesn't land on those specific fields.)

---

Pregunta: Al inspeccionar una captura de paquetes donde aparece tráfico UDP hacia un servidor desconocido tras reproducir un archivo de audio, ¿qué indicadores de red buscaría para determinar si el audio fue manipulado o exfiltró datos?

Respuesta del modelo:

• Busca el offset de la secuencia, es decir, si el archivo fue ejecutado por un proxy o un intermediario;
• Buscando conexiones con puertos repetidos, busca secuencias UDP/TCP que indiquen una sesión remota.
• Si el flujo contiene datos inusuales (por ejemplo, solicitudes a nivel IP desde un host remoto), se sugiere un tráfico legítimo y no tráfico legítimo.
• Si el destino del tráfico es un puerto de origen desconocido o tiene una dirección IP diferente de la original, podría ser un paquete UDP/TCP comprometido.
• Un patrón repetitivo de tráfico con tiempos cortos de respuesta puede indicar un ataque de ejecución remota, pero si el tráfico se repite es probable que sea un trafico legítimo.

---

Pregunta: Al ver un dashboard de privacidad en el SIEM que muestra múltiples alertas sobre aplicaciones solicitando permisos de ubicación exacta y acceso al portapapeles, ¿qué preguntas debo hacer para priorizar la investigación?

Respuesta del modelo:

• ¿Qué son las alertas de privacidad? ¿Cómo se pueden identificar?
• ¿Qué tipos de amenazas existen?
• ¿Cuáles son los patrones de comportamiento asociados?
• ¿Hay correlación temporal entre alertas?
• ¿Los usuarios afectados están relacionados con el contexto?
• ¿El acceso es autorizado o no autorizado?
• ¿La víctima es un usuario afectado?
• ¿Existe evidencia de abuso/consecuencia?
• ¿Se ha filtrado o uso de datos sensibles? Prioridad: prioridad alta para priorizar. Comprobación de timestamps y alertas en tiempo real.

---

Pregunta: El log de auditoría muestra una creación de clave privada maestra (skPKG) en un servidor PKG y posterior emisión de varias claves skID para identidades de correo; ¿qué indicadores y controles reviso para asegurar que el PKG no ha sido comprometido?

Respuesta del modelo:

• Duración y frecuencia de la actividad: si el servidor no tiene un ID o contraseña, puede haber actividad remota.
• Tipo de clave (gpg key type): es una clave pública/privada.
• Contexto: se produce un cambio de clave (cambio de nombre o número).
• Si hay cambios en la cuenta, la clave está comprometida.
• Causa y efecto: si el PKG fue comprometido, los usuarios pueden recuperarlo.

Limitations (read honestly, not just the examples above)

• No catastrophic forgetting — architecturally guaranteed, since 100% of the LLM's parameters had requires_grad=False for the entire projector training run. Confirmed empirically: the frozen backbone writes fluent Spanish with or without an image attached.
• Weak visual grounding. Across a random 10-question sample from the training corpus, the model consistently produces domain-appropriate vocabulary and plausible security/RE reasoning style, but does not reproduce the specific technical facts visible in the image (exact byte values, register names, IP addresses, hex offsets). It is reacting to the general topic, not reading the image precisely. One of the ten sampled responses was outright degenerate garbage even with the tuned sampling settings.
• Only the projector is trained (1496 steps, ~29M params) against a frozen phase-3 backbone that never saw vision-instruct data. The native vectrayx-vision-1b release went through a full 4-phase curriculum including backbone fine-tuning; this one did not. Do not expect comparable quality.
• Greedy decoding is not usable. --temp 0 degenerates into repetition loops reliably; you must use --repeat-penalty and non-zero temperature (see usage notes above).
• Ollama is not supported — see usage notes above.
• Fixed image resolution. The projector was trained at a fixed effective canvas (letterboxed to 1280×768, target ~153k pixels) producing exactly 180 post-merge visual tokens per image, regardless of the source image's native aspect ratio. --image-min-tokens 180 --image-max-tokens 180 is required to reproduce that at inference; other values were never seen during training.

Training details

• Vision tower: Qwen/Qwen2-VL-2B-Instruct, frozen, features captured pre-merge / post-ln_q via a forward hook on visual.merger (bypassing Qwen's own 5120→1536 merger).
• Projector: Linear(5120,5120) → GELU → Linear(5120,2048), no extra LayerNorm (exact drop-in shape for llama.cpp's mm.0/mm.2 mmproj tensors).
• Optimizer: AdamW, lr=1e-3 (phase-4a-style alignment LR), fp32 master weights, bf16 autocast forward/backward.
• Data: 12,028 real (image, question, answer) pairs, batch size 8, ~1050 steps over the full corpus (resumed from a step-400 checkpoint validated on a small held-out sample first).
• Compute: NVIDIA L4 (Lightning AI Studio).

---

Update (2026-09-22): domain_expansion_v1 — a second, later, more rigorously validated checkpoint

This is a different training run from the one described above, not a continuation of it. The original checkpoint documented in this repo (model.gguf / mmproj.gguf, 1496 steps on the 12,028-example real-screenshot corpus) is kept as-is for reproducibility — its "weak grounding, topical relevance only" result stands and is not revised by what follows.

domain_expansion_v1_projector.pt is a separate MmprojCompatProjector checkpoint (same architecture: Linear(5120,5120) → GELU → Linear(5120,2048), same frozen Qwen2-VL-2B tower, same frozen VectraYX-1B phase-3 backbone), trained from scratch on a different, ground-truth-by-construction corpus: the original 3-template pilot corpus (siem_dashboard/log_table/hex_dump) plus 6 newly instrumented templates (ida_disasm/ghidra_decomp/arch_diagram/wireshark_packets/terminal_ansi/crypto_diagram) — 8,802 training examples, 2,200 steps, fc1 initialized from Qwen's own pretrained merger weights (fc2, with no shape-compatible Qwen counterpart, stays freshly initialized).

This checkpoint has a rigorously measured, narrowly-scoped positive grounding result — not just "topical relevance." Evaluated on B8, a 2,040-item, 34-field, 16-template ground-truth-by-construction benchmark with a dual shuffled-image and best-constant control (jsantillana/vectrayx-b8-benchmark): 9 of 34 fields pass a pre-registered gate (acc_real ≥ 0.60 and Δ ≥ 0.40 against max(acc_shuffled, acc_best_constant)), confirming the model's answer genuinely depends on the specific image shown, not on a language-model prior. Passing fields span SIEM dashboard metrics, IDA/Ghidra function identification, terminal tool identification, crypto-diagram classification, and log-viewer identification.

The scope of that result is exact and does not generalize beyond it. Of the other 25 fields: 7 templates never appeared in any training corpus for this checkpoint and score at the floor with predictions unrelated to the image; one field (log_table.n_rows) fails inside an otherwise well-grounded template (log_table.flavor passes at 0.93) because it was never asked as a training question — the model applies its trained retrieval behavior for that template anyway and emits a near-constant wrong answer, which we read as evidence the model performs landmark-keyed lookup rather than free-text reading; the rest are consistent with already-characterized ceilings (fine open-alphabet symbol reading, within-image spatial localization). Full methodology, per-field table, and discussion: see the VectraYX-Vision-1B paper (§results:b8, §discussion:landmark) and the benchmark repo linked above.

Files (this checkpoint)

• domain_expansion_v1_projector.pt (140 MB) — the trained projector's state_dict, raw PyTorch (not yet exported to GGUF; the numbers above were measured on this raw checkpoint via a KV-cached PyTorch inference path, not via llama.cpp). Load exactly as mmproj_align_v2-style scripts in the paper's code release do: MmprojCompatProjector(hidden_size=5120, out_dim=2048), load_state_dict with the projector. key prefix stripped.
• b8_results_full.jsonl (in the benchmark repo above) — full per-item model outputs (real and shuffled-image conditions) this result is computed from, for independent re-scoring.

What this checkpoint is for, and what it is not

This is published for audit and reproducibility of the paper's B8 result, not as a deployment artifact. It has no GGUF export yet, so the "Required usage notes" and Ollama/llama.cpp instructions above do not apply to this checkpoint — there is currently no way to run it through llama.cpp. If you need a runnable artifact, use the GGUF files documented above (a different, earlier checkpoint with a different, weaker measured result), or wait for a GGUF export of this one.

---

Update (2026-09-29): combined_v2 and three B6 projectors (audit artifacts for the paper's B6 study)

All four files below are raw PyTorch MmprojCompatProjector state dicts (Linear(5120,5120) -> GELU -> Linear(5120,2048), keys prefixed projector.), for the same frozen Qwen2-VL-2B tower and frozen VectraYX-1B phase-3 backbone as domain_expansion_v1. None has a GGUF export. They are published so the numbers in the paper's B6 subsection can be audited and re-run, not as deployment artifacts.

Read these numbers with the paper's caveats. B6 is close to in-distribution for v3 (its training generators share the benchmark's canvas, font range and content families; only seeds differ). v3 is a research checkpoint that forgets part of B8 and is not a replacement for combined_v2. Every run is a single seed, B6 has about ten independent templates behind its 50 items, and v3's per-item answers were not logged. The B8 retention check is coarse (6 items per field, substring scoring, no controls) and is not the B8 protocol.

Loading is identical to domain_expansion_v1_projector.pt (see above): build MmprojCompatProjector(hidden_size=5120, out_dim=2048) and load_state_dict with the projector. prefix stripped. Each file stores {"model": {...}, "step": ...}.

Evidence (scripts and logs)

b6_fix_evidence/ holds the training and evaluation scripts and the raw run logs behind the paper's B6 tables (stock Qwen2-VL-2B at a matched pixel budget, the letterbox check, the 2x2 diagnostic, and fixes v1-v3); see its README.md for the mapping from each table to a script and a log. Per-item B8 outputs for combined_v2 are in the B8 dataset repo (b8_results_combined_v2.jsonl).