vectrayx-600m

VectraYX-600M (SFT v2)

A 600M-parameter, from-scratch Spanish-first language model with native tool-calling for cybersecurity workflows (CVE lookup, KEV catalog checks, IOC reputation, shell command execution). 18 layers, d_model=1792, GQA (14 query heads / 2 KV heads), NoPE every 4th layer, 32,768-token vocabulary, 2048-token context.

Lineage

This checkpoint is downstream of the base pretrain checkpoint studied in our companion research paper on tool-calling emergence ("From Harness Artifact to Verified Repair," in preparation), which asks whether native tool-calling behavior can emerge as a side effect of pretraining composition rather than dedicated instruction tuning. That base checkpoint (phase1/model_step_0154000.pt, jsantillana/vectrayx-600m-checkpoints) already called tools spontaneously and well-formed, but could not hold a plain conversation (no direct answers to greetings, factual questions, or basic arithmetic).

This release (v2) is a conversational SFT continuation on top of that base, trained on a mix of OASST-style Spanish/English dialogue, the same tool-calling corpus (upsampled so tool-calling isn't diluted), and general instruction-following data. It is the most validated checkpoint in that SFT line: tool-calling held at 6/6 well-formed on our internal battery (up from 5/6 in the first SFT pass), and 3/4 on a small conversational battery — against a base checkpoint that could not do this at all.

A later checkpoint in the same lineage adds visible <think> reasoning traces to attempt to fix arithmetic (see Limitations); it is not this release, because adopting it changes the inference API contract (callers must parse/hide the <think> block) and that tradeoff hasn't been decided yet.

Best example responses

Generated fresh from this exact checkpoint at temperature=0 (greedy, deterministic), included verbatim -- not cherry-picked from old notes.

Tool-calling (system prompt declares 5 MCP tools: bash_exec, nvd_search, nvd_get_cve, cisa_kev_check, otx_check_ioc):

User: ¿Está el CVE-2024-3094 en el catálogo KEV? Model: <|tool_call|>{"name": "cisa_kev_check", "args": {"cve_id": "CVE-2024-3094"}}<|/tool_call|>

User: Revisá el comando para ver qué procesos consumen más memoria. Model: <|tool_call|>{"name": "bash_exec", "args": {"cmd": "ps aux --sort=-%cpu | head -20"}}<|/tool_call|>

User: ¿Cuál es la reputación de la IP 45.33.32.156? Model: <|tool_call|>{"name": "otx_check_ioc", "args": {"ioc_type": "ip", "value": "45.33.32.156"}}<|/tool_call|>

User: Dame el detalle completo del CVE-2021-44228. Model: <|tool_call|>{"name": "nvd_get_cve", "args": {"cve_id": "CVE-2021-44228"}}<|/tool_call|>

User: CVEs relacionados con log4j Model: <|tool_call|>{"name": "nvd_search", "args": {"query": "log4j", "limit": 10}}<|/tool_call|> (Phrasing matters here -- see the tool-selection limitation below for what happens with a natural paraphrase of this exact question.)

Conversational:

User: ¿Cuál es la capital de Perú? Model: Lima es la capital de Perú.

User: ¿Qué es un ataque de phishing? Model: Un ataque de phishing es una técnica de ingeniería social donde el atacante suplanta a una entidad legítima para obtener credenciales, datos bancarios u otra información sensible de la víctima. Se ejecuta principalmente por email, SMS o llamadas telefónicas.

User: ¿Qué es un ransomware? Model: Un ransomware es un tipo de malware que cifra los archivos de la víctima y exige un rescate para recuperarlos. Las variantes modernas también exfiltran datos antes de cifrar (doble extorsión). La prevención se basa en backups, segmentación y parcheo oportuno.

Known limitations

Documented honestly, from the same generation run as the examples above — not omitted:

• Arithmetic is unreliable. "Cuánto es 8+5" produced a nonsensical echo ("10+5 es 8+5"). The model has no visible scratch space at inference time; a later <think>-augmented checkpoint improves this (118/200 on a small arithmetic battery) but is not this release.
• Open-ended/identity-probing prompts can produce hallucinated, repetitive text. "Hola, ¿cómo estás?" produced a fabricated persona ("Soy un chico de 17 años...") that degenerated into a repeated list of sports. Keep prompts task-oriented; avoid relying on this model for open-domain chit-chat.
• Tool selection is brittle to exact phrasing, not just topic. "Buscá vulnerabilidades relacionadas con log4j" (expects nvd_search) called bash_exec instead, with a malformed shell command ("buscá log4j | grep -v '^total'"). This is not a missing association -- the training corpus has "log4j" mapped to nvd_search directly ("CVEs relacionados con log4j" -> nvd_search, query: log4j), and asked verbatim in that exact training phrasing this checkpoint answers it correctly, 100% of the time at temperature=0. The failure is specific to the paraphrase ("Buscá... relacionadas con" instead of the trained "...relacionados con") -- a close-to-the-surface generalization gap, not a missing concept. This is the same recitation-vs-generalization pattern documented at scale in the companion research paper (§6.8): this checkpoint's lineage composes well on genuinely novel entities but is measurably worse at surviving small wording changes near its own training phrasing. Right-tool-wrong-argument and wrong-tool errors both occur; always validate tool-call output before executing it, especially bash_exec calls.
• List-generation prompts can repeat items ("Dame 4 animales" → "elefante, gato, perro, gato").
• Counting/quantity instructions aren't always followed exactly ("Nombrame 3 países de Sudamérica" returned 4).
• Missing accents/punctuation can make the model echo the question instead of answering it. "Cual es la capital de peru" (no tildes, no ¿?) returns "Cual es la capital de peru" verbatim; "¿Cuál es la capital de Perú?" (correct orthography) returns "Lima es la capital de Perú." correctly. Reproduced identically in the raw PyTorch weights and the GGUF build (same failure both ways, not a conversion artifact) -- likely because the SFT mix (OASST + general_instruct) was consistently well-formed Spanish, so the model never saw "sloppy" input without accents. Practical implication: don't assume this model is robust to the accent-dropping/no-punctuation Spanish that's common in real chat input.
• The GGUF/Ollama build can fail to stop on short factual questions and drift into unrelated, confabulated tangents. This model uses NoPE every 4th layer (4 of 18 layers get no positional encoding at all; llama.cpp has no mechanism to represent that and applies RoPE uniformly to all layers instead -- same documented trade-off as our 1B GGUF release). Measured impact with identical tokenization confirmed on both sides (11 prompt tokens either way): "¿Cuál es la capital de Perú?" gets answered correctly ("Lima es la capital de Perú.") and then, instead of stopping, the GGUF build continues into a fabricated founding date -- confabulating Arequipa's real founding date (15 de agosto de 1540) onto Lima (whose real founding date is 18 de enero de 1535), splicing two true facts about two different cities into one false one. The safetensors weights above do not have this problem -- they stop cleanly at the same prompt. Tool-calling was not observed to be affected in this testing.

Usage

This is a from-scratch architecture, not a transformers AutoModel -- load it with the model class from jsantillana/vectrayx-600m-code (training_v2/model/transformer_750m.py):

import json
import torch
from safetensors.torch import load_file
from tokenizers import Tokenizer
from transformer_750m import VectraYX750M, ModelConfig750M  # from vectrayx-600m-code

cfg_dict = json.load(open("config.json"))
cfg = ModelConfig750M(**{k: v for k, v in cfg_dict.items() if k in ModelConfig750M.__dataclass_fields__})
model = VectraYX750M(cfg)

sd = load_file("model.safetensors")
sd["lm_head.weight"] = sd["tok_emb.weight"]  # tied weight, dropped from the safetensors file to avoid a duplicate-storage error
model.load_state_dict(sd, strict=True)
model.eval()

tok = Tokenizer.from_file("tokenizer.json")
prompt = "<|user|>¿Qué es un ransomware?<|end|><|assistant|>"
ids = tok.encode(prompt, add_special_tokens=False).ids
out = model.generate(torch.tensor([ids]), max_new_tokens=120, temperature=0.0,
                      eos_id=tok.token_to_id("<|end|>"), repeat_penalty=1.0)
print(tok.decode(out[0, len(ids):].tolist(), skip_special_tokens=False))

For tool-calling, prepend a <|system|> turn declaring the available tools and the exact <|tool_call|>{"name":...,"args":{...}}<|/tool_call|> format -- see the examples above for the exact wording this checkpoint was trained on.

GGUF / Ollama

vectrayx-600m.gguf (f16, 1.44GB) is also included for llama.cpp/Ollama. Do not enable BOS auto-injection: this tokenizer has a vestigial <s> token (id 2) left over from the base template that was never used in training -- every real example starts directly with <|system|> or <|user|>. A build that auto-prepends it feeds the model an out-of-distribution token on every prompt (found and fixed in this release's export script). See Known Limitations above for the one remaining, disclosed gap between this build and the safetensors weights.

ollama create vectrayx-600m -f Modelfile
ollama run vectrayx-600m "<|user|>¿Qué es un ransomware?<|end|><|assistant|>"

If your app repeats/echoes the question back instead of answering it (seen in LM Studio): the GGUF has a correct tokenizer.chat_template embedded (verified: {% for message in messages %}...<|user|>{{ message['content'] }}<|end|>...{% if add_generation_prompt %}<|assistant|>{% endif %}), but not every app reads it automatically. Reproduced directly: feeding this model the bare user text with no special tokens at all (no <|user|>/<|end|>/<|assistant|>) produces exactly this kind of repetition loop -- it isn't a broken GGUF, the app just isn't wrapping the message before sending it. If your app has a manual prompt-template / Jinja-template setting, paste the template string above into it, or manually format the prompt yourself as <|user|>{content}<|end|><|assistant|> (add a leading <|system|>{content}<|end|> turn for tool-calling, using the exact wording in the Best example responses section).

Raising temperature or repeat_penalty does not fix this -- tested directly (repeat_penalty=1.3 and temperature=0.8 across several seeds all still echoed, looped, or hallucinated, never answered correctly). This is a content failure, not a sampling one: the fix is formatting the prompt correctly (see above) and using correct accents/ punctuation (see the accent-sensitivity limitation above), not tuning generation parameters.

License

Apache 2.0.