MariChatmen-4B-Experimental

MariChatmen 4B Experimental

MariChatmen 4B Experimental is a PEFT/LoRA adapter trained on top of Qwen/Qwen3.5-4B-Base.

It is an experimental research checkpoint, not a release-quality general assistant. The goal is to make the project inspectable and reproducible: this adapter corresponds to the 4B MariChatmen experimental result discussed in the project blog.

What this checkpoint is

• Base model: Qwen/Qwen3.5-4B-Base
• Adapter type: LoRA, causal language modelling
• LoRA rank: 32
• LoRA alpha: 64
• LoRA dropout: 0.05
• Extra trained token handling: trainable-token indices are enabled
• Intended language/style: Andalûh-style written Andalusian Spanish with a light fictional Sevillian persona

The adapter includes the tokenizer files used for this run. Use this tokenizer with the adapter; do not substitute a tokenizer from a different experiment.

Recommended use

Use short, conservative decoding. The current model is better as a compact experimental demo than as a long-form assistant.

max_new_tokens: 96-128
temperature: 0.2-0.4
top_p: 0.9
repetition_penalty: 1.08

Recommended system prompt:

Eres MariChatmen, también llamada MariCarmen: una sevillana ficticia nacida durante la Expo del 92. Responde con claridad, en Andalûh informal, y prioriza la respuesta útil antes que el chascarrillo.

For demos, the project also uses optional output guardrails: trim to complete sentences, cap at a few sentences, remove dangling follow-up questions, and optionally apply a protected Andalûh rendering layer.

Loading example

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

base_id = "Qwen/Qwen3.5-4B-Base"
adapter_id = "MariChatmen/MariChatmen-4B-Experimental"

tokenizer = AutoTokenizer.from_pretrained(adapter_id, trust_remote_code=True)

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    quantization_config=quantization_config,
    device_map="auto",
    trust_remote_code=True,
)

if len(tokenizer) > base_model.get_input_embeddings().num_embeddings:
    base_model.resize_token_embeddings(len(tokenizer))

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

Fixed-probe metrics

These metrics are from the project selection report. They are useful for comparison, not as a final release claim.

Known limitations

• Not release-quality as a general assistant.
• Persona strength remains low.
• Hard support prompts can still drift or produce awkward wording.
• Identity prompts can hallucinate metadata or over-associate cultural names.
• Raw generations benefit from short decoding and demo guardrails.
• A conservative ORPO continuation from this point was stopped because first probes regressed.

Data and attribution

The training pipeline used synthetic and transformed Spanish/Andalûh data, including project persona data, protected-span transliteration, and evaluation sets. The broader project also processed Spanish Wikipedia data from:

https://dumps.wikimedia.org/eswiki/20260501/

Wikipedia content is available under CC BY-SA 4.0 and GFDL terms; downstream uses should preserve the relevant attribution and licence obligations.

The Andalûh transliteration pipeline used andaluh-py / AndaluGeeks tooling. Curated project-authored data from this experiment is published at:

MariChatmen/MariChatmen-Project-Data

External-derived transformed rows are intentionally not republished there as project-owned data.

Project framing

This checkpoint is part of a staged experiment:

Qwen base
→ Qwen-Andaluh: neutral always-Andalûh assistant
→ MariChatmen: fictional Sevillian persona on top

The main engineering lesson is that dialect adaptation and persona adaptation should be separated. The model must first answer reliably in the target written variety; only then should a strong character voice be added.