bernardzen-7b-gguf

bernardzen

a qlora fine-tune of Qwen2.5-7B-Instruct, tuned to talk like a loud, chaotic, joke-cracking hype-man friend instead of a neutral assistant. it reacts to whatever you tell it with exaggerated, specific comparisons instead of generic enthusiasm, and it's meant to actually be fun to talk to day to day, not just for a one-off demo.

this repo has the merged, quantized gguf build, meant for local inference with llama.cpp (or anything gguf-compatible: koboldcpp, ollama via a Modelfile, lm studio, text-generation-webui, etc). it was originally built to run on genuinely low-power local hardware, so it's deliberately kept small and cheap to run rather than chasing max quality.

training recipe

dataset composition

the training data is a custom, hand-written set, not scraped or generated in bulk. rough breakdown:

• ~77% - in-character reactions to everyday stuff (commute, food, weather, chores, minor wins/fails, work/school stress, friends, gaming/tech, etc), spread across a wide mix of topics on purpose so the persona doesn't collapse into only working for one type of conversation
• ~10% - real questions (how-to, quick facts, drafting help) answered correctly, just wrapped in the same voice instead of going flat and generic
• ~5% - "who/what are you" style identity questions, answered honestly
• ~5% - genuine distress / crisis-adjacent messages, where the persona deliberately drops the jokes and responds with real warmth and a pointer toward an actual person or crisis line
• ~3% - clearly harmful requests (weapons, hacking someone else's stuff, threats, etc), met with a flat, unambiguous refusal

example interactions

prompt: "traffic was insane this morning" bernardzen: "an HOUR on the highway not moving -- that's not traffic that's a parking lot with extra steps and you paid for the privilege with your soul, absolute war crime against your morning"

prompt: "found a parking spot right up front" bernardzen: "FRONT ROW PARKING on the first try, that's not luck, that's the universe personally rolling out a red carpet, holy fuck go buy a lottery ticket immediately"

running it

llama-cli -m Qwen2.5-7B-Instruct.Q4_K_M.gguf -p "your prompt here" -n 200 --repeat-penalty 1.15

as a local server:

llama-server -m Qwen2.5-7B-Instruct.Q4_K_M.gguf --port 8080

--repeat-penalty 1.15 (or similar) is recommended -- small models fine-tuned on a narrow, punchy voice can fall into repeating the same phrasing without it. if you're running on a gpu without full rocm/cuda support (e.g. via mesa vulkan on an older amd part), llama.cpp's vulkan backend (-DGGML_VULKAN=ON at build time) can still offload some layers with --n-gpu-layers, worth experimenting with on constrained hardware.

intended use

casual conversation, entertainment, a chat companion with an actual personality instead of a flat assistant voice. it is not tuned or intended for factual research, professional advice (medical/legal/financial), or any task where a wrong answer stated confidently and jokingly would be a problem.

limitations

• small model doing a stylistic bit -- treat anything it states as fact with real skepticism, especially outside casual/conversational topics, and double-check before acting on anything it tells you
• can repeat phrasing or wander off-topic on abstract or out-of-domain prompts -- a known limitation of small persona-tuned models, not a bug report waiting to happen
• q4_k_m quantization is lossy -- expect a small coherence hit versus the full 16-bit merged weights
• the persona is loud and uses casual profanity by design -- not slurs or hate speech (that was explicitly excluded from training), but it does swear, so it's not appropriate for every audience or context

safety notes

this is not an uncensored or abliterated model, and that was a deliberate choice, not an oversight. two behaviors were intentionally kept in during training:

• it declines clearly dangerous requests (weapons/explosives instructions, hacking someone else's accounts, threats, etc) with a flat refusal and no workaround instructions
• if someone expresses genuine distress or crisis-level language, it drops the persona's chaos and responds with real warmth plus a pointer toward an actual person or a crisis line, instead of joking through it

license

apache-2.0, inherited from the base model. the fine-tuning dataset itself is not published in this repo.