MN-Violet-Lotus-12B-AttnRes

🪷 MN-Violet-Lotus-12B-AttnRes (Custom Architecture)

Overview

MN-Violet-Lotus-12B-AttnRes is a surgically modified version of MN-Violet-Lotus-12B.

This model features 12 Sutured AttnRes Blocks injected into Layers 14-25. These blocks act as structural anchors for state-tracking and logical consistency, specifically designed to solve the "lost in the middle" phenomenon and context collapse in long-form narratives.

⚠️ CRITICAL: How to Run This Model (No GGUF!)

Do not attempt to run this in KoboldCPP, LM Studio, or compile it to a standard GGUF.

Because this model contains a custom architectural spine (the injected AttnRes tensors), the rigid C++ llama.cpp engine will crash with an access violation. The llama.cpp engine is not built to recognize the custom math required for this model.

To run this model, you MUST use PyTorch / Huggingface Transformers via a local Python server. See the "Usage" section below for the exact API script.

Technical Specifications

• Developed by: Drakkhis
• Model type: Mistral-NeMo-12B with AttnRes Suture
• Hardware: Calibrated on NVIDIA GeForce RTX 5060 Ti (Blackwell / sm_120)
• Training Framework: Unsloth + PyTorch 2.12.0.dev (CUDA 12.8)
• Language: English

The "Suture" Calibration

The model underwent a specialized calibration process focused on aligning new attention blocks to the existing model logic.

• Method: DoRA (Weight-Decomposed Low-Rank Adaptation)
• Rank: 32 (Higher capacity for complex relational memory)
• Data: 1,500 rows of high-context, multi-turn story prose.

🛑 The Hardware Horizon (5,000 Token Limit) During native PyTorch testing without Flash Attention, we successfully pushed the model to ~5,147 tokens. At this depth, the model maintained 100% logical coherence, passing all multi-needle synthesis tests without any context smear.

However, attempting to scale to 6,000+ tokens caused the KV cache to exceed the 16GB VRAM limit of the RTX 5060 Ti. The system offloaded to shared memory, resulting in infinite hang times during the prefill phase.

Conclusion: The AttnRes spine functions perfectly up to the hardware limits of consumer GPUs in unquantized memory states. The true "Smear Horizon" of this model remains undiscovered. If you have a 24GB+ GPU, please download this model, push it to 10k+, and share your results!

---

🛠️ Usage (The Custom API Server)

Because of the custom architecture, you need to run this model using PyTorch in 4-bit mode. If you use SillyTavern, save the following script as tavern_server.py and run it in your Python environment.

import uvicorn
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from fastapi.middleware.cors import CORSMiddleware
from unsloth import FastLanguageModel

# Load the Cyborg with 4-bit Cache to save VRAM
print("📂 Loading Violet Lotus (4-bit) into VRAM...")
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="YOUR_HF_USERNAME/MN-Violet-Lotus-12B-AttnRes",
    max_seq_length=16384, # Cap at 16k for 16GB VRAM GPUs
    load_in_4bit=True, 
)
FastLanguageModel.for_inference(model)

app = FastAPI()
app.add_middleware(
    CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"],
)

@app.get("/v1/models")
async def get_models():
    return {"object": "list", "data": [{"id": "MN-Violet-Lotus-12B"}]}

async def generate_response(messages, temperature, max_tokens):
    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=max_tokens, temperature=temperature, use_cache=True)
    return tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)

@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
    data = await request.json()
    ans = await generate_response(data.get("messages", []), data.get("temperature", 0.8), data.get("max_tokens", 512))
    return JSONResponse(content={"choices": [{"message": {"content": ans}}]})

@app.post("/v1/completions")
async def legacy_completions(request: Request):
    data = await request.json()
    msgs = [{"role": "user", "content": data.get("prompt", "")}]
    ans = await generate_response(msgs, data.get("temperature", 0.8), data.get("max_tokens", 512))
    return JSONResponse(content={"choices": [{"text": ans}]})

if __name__ == "__main__":
    uvicorn.run(app, host="127.0.0.1", port=5000)

🧪 Evaluation: The "Do No Harm" 4K Stress Test

A common risk when injecting custom architecture (like our 36 AttnRes tensors) into a pre-trained LLM is catastrophic forgetting or logic degradation. To ensure the "Cyborg" spine didn't lobotomize the model's reasoning, we ran a Multi-Needle Logic Synthesis test at ~4,100 tokens.

The AttnRes architecture successfully integrated with the base Mistral NeMo layers without degrading its stellar mid-context reasoning capabilities. Both models flawlessly passed the 4K logic synthesis test before hitting the VRAM wall at 6k.

---

⚠️ Content Warning (NSFW)

This model was calibrated using a dataset containing NSFW (Not Safe For Work) story prose. As a result, the model is capable of generating adult content, graphic violence, and explicit themes. It is intended for use by adults (18+) in private roleplay or creative writing contexts. Please use responsibly.

---

📜 Accreditations & Credits

Base Model Merge: A huge credit to FallenMerick for the original MN-Violet-Lotus-12B merge. This model’s excellent prose and roleplay capabilities provided the perfect foundation for this architectural experiment.

Mistral NeMo 12B: Credits to the Mistral AI and NVIDIA teams for the base Mistral-NeMo-12B-Instruct-v1 architecture, which remains one of the most efficient 12B models for long-context handling.

Training Framework: This surgery was performed using the Unsloth library by unslothai. Their optimizations were critical for managing high-rank DoRA training on a consumer-grade Blackwell GPU.

Architecture Inspiration: The AttnRes (Attention-Residual) block concepts were inspired by recent advancements in "Sutured" neural architectures aimed at extending context coherence beyond standard KV-cache limits.

Hardware Support: Specialized optimization for this run was tailored for the NVIDIA GeForce RTX 5060 Ti (Blackwell) architecture.