shrew-2b

Shrew LoRA Adapters

LoRA adapters for Qwen/Qwen3.5-2B fine-tuned for structured extraction as part of a production RAG application. These power Shrew's structured extraction pipeline.

Adapters

The unified doc_processing/ adapter routes by system prompt — the prompt is just the task name (extract_metadata, semantic_chunk, or summarize_document). One adapter, three tasks.

doc_processing/ metrics

Usage

The unified doc_processing/ adapter routes by system prompt. Same adapter, different task per call.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-2B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B")
model = PeftModel.from_pretrained(base, "btbtyler09/shrew-2b", subfolder="doc_processing")

messages = [
    {"role": "system", "content": "extract_metadata"},   # or "semantic_chunk", "summarize_document"
    {"role": "user", "content": document_text},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)

extract_metadata and semantic_chunk produce JSON; summarize_document produces prose.

llama.cpp (GGUF)

GGUF versions can be used as LoRA adapters with llama-cli / llama-server:

llama-server -m Qwen3.5-2B.gguf --lora doc_processing.gguf

vLLM

vLLM loads Qwen3.5 as Qwen3_5ForConditionalGeneration (a VLM class), which nests the language model under a language_model. prefix. The adapters here are saved in standard PEFT format (trained with AutoModelForCausalLM), so the weight keys must be renamed before serving with vLLM's --enable-lora. Without the rename, vLLM silently zeros all LoRA weights with no error.

Apply the rename with fix_lora_keys.py from the shrew repo:

python fix_lora_keys.py path/to/adapter
vllm serve Qwen/Qwen3.5-2B \
  --enable-lora \
  --lora-modules doc_processing=path/to/adapter \
  --max-lora-rank 128 \
  --max-loras 1

Note --max-lora-rank 128 for the unified adapter (vLLM's default of 16 is too low).

Sampling parameters

Use Qwen 3.5 instruct-general parameters with enable_thinking=False:

License

Same as base model (Qwen/Qwen3.5-2B).