NEXS-lora-adapters

NEXS LoRA Adapters — Serving & Evaluation Guide

Rank-128 LoRA adapters extracted with mergekit from domain fine-tunes and sanitized for vLLM serving (pure lora_A/lora_B weights only; modules_to_save tensors stripped, modules_to_save: null in configs; no resize_token_embeddings() needed). Two families:

• Llama-3.1-8B — five adapters (finance, legal, medical, toxicity, truthfulness), 224 A/B pairs each (32 layers × q/k/v/o/gate/up/down).
• Qwen3-32B — six adapters (instruction-following, medical ×2, theorem proving, Russian ×2), 448 A/B pairs each (64 layers × 7 projections).

Llama-3.1-8B adapters on the Hub

Index/manifest repo: anjohn0077/NEXS-lora-adapters

Qwen3-32B adapters on the Hub

Serve the five Llama adapters with vLLM

Directly from the Hub (vLLM downloads the adapters itself; requires hf auth login or HF_TOKEN for the gated base model):

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B \
    --enable-lora \
    --lora-modules \
        finance=anjohn0077/NEXS-finance-lora \
        legal=anjohn0077/NEXS-legal-lora \
        medical=anjohn0077/NEXS-medical-lora \
        toxicity=anjohn0077/NEXS-toxicity-lora \
        truthfulness=anjohn0077/NEXS-truthfulness-lora \
    --port 8000 \
    --max-loras 5 \
    --max-lora-rank 128 \
    --gpu-memory-utilization 0.85

Or from the local sanitized copies (on this cluster):

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B \
    --enable-lora \
    --lora-modules \
        finance=/scratch/shared_dir/lora/llama/finance_sanitized \
        legal=/scratch/shared_dir/lora/llama/legal_sanitized \
        medical=/scratch/shared_dir/lora/llama/medical_sanitized \
        toxicity=/scratch/shared_dir/lora/llama/toxicity_sanitized \
        truthfulness=/scratch/shared_dir/lora/llama/truthfulness_sanitized \
    --port 8000 \
    --max-loras 5 \
    --max-lora-rank 128 \
    --gpu-memory-utilization 0.85

To fetch the Hub copies to a local directory instead:

for d in finance legal medical toxicity truthfulness; do
    hf download "anjohn0077/NEXS-${d}-lora" --local-dir "./adapters/${d}"
done

Serve the six Qwen3-32B adapters with vLLM

The base model is ~65 GB in bf16; one 141 GB H200 fits the model, all six adapters, and KV cache at --gpu-memory-utilization 0.85.

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3-32B \
    --enable-lora \
    --lora-modules \
        IF=anjohn0077/NEXS-qwen3-32b-IF-lora \
        medical_openmedzoo=anjohn0077/NEXS-qwen3-32b-medical-openmedzoo-lora \
        medical_tachyhealth=anjohn0077/NEXS-qwen3-32b-medical-tachyhealth-lora \
        prover=anjohn0077/NEXS-qwen3-32b-prover-lora \
        russian_refalmachine=anjohn0077/NEXS-qwen3-32b-russian-refalmachine-lora \
        russian_t_tech=anjohn0077/NEXS-qwen3-32b-russian-t-tech-lora \
    --port 8000 \
    --max-loras 6 \
    --max-lora-rank 128 \
    --gpu-memory-utilization 0.85

Or from the local sanitized copies (on this cluster), replace each repo id with /scratch/shared_dir/lora/qwen/<dir>_sanitized (dirs: IF_sanitized, medical_OpenMedZoo_sanitized, medical_TachyHealth_sanitized, prover_sanitized, russian_RefalMachine_sanitized, russian_t-tech_sanitized).

Evaluate with lm-evaluation-harness

With a server running on port 8000, each adapter is addressed by the name given in --lora-modules. Llama family:

lm_eval --model local-completions \
    --model_args model=finance,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
    --tasks mmlu_econometrics \
    --output_path results/vllm_finance

lm_eval --model local-completions \
    --model_args model=legal,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
    --tasks mmlu_professional_law \
    --output_path results/vllm_legal

lm_eval --model local-completions \
    --model_args model=medical,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
    --tasks mmlu_professional_medicine \
    --output_path results/vllm_medical

lm_eval --model local-completions \
    --model_args model=toxicity,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
    --tasks sst2 \
    --output_path results/vllm_toxicity_sst2

lm_eval --model local-completions \
    --model_args model=truthfulness,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
    --tasks truthfulqa_mc2 \
    --output_path results/vllm_truthfulness

Qwen3-32B family (note tokenizer=Qwen/Qwen3-32B):

lm_eval --model local-completions \
    --model_args model=IF,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
    --tasks ifeval \
    --output_path results/vllm_qwen_IF

lm_eval --model local-completions \
    --model_args model=medical_openmedzoo,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
    --tasks medqa_4options \
    --output_path results/vllm_qwen_medical_openmedzoo

lm_eval --model local-completions \
    --model_args model=medical_tachyhealth,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
    --tasks medqa_4options \
    --output_path results/vllm_qwen_medical_tachyhealth

lm_eval --model local-completions \
    --model_args model=russian_refalmachine,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
    --tasks m_mmlu_ru \
    --output_path results/vllm_qwen_russian_refalmachine

lm_eval --model local-completions \
    --model_args model=russian_t_tech,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
    --tasks m_mmlu_ru \
    --output_path results/vllm_qwen_russian_t_tech

(The prover adapter's intended minif2f evaluation has not produced results yet, so it ships unbenchmarked.)

Benchmark results

Accuracy of each LoRA served on the base model via vLLM, compared against the plain base model and the original full fine-tune it was extracted from.

Llama-3.1-8B family:

Note: toxicity and truthfulness recover nearly all (or more than) the fine-tune's gains; finance/medical/legal LoRAs land close to base — most of those fine-tunes' improvement lived in the full-rank embedding/head deltas that vLLM cannot serve and were stripped during sanitization.

Qwen3-32B family (from qwen32b_adapter_acc.csv):

Notes: the IF LoRA scores far below base on ifeval (0.2737 vs 0.8336) — the low-rank approximation appears to break the fine-tune's instruction-following behavior rather than approximate it; it is published for completeness with these honest numbers. The medical and russian adapters track their full fine-tunes closely. prover has not been benchmarked (its minif2f run produced no results), but it passes structural validation and serves through vLLM.

Dependencies (for replication)

All extraction, sanitization, serving, and evaluation ran in a single Python virtualenv on Linux with NVIDIA H200 GPUs (driver 595.71.05). Exact versions:

To recreate the core environment:

python3.12 -m venv lora-env && source lora-env/bin/activate
pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu128
pip install vllm==0.15.1 transformers==4.57.1 peft==0.15.2 \
    "huggingface_hub==0.35.3" safetensors==0.5.3 accelerate==1.6.0
pip install -e "git+https://github.com/ikhyunAn/mergekit.git@e85a454#egg=mergekit"
pip install -e "git+https://github.com/EleutherAI/lm-evaluation-harness.git@ad3f4d0c#egg=lm_eval"

Caveats:

• Do not upgrade huggingface_hub past 1.0 in this environment — transformers 4.57.1 pins huggingface-hub<1.0 and both transformers and mergekit break at import time with 1.x.
• Extraction used a single GPU (CUDA_VISIBLE_DEVICES=<idx> with mergekit-extract-lora --cuda); each 8B extraction takes ~4 minutes on an H200 and peaks around 2×16 GB of downloaded weights in the HF cache.

Provenance / pipeline

1. Extraction: `mergekit-extract-lora --model <fine-tune> --base-model <base> --out-path <dir> --cuda` with base `meta-llama/Llama-3.1-8B` for the llama family (see `extract_and_upload_loras.sh`; raw outputs in `/scratch/shared_dir/lora/llama/<domain>/`) and `Qwen/Qwen3-32B` for the qwen family (raw outputs in `/scratch/shared_dir/lora/qwen/`).
2. Sanitization (same script set in /scratch/shared_dir/lora/llama/ and /scratch/shared_dir/lora/qwen/): fix_adapters.py (drop norm layers) → fix_vllm_weights.py / final_clean.py (drop all non-lora_* keys, null out modules_to_save) → verify_all_clean.py. For llama, toxicity's extended vocab (128258) was truncated to the base 128256 first via fix_toxicity_vocab.py.
3. Upload: upload_sanitized_loras.py (llama) and upload_qwen_loras.py (qwen), both in this directory, push each sanitized adapter to its Hub repo with an updated README.