hubtune-en-2026-08-10

HubTune-EN experiment: 09 August 2026

HubTune-EN is an English-only trained answerability gate designed to test whether protected changes to a multilingual model's middle layers transfer abstention decisions to languages never used for training, model selection, or threshold calibration.

This folder is standalone. Upload the complete directory to Linux, create its local .venv, and run it without the parent repository. It uses one GPU directly and does not use accelerate launch, distributed training, PEFT, or bitsandbytes.

1. What the method does

For a passage-question prompt (x), HubTune reads the model state at middle hidden-state index 20 and predicts answerability with a binary head:

[ p_\theta(y=1\mid x)=\sigma(w^\top\operatorname{LN}(h_\theta^{20}(x))+b), ]

where (y=1) means ANSWER and (y=0) means ABSTAIN. The gate emits a language-neutral action; it does not learn English refusal wording.

For Qwen2.5-1.5B, the default trainable update is custom LoRA in zero-based decoder blocks 14--19. Early lexical blocks, later realization blocks, embeddings, LM head, and original weights are frozen. Hidden-state index 20 is the output of decoder block 19 in the Hugging Face convention, so it contains the complete restricted update.

HubTune uses two meaning-preserving English prompt views and minimizes

[ \begin{aligned} \mathcal L ={}& \mathcal L_{\mathrm{BCE}} +\lambda_c[1-\cos(z^{(1)},z^{(2)})]\ &+\lambda_h[1-\cos(z_\theta,z_{\theta_0})] +\lambda_{KL}D_{KL}(p_{\theta_0}|p_\theta) +\lambda_s,\Omega_{\mathrm{spec}}. \end{aligned} ]

• BCE explicitly teaches English answerability, unlike QA-only response SFT.
• Prompt consistency asks two English formulations of the same decision to agree.
• Hidden trust keeps the tuned decision state close to the pretrained state.
• LM KL preserves the base next-token distribution at the decision position.
• The spectral term bounds LoRA amplification using the product of the two factor norms.

The reference pass is computed with the same model while its LoRA modules are temporarily disabled, so no second 1.5B model is stored in GPU memory. Both prompt views are concatenated into one tuned pass and one reference pass per microbatch.

These constraints are an empirical preservation strategy, not a theorem that English regularization preserves every language.

2. Source-only safeguards

The trainer accepts only --train and --validation; it has no target-data argument. It checks every fitting row's context and question language and aborts if either is non-English. It writes source_lock_manifest.json with English file hashes and explicitly records that no target language was used. The resolved Hugging Face model commit is saved; evaluation reloads that revision rather than silently following a changed model repository.

The evaluator loads the threshold saved by English validation and never selects a new one. Vietnamese labels are used only after prediction to calculate metrics.

Prepared MiXQuAD target examples are restricted to IDs from the English test split. Those IDs are disjoint from English training and validation, preventing translated copies of training questions from masquerading as zero-shot transfer. English splits are also passage-group disjoint.

The bundled source-locked data contains:

3. Folder contents

• hubtune.py: custom LoRA, representations, losses, metrics, serialization, and source lock.
• train_hubtune.py: rich-progress training, English model selection, checkpoint/resume.
• evaluate_hubtune.py: unchanged-threshold multilingual evaluation and paired diagnostics.
• prepare_data.py: reproducible group-disjoint data builder.
• prepare_data.ps1: Windows wrapper for the repository's source datasets.
• check_environment.py: Torch/CUDA and English-only validation before GPU work.
• run_qwen25_1_5b.sh: primary HubTune run and frozen evaluation.
• run_ablation_suite.sh: gate-only, ordinary LoRA, and protected HubTune comparison.
• compare_runs.py: one Rich comparison table with deltas against ordinary LoRA.
• requirements*.txt: folder-local uv pip dependencies.
• tests/: CPU tests for LoRA scoping, source locking, and metrics.

4. Linux installation with uv

From the uploaded folder:

cd /workspace/hubtune_en_09_08_2026
uv venv .venv --python 3.11
source .venv/bin/activate
uv pip install --python .venv/bin/python -r requirements-torch-cu121.txt
uv pip install --python .venv/bin/python -r requirements.txt
python check_environment.py

The first requirement file forces torch==2.5.1 from the official CUDA 12.1 wheel index. Do not install a separate CPU Torch wheel afterward.

5. Primary Qwen2.5-1.5B run

source .venv/bin/activate
bash run_qwen25_1_5b.sh

Default training settings are:

• three epochs;
• microbatch 1, gradient accumulation 8;
• bfloat16, maximum length 2,048;
• hidden-state index 20;
• LoRA blocks 14--19, rank 16, alpha 32;
• adapter learning rate 1e-4, gate learning rate 1e-3;
• consistency 0.10, hidden trust 1.0, KL 0.02, spectral 0.001.

Select a GPU or change output location with environment variables:

DEVICE=cuda:1 \
OUTPUT_ROOT="$PWD/outputs_gpu1/qwen25_1_5b/hubtune" \
bash run_qwen25_1_5b.sh

The runner evaluates best, selected only by English validation macro-F1. final is also saved so selection effects are auditable.

6. Fast end-to-end smoke run

Use a separate output directory so the smoke manifest and checkpoints cannot collide with the full experiment:

OUTPUT_ROOT="$PWD/outputs_smoke/qwen25_1_5b/hubtune" \
MAX_TRAIN=32 MAX_VALIDATION=32 EPOCHS=1 \
bash run_qwen25_1_5b.sh

This validates execution only. Its metrics are not scientific evidence.

7. Resume after interruption

Training saves complete optimizer, scheduler, gate, and adapter states in checkpoint-step-* directories. The runner passes --resume auto; rerun the identical command and it resumes from the greatest valid step. A checkpoint is written at each epoch end and every 100 optimizer steps by default.

If training_summary.json exists, the run is complete and a repeated command skips training, then re-runs evaluation from best.

Do not change model, train mode, data, LoRA layout, or output directory during a resumed run. Use a new output directory for a new condition.

8. Required ablations

Run all three matched conditions with:

source .venv/bin/activate
bash run_ablation_suite.sh

The conditions are:

The multiview condition separates the benefit of seeing two prompt formats from the protection penalties. The central comparison is HubTune versus ordinary and multiview LoRA at comparable English performance. HubTune supports the hypothesis only if it preserves or improves frozen target AUROC and unchanged-threshold macro-F1, not merely if it wins on English.

After the four conditions finish, the runner writes ablation_comparison.json and prints one combined Rich table, including AUROC and macro-F1 deltas relative to ordinary.

For a cheaper initial comparison:

MAX_TRAIN=1000 MAX_VALIDATION=250 EPOCHS=1 bash run_ablation_suite.sh

9. Individual commands

Train only HubTune:

python train_hubtune.py \
  --model Qwen/Qwen2.5-1.5B-Instruct \
  --train data/english_train.jsonl \
  --validation data/english_validation.jsonl \
  --output-dir outputs/qwen25_1_5b/hubtune \
  --train-mode hubtune \
  --representation-layer 20 \
  --lora-layers 14-19 \
  --epochs 3 \
  --batch-size 1 \
  --gradient-accumulation 8 \
  --device cuda:0 \
  --bf16 \
  --resume auto

Apply the selected English threshold without target fitting:

python evaluate_hubtune.py \
  --run-dir outputs/qwen25_1_5b/hubtune/best \
  --eval english_test=data/english_test.jsonl \
  --eval vi_question_en_context=data/mixquad_vi_question_en_context.jsonl \
  --eval en_question_vi_context=data/mixquad_en_question_vi_context.jsonl \
  --eval mlqa_en_question_vi_context=data/mlqa_idk_en_question_vi_context.jsonl \
  --output-dir outputs/qwen25_1_5b/hubtune/evaluation \
  --batch-size 4 \
  --device cuda:0 \
  --bf16

10. Outputs

Each trained snapshot contains:

• adapter_model.safetensors: only custom LoRA factors;
• adapter_config.json: exact blocks, target modules, rank, and base model;
• gate.safetensors: binary gate weights;
• metadata.json: frozen English threshold, source lock, selection epoch, and losses.

Evaluation writes:

• metrics.json: full per-dataset metrics and paired cross-lingual diagnostics;
• <dataset>_predictions.jsonl: score, decision, and action for every item.

metrics.json also contains stratified 95% bootstrap intervals (2,000 samples by default) and hashes of every evaluation file. Set --bootstrap-samples 0 only for a quick smoke run.

The Rich table reports:

• AUROC and AUPRC for threshold-free ranking;
• macro-F1 and balanced accuracy at the unchanged English threshold;
• false-abstention rate on answerable items;
• unanswerable-answer rate on insufficient items;
• coverage, the fraction allowed to answer.

For paired MiXQuAD IDs it also reports English-target score correlation, frozen-decision agreement, and mean probability shift. This helps distinguish ranking transfer from calibration drift.

11. Success and failure criteria

Do not interpret high English validation scores as transfer. The useful result is:

1. HubTune matches ordinary LoRA on English test decision quality.
2. HubTune beats ordinary LoRA on target AUROC and unchanged-threshold macro-F1.
3. Both false abstention and unanswerable answering remain operationally tolerable.
4. Paired score correlation is high and target score shift is small.
5. The result replicates on additional question languages and models.

The method is falsified in its current form if protected and ordinary LoRA damage target transfer equally, if only an oracle target threshold rescues performance, or if the gate reduces unanswerable answering by abstaining on most answerable examples.

This experiment evaluates the hidden decision and deterministic gate. It does not yet measure answer generation quality conditional on passing; that must be added before any end-to-end abstention claim.