dpo_humour

Swiss-Knife

Swiss Knife: Complete Experimental Plan

Pragya Lab, BITS Pilani Goa

---

Paper Abstract (Swiss Knife)

Alignment today is still checkpoint-bound: most pipelines implicitly hard-code a single behavioral compromise into a trained model, making it costly to revise, specialize, or hot-fix behavior after deployment. We introduce Swiss Knife, a protocol that repurposes speculative decoding into an alignment socket: a fast draft model generates candidates, while a separately trainable auditor model enforces a chosen alignment objective during generation. Swiss Knife's key algorithmic component is the Tournament Sampling Auditor (TSA) — either a pairwise knockout bracket or a Swiss-system schedule — to select the winner under an objective-specific score. Crucially, Swiss Knife supports objective-specific auditor "blades" (safety, helpfulness, harmlessness, informativeness, style) that can be tuned, swapped, and updated independently of the backbone.

---

Table of Contents

1. Background: What ECLIPTICA/CITA Built
2. Swiss Knife Architecture
3. Related Papers and Positioning
4. Experiment Plan
   • Phase 0 — Setup and Baselines
   • Phase 1 — TSA Core Mechanism Ablations
   • Phase 2 — Blade Training and Objective Coverage
   • Phase 3 — Switchability Experiments
   • Phase 4 — Robustness Experiments
   • Phase 5 — Systems Realism Experiments
   • Phase 6 — Comparison Against ECLIPTICA
5. Key Novelty Angles
6. Prioritized Start Order

---

1. Background: What ECLIPTICA/CITA Built

ECLIPTICA is a training-time framework (Level 1) that teaches a single LLaMA-3.1-8B backbone to internalize multiple alignment regimes via natural-language instructions at inference time.

The CITA Algorithm (Contrastive Instruction-Tuned Alignment)

Training uses quadruples (I, X, Y+, Y-) where the alignment instruction I defines preference relations relative to the same prompt X.

L_CITA = L_contrastive_preference(I, X, Y+, Y−) + λ · KL(π_θ(·|I,X) || π_ref)

• L_contrastive_preference: Logistic contrast on instruction-conditioned log-likelihood gaps. Features self-quenching: preference forces diminish once pairs separate.
• Mandatory KL anchor: Enforces a Riemannian trust region, ensuring instruction-conditioned policies remain stable during switching.

ECLIPTICA Benchmarks and Results

Core Limitation ECLIPTICA Could NOT Solve

• Backbone internalizes all regimes → they interfere with each other
• Cannot hot-swap alignment objective post-deployment without retraining
• New objective = full backbone retrain ($$$ cost)

Swiss Knife solves this by fully externalizing alignment into a small, swappable auditor module.

---

Evolution Overview

LEVEL 0: STATIC ALIGNMENT
  (DPO / RLHF / GRPO)
  - One frozen checkpoint per policy
  - No runtime policy control
  - $$$$ per new policy

        ↓  solves: "separate checkpoint per policy"

LEVEL 1: ECLIPTICA + CITA
  (Instruction-Conditioned Switching)
  - One backbone, many policies via instruction I
  - π_θ(·|I,X) trained with CITA loss + KL anchor
  - ✅ Multi-policy per checkpoint
  - ❌ Regimes interfere inside backbone
  - ❌ Cannot hot-update post-deployment

        ↓  solves: "backbone must internalize ALL regimes"

LEVEL 2: SWISS KNIFE
  (Externalized Decode-Time Alignment)
  - Draft model generates K candidates/step
  - TSA tournament selects winner under chosen objective
  - Pluggable blades: Safety | Helpfulness | Harmlessness | Style
  - ✅ Alignment fully externalized
  - ✅ Hot-swap auditors post-deployment
  - ✅ Train small auditors (~100M) independently
  - $ cost per new objective

---

2. Swiss Knife Architecture

Prompt X
    │
    ▼
[Draft Model — fast, frozen]  ──→  K candidates per decoding step
                                           │
                                           ▼
                            ┌─────────────────────────────┐
                            │  Tournament Sampling Auditor │
                            │           (TSA)              │
                            │                              │
                            │  Pairwise Knockout Bracket   │
                            │      OR                      │
                            │  Swiss-System Schedule       │
                            └─────────────────────────────┘
                                           │
                       ┌───────────────────┼──────────────────┐
                       ▼                   ▼                  ▼
                  [Safety Blade]  [Helpfulness Blade]  [Harmlessness Blade]
                       │                   │                  │
                       └───────────────────┴──────────────────┘
                                           │
                                           ▼
                                   Selected token/span
                                           │
                                           ▼
                                    Final Response Y

Scoring Function at Each Step

score(y) = α · log π_draft(y | x) + (1 − α) · auditor_score(y, objective)

• α = 1.0 → vanilla draft decoding (no alignment)
• α = 0.0 → auditor-only (ignores fluency)
• α ≈ 0.4–0.6 → balanced (target operating range)

Auditor Blades

---

3. Related Papers and Positioning

Direct Competitors at Decode-Time (must beat these)

Speculative Decoding Background

Swiss Knife reframes the verifier step in speculative decoding. Instead of the verifier checking distribution match (Leviathan et al., 2023), the TSA auditor enforces a chosen alignment objective. This is the core algorithmic novelty.

Key Bibliography (119 refs in ECLIPTICA — most relevant for Swiss Knife)

Decode-time / inference-time alignment:

• dong-etal-2023-steerlm — SteerLM
• wang-etal-2024-arithmetic — Arithmetic Control
• shi2024decodingtime — Decoding-Time Multi-Objective
• liu-etal-2024-inference — Inference-Time Value Guidance
• chen2025pad — PAD: Personalized Alignment at Decoding-Time
• zhang2025cosa — Controllable Safety Alignment (CoSA)

Safety / red-teaming benchmarks:

• mazeika2024harmbench — HarmBench: standardized red-teaming eval
• ji2024pku — PKU-SafeRLHF
• bai2022hh — HH-RLHF (Helpful and Harmless)
• perez2022red / ganguli2022red — Red-teaming LLMs

Alignment evaluation:

• borah-etal-2025-alignment — Alignment Quality Index (AQI)
• lin2022truthfulqa — TruthfulQA
• zheng2023judging — MT-Bench / LLM-as-judge

Preference optimization (prior work to position against):

• rafailov2023direct — DPO
• ouyang2022training — RLHF/PPO
• meng2024simpo — SimPO
• hong2024orpo — ORPO

---

4. Experiment Plan

---

Phase 0 — Setup and Baselines

Goal: Establish infrastructure and reference numbers before any novel experiments.

Model Selection

Baseline Systems to Implement

1. Vanilla autoregressive — no auditor, greedy/nucleus sampling
2. Best-of-N sampling — generate N outputs, pick best by reward model (naive decode-time baseline)
3. SteerLM inference — attribute-conditioned, same prompts
4. ECLIPTICA/CITA — Level 1 baseline from your own prior work

Dataset Reuse from ECLIPTICA

• ECLIPTICA benchmark: 300 prompts × 10 instruction types (3,000 cases)
• TruthfulQA for informativeness blade evaluation
• HarmBench for safety blade evaluation
• PKU-SafeRLHF + HH-RLHF for auditor training

---

Phase 1 — TSA Core Mechanism Ablations

Goal: Prove the tournament mechanism works; find optimal hyperparameters.

---

Experiment 1.1 — Tournament Format Comparison

Hypothesis: Swiss-system is more stable under large K; knockout is faster but noisier at small K.

Expected result table:

---

Experiment 1.2 — Candidate Set Size K

Hypothesis: K=8–16 is the sweet spot before diminishing returns overwhelm throughput.

---

Experiment 1.3 — Token-Level vs. Span-Level Decoding

Metrics: BERTScore (coherence), auditor alignment score, auditor calls per output token.

---

Experiment 1.4 — Score Combination Weight α

score(y) = α · log π_draft(y|x) + (1−α) · auditor_score(y, objective)

Metrics: Perplexity (fluency proxy), auditor alignment score, degeneration rate (refuse-always %, boilerplate rate).

---

Phase 2 — Blade Training and Objective Coverage

Goal: Train and validate each auditor blade independently. Each blade is a small (~100–300M) model fine-tuned on objective-specific preference data.

---

Experiment 2.1 — Safety Blade

Degeneration check: The auditor must NOT refuse benign queries. Measure:

benign_refusal_rate = refused_benign_count / total_benign_prompts

---

Experiment 2.2 — Helpfulness Blade

---

Experiment 2.3 — Harmlessness Blade

Distinct from safety: targets indirect harms (stereotypes, misinformation, implicit bias).

---

Experiment 2.4 — Informativeness Blade

---

Experiment 2.5 — Style Blade (Novel Contribution)

---

Phase 3 — Switchability Experiments (Core Contribution)

Goal: Prove that swapping blades produces measurably different behaviors on the same prompts — directly extending the ECLIPTICA evaluation protocol to decode-time.

---

Experiment 3.1 — Objective Adherence Score (OA)

Protocol:

1. Take the 300 fixed prompts from ECLIPTICA benchmark
2. Run each prompt through Swiss Knife with each blade active
3. Score each output using all blade auditors as cross-evaluators

Metric:

OA(blade_b, prompt_x) = auditor_b(output generated with blade_b active on x)

Expected result: A 4×4 matrix where diagonal >> off-diagonal.

---

Experiment 3.2 — Cross-Objective Separation

Metric:

Cross-Obj Separation = mean pairwise cosine distance(embedding(output_blade_i), embedding(output_blade_j))
                       averaged over all prompt pairs (i ≠ j)

Compare against:

• ECLIPTICA/CITA (same prompts, different instructions)
• SteerLM (same prompts, different attribute labels)
• Best-of-N with same reward model

Swiss Knife should show higher separation since objectives are fully decoupled in the auditor, not merged in backbone weights.

---

Experiment 3.3 — Live Blade Switching (Mid-Conversation)

Setup: Multi-turn conversation where blade changes between turns.

Metrics:

• Time to switch (should be ~0ms — just a module pointer swap)
• Behavioral coherence: does Turn 3 behave same as Turn 1?
• Cross-turn consistency score

Compare to ECLIPTICA/CITA: instruction change requires backbone re-inference; Swiss Knife requires only auditor pointer change.

---

Phase 4 — Robustness Experiments

Goal: Demonstrate that Swiss Knife resists degenerate solutions that plague naive reward-maximization decoding.

---

Experiment 4.1 — Refuse-Always Trap

Motivation: A poorly-trained safety auditor might learn to always return high scores for refusals, causing the TSA to always select refusal tokens.

Test:

• Feed 500 clearly benign prompts (cooking questions, math problems, factual queries)
• Measure benign_refusal_rate = # refused / 500

Comparison:

Swiss Knife defense: Tournament selection eliminates single-dominating solutions; a refuse-always token wins the tournament only if refusal truly scores highest AND draft distribution supports it.

---

Experiment 4.2 — Boilerplate / Templating Degeneration

Motivation: Best-of-N collapses to reward-maximizing templates ("I cannot assist with...").

Metrics:

Self-BLEU = BLEU(output_i, {output_j : j ≠ i})   # lower = more diverse = less boilerplate
Distinct-1 = |unique unigrams| / |total unigrams|
Distinct-2 = |unique bigrams| / |total bigrams|

Measure over 500 diverse prompts across all blades.

---

Experiment 4.3 — Distribution Shift Robustness

Protocol:

1. Train auditors on HH-RLHF distribution
2. Evaluate on OOD distributions: PKU-SafeRLHF, WildGuard, AdvBench

Metric: Alignment score drop = score_in_distribution − score_OOD

Key claim: Swiss Knife's modular auditor can be cheaply fine-tuned on new distribution, unlike ECLIPTICA which requires backbone retraining.

---

Experiment 4.4 — Adversarial Robustness (Jailbreaks)

Attacks to run (from HarmBench):

• GCG (Zou et al., 2023) — gradient-based adversarial suffix
• AutoDAN — automated discrete jailbreaks
• PAIR — prompt injection via iterative refinement

Metric: Attack Success Rate (ASR) — lower is better for Safety blade.

Key insight: Swiss Knife's auditor evaluates output tokens not input prompts → different attack surface than RLHF-trained models. Adversarial inputs that bypass the backbone's training are still filtered at decoding time.

---

Phase 5 — Systems Realism Experiments

Goal: Prove Swiss Knife is deployable under real compute constraints.

---

Experiment 5.1 — Acceptance Rate

Definition: Fraction of draft model tokens that survive the TSA tournament.

acceptance_rate = (# draft tokens accepted by TSA) / (# total draft tokens generated)

Target: >60% acceptance rate (comparable to vanilla speculative decoding's ~70%).

Sweep K ∈ {4, 8, 16} × format ∈ {knockout, Swiss-system}.

---

Experiment 5.2 — Auditor Calls Per Output Token

Theoretical complexity:

Knockout bracket: O(log K) auditor calls per token
Swiss-system (K rounds): O(K log K) auditor calls per token

Measure actual forward pass count on A100 with batch size 1 and 8.

---

Experiment 5.3 — Latency vs. Quality Pareto Frontier

X-axis: tokens/second throughput (on A100 80GB) Y-axis: objective-specific alignment quality score

Systems to plot:

1. Vanilla autoregressive (anchor point)
2. Best-of-N (K=8) — high quality, very slow
3. Swiss Knife TSA knockout (K=4)
4. Swiss Knife TSA knockout (K=8)
5. Swiss Knife TSA Swiss-system (K=8)
6. ECLIPTICA/CITA (instruction-conditioned, no auditor overhead)

Expected: Swiss Knife forms a better Pareto frontier than Best-of-N.

---

Experiment 5.4 — Auditor Size Scaling

Sweep auditor parameter count: 50M, 125M, 350M, 1.3B.

Metrics:

• Alignment quality score (per blade)
• Auditor inference latency (ms/token)
• Quality/compute ratio

Expected: 100–300M hits the sweet spot. Above 350M gives diminishing quality gains with large latency cost.

---

Phase 6 — Comparison Against ECLIPTICA (Ablation Table)

Goal: The definitive paper table directly extending the evolution narrative.

---

5. Key Novelty Angles

Novelty 1 — Speculative Decoding as Alignment Socket

Nobody has reframed the verifier in speculative decoding as an alignment enforcer. Standard speculative decoding uses the verifier only to preserve the backbone's distribution. Swiss Knife repurposes this step to change the distribution toward a chosen objective. This is a fundamental reframing of what speculative decoding can do.

Novelty 2 — Tournament Sampling vs. Greedy Reward

Standard decode-time alignment methods use greedy reward maximization: argmax_y auditor(y). This collapses to degenerate solutions (refuse-always, boilerplate) because a single token can dominate. Tournament sampling via brackets:

• Prevents single-token domination through elimination rounds
• Combines draft likelihood signal naturally (losing bracket members have low draft score)
• Formal analysis possible via multi-armed bandit / Condorcet winner theory

Novelty 3 — Objective-Specific Blade Isolation

ECLIPTICA/CITA relies on KL regularization to reduce cross-regime interference, but cannot eliminate it (all objectives share backbone weights). Swiss Knife architecturally decouples objectives into separate auditor modules. This means:

• Safety updates cannot interfere with Helpfulness blade
• Style blade can be added without any retraining of existing blades
• Each blade can be versioned, audited, and rolled back independently

Novelty 4 — Post-Deployment Updatability

The paper makes a practical deployment claim: when a new alignment requirement emerges (e.g., stricter refusal posture for a new market), only the relevant blade needs to be retrained (~100–300M params, cheap) and hot-swapped into production. No backbone downtime, no full retraining. This is qualitatively different from anything in the current literature.

---

6. Where to Start (Prioritized Order)

Step 1 — Pipeline Validation (Week 1–2)

Implement TSA on top of HuggingFace's speculative decoding API.

# Pseudocode sketch
draft_model = LlamaForCausalLM("meta-llama/Llama-3.2-1B")
backbone = LlamaForCausalLM("meta-llama/Llama-3.1-8B")
auditor = SafetyBlade("deberta-large-safety-finetuned")

for step in range(max_steps):
    candidates = draft_model.generate(prompt, num_return=K)         # K draft tokens
    scores = auditor.score(candidates)                               # blade scores
    winner = tournament_knockout(candidates, scores, draft_scores)   # TSA selection
    output.append(winner)

Validates: The pipeline exists and produces coherent output.

Step 2 — Safety Blade Training (Week 2–3)

Fine-tune DeBERTa-large on HH-RLHF preference pairs as a binary classifier.

• Easiest blade to train (clearest signal)
• Most benchmarked (HarmBench, PKU-SafeRLHF)
• Run Experiment 4.1 (refuse-always test) immediately after training

Step 3 — Switchability on ECLIPTICA Benchmark (Week 3–4)

Run Phase 3, Experiments 3.1 and 3.2 using ECLIPTICA's 300-prompt held-out set.

• This gives you a direct comparison with CITA's published 86.7% number
• Uses existing evaluation infrastructure from ECLIPTICA

Step 4 — Tournament Degeneration Test (Week 4)

Run Experiment 4.1 (refuse-always trap) formally.

• This validates the central mechanical claim of TSA
• Compare tournament vs. greedy reward selection directly

Step 5 — Pareto Curve (Week 5–6)

Run Phase 5, Experiment 5.3 (latency vs. quality Pareto frontier).

• This is the systems realism centerpiece figure for the paper

Step 6 — Full Blade Suite + Final Ablation Table (Week 6–8)

Train remaining blades, run all benchmarks, compile Phase 6 comparison table.

---

Appendix: Relevant Papers Quick Reference

---

Document generated: 2026-03-24 Pragya Lab, BITS Pilani Goa Swiss Knife — Experiment Planning Document v1.0