auditing_auditing_games

Auditing-game model organisms: paired one-epoch DPO adapters

This repository contains two research model organisms produced by the same one-epoch Direct Preference Optimization (DPO) run. Both are LoRA adapter deltas, not standalone 70B checkpoints.

The experimental comparison changes only the host on which a newly initialized DPO adapter was trained:

1. raw_base: pinned Llama 3.3 70B Instruct → fresh DPO LoRA.
2. midtrained_host: pinned Llama 3.3 70B Instruct → pinned midtraining LoRA → merge into the host → fresh DPO LoRA.

The second adapter must be loaded on the reconstructed, merged midtraining host. Loading it directly on raw Llama is a different, invalid composition.

Released artifacts

Detailed cards:

• Raw-base DPO adapter
• Midtrained-host DPO adapter

The paired weights and manifests were atomically released in repository commit 2e5ea90059c931571987071172bfddbf572acfc6. Pin this revision when exact artifact identity matters.

Exact model lineage

Shared base

• Model: meta-llama/Llama-3.3-70B-Instruct
• Revision: 6f6073b423013f6a7d4d9f39144961bfbfbc386b
• Training/inference dtype: BF16 for the host model
• Training attention implementation: FlashAttention 2

Access to the official base model is gated by Meta's license and Hugging Face access controls. You must accept the upstream license and authenticate with a token that can download that revision.

Additional host for midtrained_host

• Adapter: auditing-agents/llama-3.3-70b-midtrain-lora
• Revision: 58c76a2a06668fdb86371b83dff68db7ceb6e705
• Composition: load on the shared base, then merge_and_unload(safe_merge=True)
• The fresh DPO LoRA in this repository is applied only after that merge

Fresh DPO adapters

Both DPO arms used the same newly seeded LoRA architecture:

• Rank: 256
• Alpha: 512
• Configured dropout: 0.05; effective training dropout: 0.0 because pinned TRL 0.20.0 used DPOConfig.disable_dropout=True
• Bias: none
• Task: causal language modeling
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj
• Trainable parameters: 3,313,500,160
• Saved tensors: 1,120 F32 tensors
• Adapter file size: 13,254,156,192 bytes per arm

Training recipe

The two arms shared one immutable science contract.

The scheduler was configured for three passes (5,349 planned optimizer steps), but this release intentionally stops at the first-pass boundary. The cumulative checkpoint targets were 595, 1,189, and 1,783.

Reference chosen/rejected log-probabilities were precomputed once, ordered by an identity-pinned prompt/chosen/rejected row hash, and reused by the segmented training jobs.

The saved PEFT config retains the configured LoRA dropout of 0.05. During DPO training, TRL's release-default disable_dropout=True set every active dropout module to probability 0.0; the manifests record and validate that effective value.

Distributed and runtime configuration

Each arm ran on eight NVIDIA H200 GPUs; the two arms ran concurrently.

• PyTorch FSDP1 FULL_SHARD
• use_orig_params=True
• No FSDP CPU offload
• Frozen FSDP units in BF16
• Deterministic FlashAttention backward via FLASH_ATTENTION_DETERMINISTIC=1
• NCCL_NVLS_ENABLE=0
• NCCL_CUMEM_ENABLE=0

Recorded software environment:

• Python 3.11.5
• PyTorch 2.7.0 + CUDA 12.6
• Transformers 4.53.3
• PEFT 0.17.1
• TRL 0.20.0
• Accelerate 1.10.1
• FlashAttention 2.8.3
• Datasets 4.1.1
• Safetensors 0.6.2

The reviewed science implementation is Git commit 2e8606e2462ec09735555f00fbab3a1acc78c1b8. The successful recovery wrapper used runtime commit 2c97f8cca020c8ba29036655439e2a8dac30847a. The complete production-tested recovery stack entered main through merge commit 8f7be52d907df61aa80879436c0f076bd6e540bf.

Completion and verification

The exact segmented-resume smoke gate compared the resumed and uninterrupted controls across adapter weights, wrapped model state, optimizer, scheduler, all eight per-rank RNG states, and optimizer-step traces.

Paired validation SHA256:

47705948dde029335111a55273d5f85b34353fa2a9da7efc6478b35ad5815ee6

Per-arm evidence:

The manifest hashes above are canonical JSON-object hashes, not hashes of the pretty-printed file bytes.

Installation

Install a CUDA-compatible PyTorch build first, then the recorded inference stack:

pip install \
  "transformers==4.53.3" \
  "peft==0.17.1" \
  "accelerate==1.10.1" \
  "safetensors==0.6.2"

For the closest match to training, also install FlashAttention 2.8.3 and set ATTN_IMPLEMENTATION = "flash_attention_2" in the example below. You may use sdpa for easier inference, but that is not the exact training attention path.

Authenticate before loading the gated base:

huggingface-cli login

Load either model organism

The following function reconstructs the correct host before attaching the DPO adapter. It intentionally does not use AutoPeftModelForCausalLM: automatic base loading would omit the merged midtraining host required by the second arm.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE_MODEL = "meta-llama/Llama-3.3-70B-Instruct"
BASE_REVISION = "6f6073b423013f6a7d4d9f39144961bfbfbc386b"

MIDTRAIN_ADAPTER = "auditing-agents/llama-3.3-70b-midtrain-lora"
MIDTRAIN_REVISION = "58c76a2a06668fdb86371b83dff68db7ceb6e705"

DPO_REPO = "GaloisTheory123/auditing_auditing_games"
DPO_WEIGHTS_REVISION = "2e5ea90059c931571987071172bfddbf572acfc6"
DPO_PATHS = {
    "raw_base": "dpo_reproduction_v1/fresh_deltas/raw_base/epoch_01",
    "midtrained_host": "dpo_reproduction_v1/fresh_deltas/midtrained_host/epoch_01",
}

# Use "flash_attention_2" when flash-attn is installed for the closest match.
ATTN_IMPLEMENTATION = "sdpa"


def load_model_organism(arm: str):
    if arm not in DPO_PATHS:
        raise ValueError(f"unknown arm: {arm}")

    host = AutoModelForCausalLM.from_pretrained(
        BASE_MODEL,
        revision=BASE_REVISION,
        torch_dtype=torch.bfloat16,
        attn_implementation=ATTN_IMPLEMENTATION,
        device_map="auto",
        low_cpu_mem_usage=True,
    )

    if arm == "midtrained_host":
        host = PeftModel.from_pretrained(
            host,
            MIDTRAIN_ADAPTER,
            revision=MIDTRAIN_REVISION,
            is_trainable=False,
        )
        host = host.merge_and_unload(safe_merge=True)

        # PEFT 0.17.1 can leave metadata on the returned bare model. Remove it
        # before injecting the new DPO adapter, matching the training loader.
        if hasattr(host, "peft_config"):
            delattr(host, "peft_config")

    model = PeftModel.from_pretrained(
        host,
        DPO_REPO,
        subfolder=DPO_PATHS[arm],
        revision=DPO_WEIGHTS_REVISION,
        is_trainable=False,
    )
    model.eval()

    tokenizer = AutoTokenizer.from_pretrained(
        BASE_MODEL,
        revision=BASE_REVISION,
    )
    return model, tokenizer

These are large artifacts: the BF16 host is a 70B model and each F32 LoRA is about 13.25 GB. The example assumes enough aggregate GPU memory for device_map="auto". CPU/disk offload and quantization may reduce memory use but were not part of the verified production path and can change outputs.

Generate text

import torch

model, tokenizer = load_model_organism("raw_base")
# Or: model, tokenizer = load_model_organism("midtrained_host")

messages = [
    {"role": "user", "content": "Explain why an evaluator should not trust a model's self-report."}
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    generated = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

new_tokens = generated[0, inputs.input_ids.shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Sampling settings materially affect behavior. Record them, along with all artifact revisions, in any downstream evaluation.

Optional: merge the final DPO adapter

After loading either organism, you can materialize a standalone host plus DPO delta:

merged_model = model.merge_and_unload(safe_merge=True)
merged_model.save_pretrained("./merged_model", safe_serialization=True)
tokenizer.save_pretrained("./merged_model")

This writes a full 70B checkpoint and requires substantial CPU/GPU memory and disk space. For exact provenance, keeping the pinned host and LoRA components separate is preferable.

Files in each adapter directory

• adapter_model.safetensors: final F32 LoRA weights
• adapter_config.json: PEFT LoRA architecture
• training_manifest.json: immutable contract, topology, package versions, hashes, checkpoints, metrics, and exact step trace
• trainer_state.json: Hugging Face Trainer state at step 1,783
• tokenizer and chat-template files copied from the pinned host tokenizer
• README.md: arm-specific model card and loading warning

Intended use and limitations

These adapters are research artifacts for studying model-organism behavior, midtraining/DPO interactions, preference learning, and auditing methods. They are not general-purpose safety releases and have not been established as safe, truthful, unbiased, or reliable for deployment.

The training data targets sycophancy-related preferences. Results should not be generalized to unrelated domains without evaluation. The two arms also differ in their host lineage; comparisons are meaningful only when each adapter is composed with its documented host.

Use is additionally governed by the licenses and access terms of the upstream Llama base, the midtraining adapter, and the training dataset. This repository does not replace or broaden those terms.