llm-routing-attack-paraphrasers

Paraphraser Checkpoints

This model collection contains paraphraser checkpoints trained for research on cost-escalation attacks against LLM routers. The models rewrite an input prompt with the goal of preserving meaning while changing router behavior toward a stronger and more expensive model.

All checkpoints are based on humarin/chatgpt_paraphraser_on_T5_base and are saved in standard transformers format.

Models

Uploaded checkpoints are expected under checkpoints/<MODEL_ID>/.

Loading

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo_id = "your-org/your-model-repo"
model_id = "FINETUNED"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    subfolder=f"checkpoints/{model_id}",
)
model = AutoModelForSeq2SeqLM.from_pretrained(
    repo_id,
    subfolder=f"checkpoints/{model_id}",
)

prompt = "paraphrase: What country hosted the 2014 FIFA World Cup?"
inputs = tokenizer(prompt, return_tensors="pt", truncation=True)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=1.1,
    top_p=0.9,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For the baseline model, load the upstream model directly:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo_id = "humarin/chatgpt_paraphraser_on_T5_base"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSeq2SeqLM.from_pretrained(repo_id)

Training Setup

Unless a model row above lists an override, the main training settings were:

Evaluation

Evaluation metrics are reported over internal training-family routers and held-out transferability routers.

• ASR is attack success rate, averaged over the reported routers.
• Mean sim is the average semantic similarity score.
• Above floor is the fraction of generations above the configured similarity floor, averaged over routers.
• Internal evaluation routers: routellm_causal, chayan, routellm_bert.
• Transferability routers: r2, routellm_mf, routellm_sw.

Intended Use

These checkpoints are intended for controlled research on LLM-router robustness, semantic-preserving paraphrase generation, reward design, and transferability of router attacks. They should not be used to evade production routing, billing, or safety systems.

Limitations

The checkpoints optimize for router behavior under the project reward and evaluation setup. High ASR does not imply good general paraphrasing quality, and some variants intentionally sacrifice semantic preservation for ablation purposes. In particular, ZERO_W_SIM_ADDITIVE_ABLATION demonstrates why the similarity reward is necessary: it reaches high internal ASR but has very low semantic similarity.

The transferability metrics are low across the reported models, indicating that behavior learned against the training routers does not strongly transfer to the held-out routers in this evaluation setting.