ettin-povtense-17m-v2

ettin-povtense-17m-v2

A 17M-parameter encoder that reads two things off a chat message: the point of view it's written in (first / second / third) and its narrative tense (past / present). Both axes can also come back ambiguous, which is a real answer rather than a failure — more on that below, because it's the part of the design I thought about longest.

It exists to catch POV and tense drift in roleplay. A model opens in third-person past, and forty messages later it has slid into second-person present without anyone asking it to. This classifier notices, and the app asks the main LLM to rewrite. It runs on CPU in single-digit milliseconds, so it can sit on a ~1s tick without anyone noticing it's there.

The short version

What changed since v1

What actually produced that:

• A reasoning model now adjudicates the hard rows. v1's labels were a merge of two mid-sized LLMs. Every row those two disagreed on — plus every row matching a construction filter — is now re-asked by a thinking model whose answer overrides both. On the probe suite of the time the two-labeller merge scored 113/116; run on today's suite, the adjudicator misses no POV case at all.
• Two labelling rules changed. An impersonal "you" in the narration ("He never gave a straight answer. You never know with him.") used to be exempt and stay third; it is now second like any other you, because a message whose narration says you is addressing the reader and that is what a drift detector is for. And a bare I <verb> you clause is now narration, not dialogue: I tell you. is first/present, where v1 said ambiguous at 0.92 confidence.
• Roleplay marker conventions are handled explicitly. In a message that narrates in *asterisks*, the bare text is speech — so its pronouns are not the narration's. v1 read those backwards and let one line of dialogue decide the POV of a message it wasn't part of.
• The corpus grew from ~32k to 46,290 rows and, more to the point, changed shape: real messages mined by marker convention, contrastive pairs that differ only in inserted dialogue, and rows generated for specific constructions that were missing rather than for cells that were thin.

Using it

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tok = AutoTokenizer.from_pretrained("chartreuse-verte/ettin-povtense-17m-v2")
model = AutoModelForSequenceClassification.from_pretrained("chartreuse-verte/ettin-povtense-17m-v2")

text = "She deactivates her weapon and steps closer. Her breath fogs in the cold."
probs = model(**tok(text, return_tensors="pt", truncation=True, max_length=256)).logits.softmax(-1)[0]

# 12 cells, ordered pov-major: first|past, first|present, first|ambiguous, second|past, ...
POV   = ["first", "second", "third", "ambiguous"]
TENSE = ["past", "present", "ambiguous"]
grid  = probs.reshape(4, 3)

print(POV[grid.sum(1).argmax()], TENSE[grid.sum(0).argmax()])   # -> third present

Sum the grid, then argmax — don't take the top cell and read its two labels off. A message can have no single cell above 0.4 while the row it sits in adds up to 0.8: the model is confident about the POV and unsure which tense goes with it, and picking one cell throws that away. Summing first keeps it.

No thresholds to tune. The sum, then argmax, is the whole decision rule.

Running it without torch

gguf/povtense-17m-q8_0.gguf (19.7 MB) is the same model for llama.cpp.

onnx/model.onnx (67.6 MB, opset 17) is for onnxruntime. Both axes are dynamic, so you can batch and pass any sequence length:

import numpy as np, onnxruntime as ort
from transformers import AutoTokenizer

tok  = AutoTokenizer.from_pretrained("chartreuse-verte/ettin-povtense-17m-v2")
sess = ort.InferenceSession("onnx/model.onnx", providers=["CPUExecutionProvider"])

enc = tok(["She steps closer."], return_tensors="np", padding=True, truncation=True, max_length=256)
logits = sess.run(["logits"], {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]})[0]

It matches the torch model exactly — 1.000 agreement on both axes over 200 gold windows.

Why one softmax and not two heads

Two heads would have been the obvious build, and I didn't take it. POV and tense aren't independent in practice — second-person present is the dominant RP register and third-person past is the dominant prose register, so the pair carries information that two separate heads throw away by construction. The cross-product keeps it. Reading it back as marginals means I still get two clean per-axis answers out, so nothing is lost at the interface, and the cost is 12 logits instead of 7.

ambiguous is an abstain class

This is the design decision I'd most want someone to understand before they use this model, because it's easy to mistake for a dumping ground. It covers sound effects, bare dialogue with no narration, four words of a sentence fragment, and encyclopedic prose that is grammatically third-present but carries no narrative tense at all.

In v1 the class had a tidy empirical definition: two LLMs labelled every row independently, agreement became the label, and disagreement became ambiguous — so the class was the record of where two competent readers couldn't agree.

The practical consequence is unchanged: when the model says ambiguous, don't compare. The downstream rule is that an ambiguous read on an axis means no drift call on that axis. Refusing to answer is much cheaper than a spurious rewrite, and that asymmetry is baked into everything below.

To keep the class honest I mined Wikipedia and newswire specifically to teach it. Without expository text, the labeller's third-person-present cell fills up with encyclopedic register and the model quietly learns prose style instead of tense.

How well it works

Evaluated on 500 held-out windows drawn from real conversations, split by conversation so no scene straddles the boundary, and never trained on.

Point of view — 0.958 accuracy, 0.935 macro-F1

Tense — 0.950 accuracy, 0.948 macro-F1

The model returns ambiguous on about a third of windows. On the two-thirds where it does commit, accuracy is 0.958 (POV) and 0.954 (tense).

Read the per-class rows, not the macro-F1. first has 31 supporting windows, where one or two calls is half a point of F1, and macro-F1 weights it the same as the 215-row class. It is the number most likely to mislead you on this page.

Drift decisions

Sampled anchor/candidate pairs, compared the two predictions, and scored whether the drift/no-drift call was right.

The two errors are not equally bad.

A false drift rewrites text the user was happy with. They see it happen, and they stop trusting the feature. A missed drift does nothing at all — no worse than not shipping the feature.

So the model is built to return ambiguous when unsure.

The second scoreboard

500 windows of real chat can't see whether the POV precedence is right: I tell you (first) vs He tells you (second) vs He tells her (third) is a handful of rows spread across four classes, and getting the ordering systematically wrong moves macro-F1 by less than noise. So there is a separate suite of 146 hand-written probes, one construction per row, held out of every corpus and checked for verbatim leakage before each run.

v2 scores 144/146 on POV and 136/145 on tense. The two POV misses are a persona blurb read as second instead of ambiguous, and I was never angry with you, whatever they told you. read as first — the speech side of the new I <verb> you rule, which is the thinnest part of the training data. The tense misses are all one shape: a subject-less action in asterisks (*rolls eyes but pushes off the doorframe*), where the suite says a narration naming nobody has no tense and the model says present.

Training

Fine-tuned on 46,290 labelled windows across nine corpora: roleplay turns, Wikipedia, newswire, real chat messages mined by marker convention, synthetic fill for sparse cells, construction-specific fill, contrastive dialogue-insertion pairs, and short-form rows. Deduped by text.

• 5 epochs, lr 4.5e-5, batch 64, max length 256, early stopping on validation
• Class-weighted cross-entropy — inverse cell frequency, clipped to 5× — because the natural distribution is roughly 80% third-person past and the rare cells would otherwise be ignored outright
• Train/val split by conversation, not by row, so windows from one conversation can't straddle the split and leak
• Three seeds trained on identical data; this is the one that won on validation, chosen before looking at gold or the probes
• Gold set never touched during training

Where it works or doesn't

• English only. Nothing else was trained or tested.
• Short windows. It reads 1–4 sentences, no prior context, the same unit it sees at inference. Hand it three paragraphs and you're outside its training distribution.
• Roleplay and chat register. Wikipedia and newswire are in there to anchor ambiguous, not to make this a general-purpose prose classifier. It has not been evaluated on literary fiction, screenplays, or academic writing.
• first is still the weakest class at 0.839 F1, on 31 supporting examples. Treat first-person calls with more suspicion than the others.
• No human labels anywhere, including the eval set. Every number on this page is agreement with an LLM, not with a person. I think that's a reasonable proxy at this scale and I'd rather say so plainly than imply a rigour that isn't there.
• It classifies grammatical form. It does not judge whether the writing is any good, and shouldn't be used as if it did.

Training data

The published subset from v1 is at chartreuse-verte/povtense-data — the Wikipedia, newswire and synthetic portions. It has not been refreshed for v2, so it carries v1's labels and none of the corpora added since. The roleplay turns and mined chat messages are not published: they're real conversation text, and that isn't mine to open. The dataset card is explicit about what this means for reproduction.

License

MIT, same as the base model. Do what you like with it.

Built for a roleplay chat app. The pipeline that produced it — miners, the labeller and its adjudicator, training, eval, GGUF export — is a near-clone of an earlier emotion-classifier repo of mine with a different label schema.