Llama-3.1-breakingnews

Llama-3.1-breakingnews

A LoRA adapter on Llama-3.1-8B-Instruct that locates story boundaries in broadcast-news transcripts. Given a transcript as one long string, it returns the word offsets where one story ends and the next begins. A transcript arrives as an undifferentiated block of 2,000–17,000 words covering several unrelated stories; content analysis needs a comparable unit, and a television transcript is not one article. A boundary means the broadcast moves to a genuinely different story — new topic, new event, different actors — and explicitly not a change of speaker, correspondent, location or sub-angle within a continuing story.

Boundaries only. This adapter does not label, classify or summarise the segments it produces.

Use it

pip install "breakingnews[gpu]"

from breakingnews import Segmenter

seg = Segmenter.from_pretrained("sakshib3/Llama-3.1-breakingnews", revision="v1")
breaks = seg.segment(transcript)         # [909, 1333, 2351, ...]
spans = seg.segment_spans(transcript)    # [(0, 909), (909, 1333), ...]

Offsets index transcript.split(). The package also turns boundaries into one row per story and merges the rows back byte-for-byte: github.com/sakshi-bhalla/breakingnews.

Loading this adapter also downloads Llama-3.1-8B-Instruct (~16 GB) and needs a GPU with at least 24 GB. Pin revision; unpinned resolves to main, which moves.

Accuracy

τ (tau) is the decision threshold. For every window the model emits a probability that a story boundary is present; τ is the cut-off above which that window's boundaries are kept. τ = 0.010, selected on validation and applied unchanged to the held-out test set.

Quote the validation row: it rests on 322 boundaries against the test split's 64, where the standard error on recall alone is ≈0.05. Precision is a lower bound, not an estimate — many scored false positives are real topic changes grouped into one thematic block, so do not compute a derived statistic that treats a false positive as clean error.

A prediction counts as correct if it lands within N words of a true boundary: ±25 asks "to within a sentence?", ±100 asks "did it find the seam at all?". Baselines on the same test set are 0.000 for predicting nothing and 0.062 for predicting N boundaries at uniform spacing.

τ is not a tuning knob. The confidences are saturated and bimodal — 49% of validation windows above 0.5, 38% below 0.001 — so any value in roughly [0.005, 0.5] gives the same answer, and it exists only to exclude τ = 0, where every window fires. This geometry has no high-precision regime: it cannot exceed precision 0.564 at any threshold, so if your use is sensitive to false boundaries the fix is a different geometry, not a different threshold.

Training

Trained on 998 annotated transcripts containing 2,829 story boundaries, sampled from US TV news broadcasts 1992–2020 (CNN, FOX, MSNBC, ABC, CBS). The transcripts are licensed and are not distributed; the annotations are word offsets carrying no text, available for review on request.

The adapter is rank 16, alpha 32, on all attention and MLP projections, plus a trained <|STORY_BREAK|> token in both embed_tokens and lm_head. Window geometry — 3072-token windows at 1536 stride — travels with the adapter in segmentation_config.json and is read at load time; running at a different window size silently costs accuracy.

Citation

Sakshi Bhalla (0000-0002-6803-7657). See CITATION.cff in the source repository.

License

Governed by the Llama 3.1 Community License, whose terms pass through to anyone using these weights; a copy ships as LICENSE in this repository alongside the required NOTICE. The accompanying Python package is MIT. Built with Llama.