meeting-summarizer-dev

bart-base-samsum-summarizer

facebook/bart-base fine-tuned on the SAMSum dialogue summarization corpus.

Note: Front-matter ROUGE scores reflect the champion decoding config (D27: beam=5, length_penalty=1.33). Default generation config (beam=4, lp=1.0) yields ROUGE-1=47.86, ROUGE-2=23.22, ROUGE-L=39.85.

⚠️ License: SAMSum is released under CC BY-NC-ND 4.0 (non-commercial, no derivatives). This model card, the model weights, and any outputs produced with them are subject to the same terms. Commercial use is prohibited.

---

Model Description

---

Intended Use

• Intended use: Summarizing short chat conversations (≤ 512 tokens) into 1–3 sentence abstractive summaries.
• Out-of-scope: Real-time transcription, audio processing, multi-lingual dialogues, or any commercial product.
• Not recommended for: Mission-critical applications where hallucinations cannot be tolerated. The model hallucinates entity-level details in ~10% of test examples.

---

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

model_id = "your-hf-username/bart-base-samsum-summarizer"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model     = AutoModelForSeq2SeqLM.from_pretrained(model_id, dtype=torch.bfloat16)
model.eval()

dialogue = """
Amanda: I baked cookies. Do you want some?
Jerry: Sure!
Amanda: I'll bring you tomorrow :-)
Jerry: Thanks! Do you know how to make the lemon ones?
Amanda: The biscuits? I'll send you the recipe. It's easy!
""".strip()

inputs = tokenizer(dialogue, return_tensors="pt", max_length=512, truncation=True)
with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens = 128,
        num_beams      = 5,
        length_penalty = 1.33,  # D27 champion config (ROUGE-L 40.12)
        early_stopping = True,
    )
print(tokenizer.decode(out[0], skip_special_tokens=True))
# → "Amanda will bring Jerry some cookies tomorrow and send him the recipe."

---

Performance

All metrics are macro-averaged ROUGE F-measures × 100 on the 819-sample SAMSum test set.

Test-Set ROUGE

Comparison: Fine-Tuned vs Zero-Shot

Decoding Strategy Ablation (11 configs)

Full 29-config sweep results in results/metrics/decoding_D*.json. Champion: D27 (beam=5, lp=1.33) at ROUGE-L 40.12 — see docs/EXPERIMENTS.md for complete E3 table.

Faithfulness Metrics

LoRA Parameter-Efficient Fine-Tuning

LoRA achieves 94.2% of full fine-tune ROUGE-L with only 0.63% trainable parameters.

PEGASUS Cross-Domain Transfer

Training failure: gradient_accumulation_steps=8 on MPS caused 8× gradient inflation (effective lr=1.6e-4). eval_loss=9.601 at epoch 3 ≈ random baseline. Fixed in script (grad_accum=1); ROUGE-L 40–44 expected on re-run.

Extended Training (E8 — 8 epochs, cosine LR, lr=3e-5)

Finding: Δ ROUGE-L = −1.39. Lower peak LR caused underfitting; baseline with lr=5e-5 linear decay converges to a better optimum. Hypothesis not supported.

---

Training Procedure

Dataset

• Train: 14,731 examples
• Validation: 818 examples
• Test: 819 examples
• Variant used: with_speakers — speaker attribution tags (Name: ) preserved. Ablation shows this contributes +6.62 ROUGE-L vs stripping tags.

Preprocessing

Dialogues are tokenized with AutoTokenizer from facebook/bart-base. max_source_length=512, max_target_length=128 (covers 99%+ of SAMSum examples at these lengths). No task prefix (BART does not require one; T5 uses "summarize: ").

Hyperparameters

Compute

Trained on Apple M4 Pro (T6041), 24 GB Unified Memory, 20 GPU cores. PyTorch 2.10.0 MPS backend, BF16.

---

Limitations

• Synthetic training data: SAMSum was constructed by human annotators writing fictional WhatsApp-style dialogues. The model has not been evaluated on real meeting transcripts or audio-derived text.
• Two-speaker bias: ~75% of SAMSum examples involve exactly 2 participants. Summarization quality for 3+ speaker conversations is likely lower.
• Hallucination: ~10.1% of test summaries contain at least one NER-detected hallucinated entity. The actual hallucination rate is higher for non-entity errors (e.g. fabricated scores, inverted speaker actions).
• Speaker attribution errors: ~25% of summaries have at least one speaker attribution mistake (e.g. "X will call Y" when it is Y who called).
• Non-commercial only: CC BY-NC-ND 4.0 applies to all outputs.

---

Citation

@inproceedings{gliwa-etal-2019-samsum,
    title     = "{SAMS}um Corpus: A Human-annotated Dialogue Dataset
                 for Abstractive Summarization",
    author    = "Gliwa, Bogdan and Mochol, Iwona and Biesek, Maciej
                 and Wawer, Aleksander",
    booktitle = "Proceedings of the 2nd Workshop on New Frontiers in
                 Summarization",
    year      = "2019",
    publisher = "Association for Computational Linguistics",
    doi       = "10.18653/v1/D19-5409",
}

---

How to Push to HuggingFace Hub

# 1. Log in
huggingface-cli login

# 2. Create the repository (replace <username>)
huggingface-cli repo create bart-base-samsum-summarizer --type model

# 3. Push model weights + tokenizer
python3 - <<'EOF'
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

model_path = "models/best/facebook_bart-base_with_speakers"
repo_id    = "your-hf-username/bart-base-samsum-summarizer"   # ← replace

tok = AutoTokenizer.from_pretrained(model_path)
mdl = AutoModelForSeq2SeqLM.from_pretrained(model_path, dtype=torch.bfloat16)

tok.push_to_hub(repo_id)
mdl.push_to_hub(repo_id)
print(f"✅ Pushed to https://huggingface.co/{repo_id}")
EOF

# 4. Push model card
huggingface-cli upload your-hf-username/bart-base-samsum-summarizer \
    model_card.md README.md

# 5. Verify
huggingface-cli whoami
# → Opens https://huggingface.co/your-hf-username/bart-base-samsum-summarizer

Note: Do NOT push models/best/ to GitHub — model weights belong on the HuggingFace Hub only. The .gitignore should already exclude models/.