amazon-all-beauty-led-large-summarization

LED-Large for Amazon All_Beauty Multi-Review Summarization

A fine-tuned allenai/led-large-16384 model for product-level multi-review summarization in the Amazon Reviews 2023 All_Beauty category.

The model takes multiple customer reviews for one product and generates a single abstractive summary that captures the main review consensus, recurring strengths, and meaningful complaints.

Input: product title + multiple review blocks
Output: one product-level summary

The product's numeric rating is not generated by the model. It is computed separately from the original review ratings.

---

Quick Start

Fastest option: open AllBeauty_LED_Large_Inference.ipynb if you want a ready-to-run example without reading the full training notebook.

1. Install dependencies

pip install -U transformers torch sentencepiece

2. Load the model

import torch
from transformers import AutoTokenizer, LEDForConditionalGeneration

MODEL_ID = "vltruong01/amazon-all-beauty-led-large-summarization"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

model = LEDForConditionalGeneration.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
model.eval()

3. Prepare the input

The model was trained on inputs with this structure:

Product: Example Beauty Product

[Review 1]
Rating: 5/5
Title: Works very well
Text: I have used this product for several weeks and it works well...

[Review 2]
Rating: 4/5
Title: Good but expensive
Text: The quality is good, although the price is higher than expected...

[Review 3]
Rating: 2/5
Title: Not for me
Text: I experienced irritation and stopped using it...

Keep each review as a complete block whenever possible.

4. Generate a summary

LED uses sparse attention. The inference code below reproduces the global-attention pattern used in training: the first token, the product-title line, each review header, and each rating line receive global attention.

import re
import torch

MAX_SOURCE_LENGTH = 4096

def build_global_attention_mask(source, input_ids, attention_mask, tokenizer):
    global_attention = torch.zeros_like(input_ids)

    # Always give global attention to the first non-padding token.
    global_attention[:, 0] = 1

    lines = source.splitlines()
    important_prefixes = ("Product:", "[Review ", "Rating:")

    important_texts = [
        line.strip()
        for line in lines
        if line.strip().startswith(important_prefixes)
    ]

    for text in important_texts:
        token_ids = tokenizer(
            text,
            add_special_tokens=False,
        )["input_ids"]

        if not token_ids:
            continue

        first_token = token_ids[0]

        matches = (
            (input_ids[0] == first_token)
            & (attention_mask[0] == 1)
        ).nonzero(as_tuple=False)

        for match in matches:
            global_attention[0, match.item()] = 1

    return global_attention


source = '''Product: Example Beauty Product

[Review 1]
Rating: 5/5
Title: Works very well
Text: I have used this product for several weeks and it works well.

[Review 2]
Rating: 4/5
Title: Good but expensive
Text: The quality is good, although the price is higher than expected.

[Review 3]
Rating: 2/5
Title: Not for me
Text: I experienced irritation and stopped using it.
'''

inputs = tokenizer(
    source,
    return_tensors="pt",
    max_length=MAX_SOURCE_LENGTH,
    truncation=True,
    padding=False,
)

input_ids = inputs["input_ids"].to(device)
attention_mask = inputs["attention_mask"].to(device)

global_attention_mask = build_global_attention_mask(
    source,
    input_ids,
    attention_mask,
    tokenizer,
).to(device)

with torch.inference_mode():
    generated = model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        global_attention_mask=global_attention_mask,
        num_beams=6,
        max_new_tokens=192,
        no_repeat_ngram_size=3,
        repetition_penalty=1.05,
        early_stopping=True,
    )

summary = tokenizer.decode(
    generated[0],
    skip_special_tokens=True,
)

print(summary)

---

Inference Notebook

If you only want to use the trained model, open:

AllBeauty_LED_Large_Inference.ipynb

The notebook is intentionally much shorter than the training notebook and contains only the steps required for inference:

install dependencies
        ↓
load tokenizer + fine-tuned LED-Large
        ↓
paste one product's review collection
        ↓
build structured global attention
        ↓
generate one product-level summary
        ↓
optionally compute the numeric rating separately

The inference notebook reproduces the same structural global-attention pattern used during training.

Global attention is assigned to:

• the first source token,
• the Product: line,
• every [Review N] header,
• every Rating: line.

This matters because LED uses sparse attention for long inputs. These structural anchors help the model connect product identity, review boundaries, and ratings across a long multi-review document.

The notebook also keeps the numeric product rating outside the model. LED generates only the summary.

---

What the Model Does

The model performs multi-review aggregation, not single-review summarization.

It is intended to learn patterns such as:

• what customers most consistently agree on,
• recurring positive experiences,
• recurring negative experiences,
• meaningful minority complaints,
• the overall product-level impression.

The model should not simply concatenate or extract individual review sentences.

---

Dataset

Training data comes from:

vltruong01/amazon-all-beauty-led-summarization

The dataset was derived from Amazon Reviews 2023 All_Beauty product reviews.

Each example contains fields such as:

parent_asin
product_title
rating
num_reviews
source
target_summary
target_word_count
target_qc_ok
source_title_mismatch
mismatch

Before training, rows failing quality control or marked as source/title mismatches were excluded.

The final training experiment used:

The split was deterministic with seed 42.

---

Synthetic Reference Summaries

The target summaries used for training are synthetic reference summaries generated by a teacher model, not human-written gold summaries.

They were produced under constraints intended to:

• preserve product identity,
• summarize the review consensus,
• retain recurring strengths,
• retain meaningful minority complaints,
• avoid unsupported claims,
• avoid numeric rating generation,
• produce one natural paragraph.

This distinction is important when interpreting the evaluation results below: automatic metrics compare LED outputs against these synthetic teacher-generated references.

---

Training Configuration

This was a full fine-tuning of LED-Large, not LoRA.

Global attention

Sparse global attention was assigned to:

• the first source token,
• the product-title line,
• the first token of every [Review N] block,
• every Rating: line.

This gives LED explicit anchors for the product identity and review boundaries across long inputs.

---

Held-Out Test Results

Evaluation was performed on the 196-example held-out test split.

Validation loss for the final run:

2.6452

Automatic metrics are useful for reproducibility, but they do not fully measure factual consistency, consensus weighting, or product-identity preservation.

---

Qualitative Behavior

A manual audit of the 196 held-out generations found several strengths:

• summaries generally reflect the overall review consensus,
• product identity is usually preserved,
• outputs are concise and readable,
• recurring positive and negative opinions are often combined effectively.

Observed limitations include:

• repetitive summary openings,
• occasional local word or phrase repetition,
• product or specification drift,
• occasional polarity or factual drift,
• isolated opinions sometimes being promoted to consensus,
• minority complaints sometimes being over- or under-weighted.

These limitations are important for downstream use.

---

Example Application

A simple application can combine the deterministic product rating with the generated review summary:

Rating: 4.3/5
Summary: Customers generally appreciate ...

The rating should be computed from the original review ratings in application code.

Do not ask LED to generate the numeric rating.

---

Reproduce the Training

The full experiment notebook is included in this repository:

AllBeauty_LED_Large_Training.ipynb

The training workflow is:

prepared product-level review dataset
        ↓
quality filtering
        ↓
deterministic train / validation / test split
        ↓
LED-Large tokenization
        ↓
structured global attention
        ↓
full fine-tuning
        ↓
best-model selection
        ↓
held-out generation
        ↓
ROUGE + BERTScore + qualitative diagnostics

Experiment artifacts are stored under:

artifacts/

Typical files include:

experiment_config.json
final_metrics.json
split_manifest.json
test_predictions.csv

---

Repository Structure

amazon-all-beauty-led-large-summarization/
├── README.md
├── AllBeauty_LED_Large_Training.ipynb
├── AllBeauty_LED_Large_Inference.ipynb
├── config.json
├── generation_config.json
├── model.safetensors
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
├── merges.txt
├── vocab.json
└── artifacts/
    ├── experiment_config.json
    ├── final_metrics.json
    ├── split_manifest.json
    └── test_predictions.csv

The model files are stored at the repository root so the model can be loaded directly with:

LEDForConditionalGeneration.from_pretrained(
    "vltruong01/amazon-all-beauty-led-large-summarization"
)

---

Which File Should I Use?

---

Expected Input

Use the same structured format as training:

Product: <product title>

[Review 1]
Rating: <rating>/5
Title: <review title>
Text: <review text>

[Review 2]
Rating: <rating>/5
Title: <review title>
Text: <review text>

For best results:

• keep the product title at the beginning,
• preserve complete review blocks,
• include review rating lines,
• keep the full input within 4096 tokens,
• avoid mixing reviews from different products.

---

Expected Output

The intended output is:

• one paragraph,
• product-level rather than review-by-review,
• focused on recurring customer opinions,
• balanced across positives and meaningful complaints,
• no bullet list,
• no numeric product rating.

---

Intended Use

Suitable uses include:

• research on long-context review summarization,
• product-level multi-review summarization,
• comparison of LED against text-only or multimodal variants,
• studying consensus aggregation from many customer reviews,
• educational and academic experiments.

---

Limitations

This model has several important limitations:

• It is trained only on the Amazon All_Beauty domain.
• Training targets are synthetic teacher-generated references rather than human-written gold summaries.
• The model can hallucinate or distort product details.
• It can occasionally reverse or flatten sentiment.
• It may over-emphasize isolated opinions or miss minority complaints.
• Summary phrasing can become repetitive.
• Inputs longer than the configured context budget require review selection or truncation before inference.
• Automatic metrics do not guarantee factual faithfulness.

For high-stakes or user-facing deployment, summaries should be treated as generated text and checked against the source reviews.

---

Related Dataset

Training dataset:

vltruong01/amazon-all-beauty-led-summarization

Source-only review dataset:

vltruong01/amazon-all-beauty-led-reviews

---

Citation and Attribution

Base model:

• allenai/led-large-16384

Dataset source:

• Amazon Reviews 2023, All_Beauty category

This repository is provided for research and educational use.