roberta-nli-classifier

Model Card for roberta-nli-classifier

This is a binary Natural Language Inference (NLI) classifier that determines whether a hypothesis is entailed by a given premise. It is a Category A (Transformer-based) implementation that fine-tunes roberta-large with a multi-pooling MLP head, achieving 92.43% dev accuracy and 92.42% macro F1 — a statistically significant improvement of +9.93 percentage points over the given BERT baseline (one-sample t-test: t = 28.58, p < 0.0001).

Model Details

Model Description

This model builds on the roberta-large transformer encoder (Liu et al., 2019). A premise-hypothesis pair is tokenised jointly using the RoBERTa tokeniser and fed through the full 24-layer encoder. Rather than relying on the CLS token alone, the model aggregates the final hidden states using three parallel pooling strategies:

1. CLS token — the [CLS] representation conventionally used for sequence classification.
2. Mean pooling — the attention-masked mean of all non-padding token hidden states, capturing the average contextual signal across the sequence.
3. Max pooling — the element-wise maximum over all non-padding token positions, preserving the strongest activated feature per dimension.

The three vectors (each of size H = 1024) are concatenated into a single H×3 = 3072-dimensional representation and passed through a 3-layer MLP classifier:

Linear(3072 → 1024) → GELU → Dropout(0.1)
Linear(1024 → 512)  → GELU → Dropout(0.1)
Linear(512  → 2)

The final logits are passed to a standard cross-entropy loss during training. The best checkpoint was selected based on peak dev accuracy across 5 epochs, rather than the final epoch, to prevent overfitting.

• Developed by: 11309715, 11303252 and 11382801 (Group 8)
• Language(s): English
• Model type: Supervised fine-tuning
• Model architecture: Transformer encoder (RoBERTa-large) with CLS+Mean+Max pooling and 3-layer MLP classifier head
• Finetuned from model: roberta-large
• Total parameters: 359,032,322
• Model size: ~1.42 GB

Model Resources

• Repository: https://huggingface.co/hk414xin/roberta-nli-classifier/

• Foundational Architecture: RoBERTa Pre-trained Encoder (Liu et al., 2019)

  • Link: https://arxiv.org/abs/1907.11692

Training Details

Training Data

24,432 premise-hypothesis pairs for training and 6,736 pairs for development, sourced from the COMP34812 NLI track dataset. Labels are near-balanced: 12,648 positive (entailment) and 11,784 negative (non-entailment) in training, and 3,478 positive and 3,258 negative in development. No class weighting or oversampling was applied given this near-balance.

Each example was tokenised as a joint sequence [CLS] premise [SEP] hypothesis [SEP], padded or truncated to a maximum of 256 subword tokens.

Training Procedure

Training proceeded in two stages: a hyperparameter grid search, followed by a full 5-epoch training run using the best identified configuration.

Stage 1 — Hyperparameter Grid Search

A 16-configuration grid search was conducted, evaluating each configuration for 2 warm-up epochs on the full training set and recording dev accuracy. The search space crossed learning rate ∈ {1e-5, 2e-5}, batch size ∈ {16, 32}, dropout ∈ {0.1, 0.2}, and max sequence length ∈ {128, 256}, with warmup ratio and weight decay fixed at 0.1. The best configuration by 2-epoch dev accuracy was lr=2e-5, batch=32, dropout=0.1, max_len=256, achieving 0.9256.

Stage 2 — Full Training Run

The best configuration was trained for 5 full epochs with seed 42. The checkpoint at epoch 4 (highest dev accuracy) was saved as the final model.

Training Hyperparameters

• learning_rate: 2e-05
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• num_epochs: 5
• max_seq_length: 256
• dropout: 0.1
• warmup_ratio: 0.1
• weight_decay: 0.1
• optimizer: AdamW with three parameter groups — encoder parameters with decay (lr=2e-5, wd=0.1), encoder bias/LayerNorm parameters without decay (lr=2e-5, wd=0.0), and classifier head at 5× encoder lr (lr=1e-4, wd=0.1)
• lr_scheduler: Linear warmup + linear decay (get_linear_schedule_with_warmup, 10% warmup steps)
• fp16: True (torch.amp)
• gradient_clipping: 1.0

Speeds, Sizes, Times

• GPU used: NVIDIA GH200 120GB (102 GB VRAM)
• Duration per training epoch: ~67–68 seconds
• Total training time (5 epochs): ~5.7 minutes
• Total parameters: 359,032,322
• Model size: ~1.42 GB (roberta-large base weights) + custom MLP head

Evaluation

Testing Data

The full development set of 6,736 premise-hypothesis pairs provided as part of the COMP34812 NLI track, used as a held-out evaluation set after the best checkpoint (epoch 4) was selected.

Metrics

• Accuracy
• Macro F1-score
• Per-class Precision, Recall, F1-score (via sklearn.metrics.classification_report)
• Statistical significance (one-sample t-test and two-sample t-test via scipy.stats)

Fine Tuning Results

The best checkpoint (saved at epoch 4) achieved a dev accuracy of 92.43% and macro F1 of 92.42%.

Training Trajectory:

By epoch 5, training loss continued to drop while dev loss rose from 0.4174 to 0.4744 and dev accuracy declined — a clear sign of overfitting.

Per-Class Classification Report (Epoch 4 checkpoint):

The model performs symmetrically across both classes, with slightly higher recall on entailment (93.01%) and slightly higher precision on non-entailment (92.49%).

Multi-Run Reliability & Statistical Significance

To ensure results are robust against random seed variation and to confirm that gains are not due to "lucky" initialisations, the model was trained 5 times using independently sampled random seeds.

Baseline Comparison

Performance was benchmarked against the official COMP34812 NLI track BERT baseline, evaluated on the consistent 6,736-pair development set. While BERT provides a strong transformer-based baseline, it remains significantly below the performance of the proposed RoBERTa-based approach.

Experimental Results & Variance

The table below summarises the reliability of the fine-tuned model across multiple runs. The low standard deviation (0.70%) indicates reasonable training stability. Note that Run 5 converged more slowly (seed=67935 produced a difficult initialisation), which pulled the mean down slightly relative to the single-seed best result.

Note on Run 5: This run used seed 67935, which led to a slower warm-up phase. By epoch 5 it had recovered to 90.59%, but never reached the peak achieved by the other runs. This is reflected in the wider standard deviation compared to the single fixed-seed result.

Statistical Hypothesis Testing

A one-sample, one-sided t-test was conducted to evaluate the significance of the improvement over the strongest baseline (BERT Accuracy = 0.8202).

• Null Hypothesis (H₀): μ_acc ≤ 0.8202
• Alternative Hypothesis (H₁): μ_acc > 0.8202

The test yielded a t-statistic of 28.58 (p = 0.000004) at a significance level of α = 0.05. The mean multi-run accuracy of 92.00% represents an improvement of +9.93 percentage points over the BERT baseline. We reject the null hypothesis, confirming with high confidence that the performance gains are statistically significant and not merely due to a favourable random seed.

K-Fold Cross-Validation & Consistency Check

To confirm performance is not an artefact of the particular train/dev split, 5-fold stratified cross-validation was run on the full training set (19,545–19,546 train / 4,886–4,887 val per fold):

A two-sample t-test comparing the 5 multi-run dev accuracies against the 5 k-fold val accuracies found no significant difference: t = −1.5289, p = 0.1648 (α = 0.05). The non-significant result confirms that performance is consistent across evaluation strategies. The reported accuracy is a reliable estimate of true generalisation ability, not an optimistic artefact of a favourable train/dev partition.

Technical Specifications

Hardware

• GPU: NVIDIA GH200 120GB
• VRAM: 102 GB
• Mixed precision (fp16) training enabled via torch.amp

Software

• PyTorch (with torch.amp for mixed-precision training)
• Transformers (HuggingFace) — AutoModel, AutoTokenizer, get_linear_schedule_with_warmup
• scikit-learn — accuracy_score, f1_score, classification_report, StratifiedKFold
• SciPy — scipy.stats.ttest_1samp, scipy.stats.ttest_ind
• pandas, numpy

Bias, Risks, and Limitations

• Any premise-hypothesis pair tokenised to more than 256 subword tokens will be truncated, which may cause information loss for longer inputs.
• The model was trained and evaluated on a single academic NLI dataset (COMP34812). Performance on out-of-distribution or domain-shifted text pairs (e.g., legal, medical, conversational) may be lower.
• The model predicts binary entailment only (entailment vs. non-entailment) and is not designed for three-way NLI (entailment / neutral / contradiction).
• Alternative pooling strategies (gated pooling, attention pooling) combined with LLRD, label smoothing, and freeze-then-unfreeze scheduling were explored but did not yield meaningful improvements over the CLS+mean+max baseline (see Additional Information).
• As with all NLI datasets, annotation artefacts (e.g., lexical overlap, negation patterns) may have been learned by the model rather than genuine semantic reasoning.
• The multi-run standard deviation (0.70%) is higher than the k-fold standard deviation (0.32%), partly driven by Run 5's slow initialisation. Users should be aware that occasional seeds may produce a weaker result.

Additional Information

• The initial training run used a fixed seed (42) and max_len=128. To determine whether a longer context window or different regularisation would help, a systematic 16-configuration grid search was conducted crossing learning rate ∈ {1e-5, 2e-5}, batch size ∈ {16, 32}, dropout ∈ {0.1, 0.2}, and max sequence length ∈ {128, 256}. The winning configuration — lr=2e-5, batch=32, dropout=0.1, max_len=256 — outperformed its max_len=128 equivalent (0.9256 vs 0.9219 at 2 epochs), confirming that the longer context window captures nuanced wording in longer premise-hypothesis pairs that would otherwise be truncated.

• In parallel, two alternative pooling architectures were implemented and evaluated against the CLS+mean+max baseline:

  • Gated Pooling replaced fixed pooling with a learned sigmoid gate: gate_scores = sigmoid(Linear(H→1)(seq)), output = (gate_scores * seq).sum(1) / gate_scores.sum(1). This allows the model to softly weight each token's contribution to the final representation.

  • Attention Pooling used a trainable query vector to compute a softmax distribution over token hidden states: scores = Linear(H→1)(tanh(Linear(H→H)(seq))), output = (softmax(scores) * seq).sum(1).

  Both variants were further combined with Layer-wise Learning Rate Decay (LLRD), label smoothing, and a freeze-then-unfreeze training schedule. Despite these additions, neither architecture produced meaningful gains over the simpler CLS+mean+max baseline. Dev accuracy remained largely unchanged across all configurations, suggesting the baseline multi-pooling strategy already captures sufficient contextual information for binary NLI on this dataset, and that the added complexity introduced variance without benefit.