llama3.1-8b-prefixmemory-cefr-balanced-topic-aligned

Low Rank PrefixMemory-Tuning: Decoupled CEFR Baseline

This repository contains the PrefixMemory-Tuning (PMT) baseline developed for controllable CEFR-level text generation.

The model extends Meta Llama-3.1-8B-Instruct with a lightweight external PrefixMemory controller while keeping all base LLM parameters frozen. Only the PrefixMemory controller is optimized during training.

This model serves as the baseline architecture against which all subsequent hierarchical and dual-stream ablation models are compared.

---

Base Model

Foundation Model

• Model: Meta-Llama-3.1-8B-Instruct
• Parameters: 8 Billion
• Backbone: Decoder-only Transformer
• Transformer Layers: 32
• Hidden Dimension: 4096
• Frozen during training: Yes

Only the PrefixMemory controller is trainable.

---

Architecture

The architecture follows the PrefixMemory-Tuning paradigm by attaching a lightweight external memory controller to every transformer attention block.

<p align="center"> <img src="images/PMT_Arc.png" width="800"> </p> <p align="center"> <b>Figure 1.</b> Overall architecture of the CEFR controller integrated with PrefixMemory.. </p>

Controller Pipeline

For a target CEFR level

$$ c \in {A1, A2, B1, B2, C1, C2} $$

the controller first retrieves an embedding

$$ e_c = \mathrm{Embedding}(c) $$

which is transformed by a two-layer MLP

$$ m = \mathrm{MLP}(e_c) $$

The modulation vector is reshaped into

$$ m \in \mathbb{R}^{32 \times r} $$

where

• 32 = transformer layers
• r = PrefixMemory rank (64)

For transformer layer $l$, query states are projected into the low-rank PrefixMemory space

$$ Q_l = XW^{\mathrm{down}}_l $$

where

• $X$ is the hidden representation.
• $W^{\mathrm{down}} \in \mathbb{R}^{4096 \times 64}$.

The controller applies element-wise modulation

$$ \phi(Q_l) = \mathrm{ELU}!\left(Q_l \odot m_l\right) $$

The external PrefixMemory bias becomes

$$ B_l = \phi(Q_l)W^{\mathrm{up}}_l $$

where

$$ W^{\mathrm{up}} \in \mathbb{R}^{64 \times 4096} $$

Finally,

$$ H_l^{\mathrm{out}}

H_l^{\mathrm{attn}} + \alpha B_l $$

where

• $H_l^{\mathrm{attn}}$ is the original attention output.
• $\alpha$ is a learnable scaling parameter.

---

Training Objective

Only the PrefixMemory controller parameters are optimized.

The frozen Llama backbone receives the modified hidden representations.

Training minimizes the standard causal language modeling objective

$$ \mathcal{L}

\sum_t \log P(x_t \mid x_{<t}, c) $$

using masked prompt tokens.

Optimization

• Optimizer: AdamW
• Learning rate: $$2 \times 10^{-4}$$
• Batch size: 8
• Epochs: 5

---

Training Dataset

Training uses a balanced subset derived from the EFCAMDAT learner corpus.

Original dataset:

https://ef-lab.mmll.cam.ac.uk/EFCAMDAT.html

Due to the EFCAMDAT data usage policy, the processed training subset cannot be redistributed.

Dataset Columns

Distribution

Balanced across all CEFR levels.

Total:

6,000 samples

---

Training Performance

Best checkpoint:

Epoch 2

Validation Perplexity:

7.08

[
  {
    "epoch": 1,
    "train_loss": 2.1147,
    "val_loss": 2.0288,
    "val_ppl": 7.61
  },
  {
    "epoch": 2,
    "train_loss": 1.8625,
    "val_loss": 1.9573,
    "val_ppl": 7.08
  },
  {
    "epoch": 3,
    "train_loss": 1.5514,
    "val_loss": 2.0191,
    "val_ppl": 7.53
  },
  {
    "epoch": 4,
    "train_loss": 1.1617,
    "val_loss": 2.2208,
    "val_ppl": 9.21
  },
  {
    "epoch": 5,
    "train_loss": 0.7790,
    "val_loss": 2.5407,
    "val_ppl": 12.69
  }
]

---

In-Domain Evaluation

Benchmark

Balanced benchmark constructed from held-out EFCAMDAT prompts.

• Total prompts: 702
• Source: EFCAMDAT
• Prompt-only evaluation
• Topic distribution balanced across CEFR levels

Dataset columns

Distribution

---

Results

PMT Evaluation Results

============================================================
📊 FINAL COMPILED MACRO-STATISTICS (PREFIXMEMORY-TUNING BASELINE)
============================================================

Total Processed       : 702
Strict Accuracy       : 56.27%
Adjacent Accuracy     : 73.36%
Mean Abs Error (MAE)  : 0.9103
Avg MDD Score         : 1.77
Avg Reading Ease      : 83.51
Avg Sentence Drift    : 2.30 levels
Avg Perplexity        : (Computed externally)

============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================

             strict_accuracy  avg_mdd  avg_flesch  avg_drift
target_cefr
-------------------------------------------------------------
A1                     74.36     1.35       89.07       1.01
A2                     69.23     1.67       87.02       1.73
B1                     60.68     1.85       83.37       2.07
B2                     50.43     1.87       82.56       2.94
C1                     45.30     1.93       78.48       2.86
C2                     37.61     1.95       80.55       3.20

============================================================
📝 DETAILED CLASSIFICATION REPORT
============================================================

              precision    recall  f1-score   support

A1               0.56       0.74      0.64       117
A2               0.57       0.69      0.62       117
B1               0.46       0.61      0.53       117
B2               0.48       0.50      0.49       117
C1               0.73       0.45      0.56       117
C2               0.77       0.38      0.51       117

-------------------------------------------------------------

Accuracy                                 0.56       702
Macro Avg         0.60       0.56      0.56       702
Weighted Avg      0.60       0.56      0.56       702

PMT Confusion Matrix

<p align="center"> <img src="images/PMT-Confusion-Matrix-700-Prompt.png" width="800"> </p> <p align="center"> <b>Figure 2.</b> Confusion Matrix of PrefixMemory Tuning on In-Domain Evaluation Dataset (Inference). </p>

Improvement over Base Llama

• +31.63 percentage points Strict Accuracy
• +10.17 percentage points Adjacent Accuracy
• Lower MAE
• Lower syntactic deviation
• Much stronger readability control

Base LLM Evaluation Results

Baseline LLM Evaluation Results

============================================================
📊 FINAL COMPILED MACRO-STATISTICS (BASE LLM)
============================================================

Total Processed       : 690
Strict Accuracy       : 24.64%
Adjacent Accuracy     : 63.19%
Mean Abs Error (MAE)  : 1.3275
Avg MDD Score         : 2.00
Avg Reading Ease      : 49.47
Avg Sentence Drift    : 3.04 levels
Avg Perplexity        : 4.14

============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================

             strict_accuracy  avg_mdd  avg_flesch  avg_drift  avg_perplexity
target_cefr
-------------------------------------------------------------------------
A1                     16.24     1.73       82.56       3.25            5.16
A2                     23.08     1.64       85.44       3.27            5.12
B1                     17.39     2.03       57.80       3.11            3.86
B2                     41.74     2.15       34.18       2.97            3.32
C1                     33.63     2.24       18.95       2.94            3.59
C2                     15.93     2.25       15.54       2.69            3.74

============================================================
📝 DETAILED CLASSIFICATION REPORT
============================================================

              precision    recall  f1-score   support

A1               0.28       0.16      0.20       117
A2               0.38       0.23      0.29       117
B1               0.21       0.17      0.19       115
B2               0.22       0.42      0.29       115
C1               0.24       0.34      0.28       113
C2               0.23       0.16      0.19       113

-------------------------------------------------------------

Accuracy                                 0.25       690
Macro Avg         0.26       0.25      0.24       690
Weighted Avg      0.26       0.25      0.24       690

Base LLM Confusion Matrix

<p align="center"> <img src="images/base_llm_confusion_matrix.png" width="800"> </p> <p align="center"> <b>Figure 3.</b> Confusion Matrix of Base LLM on In-Domain Evaluation Dataset (Inference). </p>

---

Out-of-Domain Evaluation

Benchmark

MohammadKhosravi/ielts-cefr-benchmark-prompts

A manually curated benchmark containing IELTS-style prompts designed to evaluate prompt generalization.

Dataset:

https://huggingface.co/datasets/MohammadKhosravi/ielts-cefr-benchmark-prompts

Columns

Distribution

Balanced

Total:

300 prompts

---

Results

Improvement over Base Llama

• +25.67 percentage points Strict Accuracy
• +17.34 percentage points Adjacent Accuracy
• Lower MAE
• Significantly improved readability control
• Strong transfer to an unseen prompt style

---

Intended Use

This model is intended for research on

• controllable text generation
• CEFR-level writing generation
• educational NLP
• readability control
• parameter-efficient fine-tuning
• PrefixMemory-Tuning

---

Citation

If you use this model, please cite:

• PrefixMemory-Tuning (arXiv:2506.13674)
• Meta Llama-3.1-8B-Instruct
• EFCAMDAT Corpus
• This repository