Downloads · 30 days
0
MohammadKhosravi/llama3.1-8b-pure-pmt-cefr-gating-6k
llama3.1-8b-pure-pmt-cefr-gating-6k is a machine learning model from MohammadKhosravi. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as mit.
This model implements a highly faithful replication of the uncompressed PrefixMemory-Tuning architecture (arXiv:2506.13674), modified for explicit CEFR-aligned text generation. By replacing low-rank bottlenecks with f…
Downloads · 30 days
0
Access
Public
Updated Aug 4, 2026
Repo size
1.1 GB
Likes
1
Public
Click a slice to open those files.
.pt1.1 GB · 100%
From the Hugging Face model README
This model implements a highly faithful replication of the uncompressed PrefixMemory-Tuning architecture (arXiv:2506.13674), modified for explicit CEFR-aligned text generation. By replacing low-rank bottlenecks with full-dimensional layer-wise memory matrices and introducing a feature-wise CEFR gating mechanism, it achieves state-of-the-art parameter-efficient controllability. The model steers linguistic complexity architecturally, meaning the desired CEFR proficiency level is enforced directly through learned parameters rather than textual prompt instructions.
This iteration eliminates the low-rank bottleneck projections and instead uses a fully uncompressed memory parameterization.
Specifically, the model removes:
$$ W_{\mathrm{down}} \quad \text{and} \quad W_{\mathrm{up}} $$
and replaces them with a direct memory matrix formulation.
The resulting controller contains approximately
$$ 536.9 \text{ million} $$
trainable parameters.
A trainable CEFR embedding
$$ e_{\mathrm{cefr}} \in \mathbb{R}^{4096} $$
is used to perform feature-wise gating of the activated hidden states. The embedding is applied through element-wise modulation of the ELU feature map before querying the layer-specific uncompressed memory matrix.
This design removes the low-rank information bottleneck and allows CEFR control signals to directly influence all 4096 hidden dimensions at every transformer layer.
<p align="center"> <img src="images/Pure_PMT_Arc.png" width="800"> </p> <p align="center"> <b>Figure 1.</b> Overall architecture of the Pure PrefixMemory-Tuning (PMT) + CEFR Gating </p>For each transformer layer $l \in [0, 31]$:
Hidden states $X$ are passed through an Exponential Linear Unit (ELU):
$$ \phi(X) = \mathrm{ELU}(X) $$
The target CEFR embedding vector
$$ e_{\mathrm{cefr}} \in \mathbb{R}^{4096} $$
is broadcasted and multiplied element-wise with the activated hidden states:
\phi(X) \odot e_{\mathrm{cefr}} $$
The modulated features are multiplied by the layer-specific uncompressed memory matrix:
$$ M^{(l)} \in \mathbb{R}^{4096 \times 4096} $$
yielding
\widetilde{X} M^{(l)} $$
The memory bias is scaled by a learnable parameter $\alpha$ and injected directly into the frozen Llama attention output:
\mathrm{Output}{\mathrm{attn}} + \alpha , \mathrm{Bias}{\mathrm{mem}} $$
The controller was trained using the standard cross-entropy causal language modeling objective via teacher forcing.
The entire Meta-Llama-3.1-8B-Instruct backbone remained strictly frozen throughout training.
Gradients flowed exclusively into:
The objective is to maximize the likelihood of human-written CEFR-aligned reference texts:
\sum_t \log P(x_t \mid x_{<t}, c) $$
where $c$ denotes the target CEFR level.
Training uses a balanced subset derived from the EFCAMDAT learner corpus.
Original Dataset
EFCAMDAT (Cambridge English)
Due to the EFCAMDAT data usage policy, the processed training subset cannot be redistributed.
| Column | Description |
|---|---|
topic_title | Writing prompt |
clean_text | Human learner response |
cefr | Target CEFR level |
Balanced across all six CEFR proficiency levels.
| CEFR | Samples |
|---|---|
| A1 | 1,000 |
| A2 | 1,000 |
| B1 | 1,000 |
| B2 | 1,000 |
| C1 | 1,000 |
| C2 | 1,000 |
Total Samples: 6,000
Best checkpoint Saved:
Epoch 1
[
{
"epoch": 1,
"train_loss": 2.4011,
"val_loss": 2.0971,
"val_ppl": 8.14
},
{
"epoch": 2,
"train_loss": 1.6818,
"val_loss": 2.1277,
"val_ppl": 8.40
},
{
"epoch": 3,
"train_loss": 1.0301,
"val_loss": 2.4258,
"val_ppl": 11.31
},
{
"epoch": 4,
"train_loss": 0.5113,
"val_loss": 2.7793,
"val_ppl": 16.11
},
{
"epoch": 5,
"train_loss": 0.2538,
"val_loss": 3.1331,
"val_ppl": 22.94
}
]
Epoch 1 is used for inference.
Although later epochs achieve lower training loss, validation perplexity steadily increases, indicating overfitting of the uncompressed 536 million parameter PrefixMemory controller. The Epoch 1 checkpoint provides the best balance between controllability and generalization.
A balanced benchmark constructed from completely held-out EFCAMDAT prompts.
Compared to the frozen Meta-Llama-3.1-8B-Instruct baseline:
+44.16% Strict Accuracy
+18.15% Adjacent Accuracy
Halved Prediction Error
Natural Readability Maintained
Pure Architectural Control
============================================================
📊 FINAL COMPILED MACRO-STATISTICS (PURE PMT + CEFR GATING)
============================================================
Total Processed : 702
Strict Accuracy : 68.80%
Adjacent Accuracy : 81.34%
Mean Abs Error (MAE) : 0.6368
Avg MDD Score : 1.86
Avg Reading Ease : 82.44
Avg Sentence Drift : 2.05 levels
============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================
strict_accuracy avg_mdd avg_flesch avg_drift
target_cefr
-------------------------------------------------------------
A1 85.47 1.42 81.12 0.56
A2 86.32 2.16 91.05 1.03
B1 76.92 1.98 81.09 2.04
B2 68.38 1.78 82.50 2.95
C1 51.28 1.82 79.00 2.68
C2 44.44 2.00 79.90 3.02
============================================================
📝 DETAILED CLASSIFICATION REPORT
============================================================
precision recall f1-score support
A1 0.71 0.85 0.78 117
A2 0.71 0.86 0.78 117
B1 0.60 0.77 0.68 117
B2 0.55 0.68 0.61 117
C1 0.87 0.51 0.65 117
C2 0.93 0.44 0.60 117
-------------------------------------------------------------
Accuracy 0.69 702
Macro Avg 0.73 0.69 0.68 702
Weighted Avg 0.73 0.69 0.68 702
============================================================
📊 FINAL COMPILED MACRO-STATISTICS (BASE LLM)
============================================================
Total Processed : 690
Strict Accuracy : 24.64%
Adjacent Accuracy : 63.19%
Mean Abs Error (MAE) : 1.3275
Avg MDD Score : 2.00
Avg Reading Ease : 49.47
Avg Sentence Drift : 3.04 levels
============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================
strict_accuracy avg_mdd avg_flesch avg_drift
target_cefr
-------------------------------------------------------------
A1 16.24 1.73 82.56 3.25
A2 23.08 1.64 85.44 3.27
B1 17.39 2.03 57.80 3.11
B2 41.74 2.15 34.18 2.97
C1 33.63 2.24 18.95 2.94
C2 15.93 2.25 15.54 2.69
Dataset:
MohammadKhosravi/ielts-cefr-benchmark-prompts
A manually curated benchmark containing IELTS-style prompts designed to evaluate out-of-domain CEFR controllability.
Compared to the frozen baseline:
+40.00% Strict Accuracy
+21.34% Adjacent Accuracy
Robust Generalization
Substantial Error Reduction
============================================================
📊 FINAL COMPILED MACRO-STATISTICS
(PURE PMT + EMBEDDING GATING - IELTS OOD)
============================================================
Total Processed : 300
Strict Accuracy : 58.33%
Adjacent Accuracy : 75.67%
Mean Abs Error (MAE) : 0.7700
Avg MDD Score : 1.88
Avg Reading Ease : 78.13
Avg Sentence Drift : 2.08 levels
============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================
strict_accuracy avg_mdd avg_flesch avg_drift
target_cefr
-------------------------------------------------------------
A1 60.0 1.64 80.38 1.08
A2 86.0 1.79 93.90 0.74
B1 70.0 2.02 71.64 2.02
B2 48.0 1.88 72.88 2.72
C1 64.0 1.93 74.00 2.92
C2 22.0 2.03 75.99 2.98

This model is intended for research on:
If you use this model, please cite: