Downloads · 30 days
0
MohammadKhosravi/llama3.1-8b-dual-fpmt-ablation-cefr-only-6k
llama3.1-8b-dual-fpmt-ablation-cefr-only-6k is a text generation model from MohammadKhosravi. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as mit.
This repository contains the Eexperiment 2A architectural ablation of the proposed Dual-Stream Factorized PrefixMemory-Tuning (Dual-FPMT) framework.
Downloads · 30 days
0
Access
Public
Updated Jul 27, 2026
Repo size
74.5 MB
Likes
1
Public
Click a slice to open those files.
.pt69.4 MB · 93%
From the Hugging Face model README
This repository contains the Eexperiment 2A architectural ablation of the proposed Dual-Stream Factorized PrefixMemory-Tuning (Dual-FPMT) framework.
Unlike the complete Dual-FPMT architecture, this model does not receive independent linguistic conditioning signals. Instead, it employs two parallel PrefixMemory streams (Lexical and Syntactic) that are both conditioned on the same shared CEFR embedding.
The purpose of this ablation is to determine whether the improvements observed in the full Dual-FPMT architecture arise from:
Therefore, this experiment isolates the architectural contribution of the dual-stream design while removing explicit lexical and syntactic feature conditioning.
| Property | Value |
|---|---|
| Model | Meta-Llama-3.1-8B-Instruct |
| Parameters | 8 Billion |
| Backbone | Decoder-only Transformer |
| Transformer Layers | 32 |
| Hidden Dimension | 4096 |
| Frozen During Training | Yes |
Only the Dual-FPMT controller parameters are optimized during training. All Llama-3.1-8B backbone weights remain frozen.
The E2A controller consists of two independent PrefixMemory retrieval pathways, referred to as the Lexical Stream and Syntactic Stream. Unlike the complete Dual-FPMT architecture, both streams receive the same CEFR embedding as input.
Each stream maintains its own:
The retrieved memory biases are combined additively before being injected into every self-attention layer of the frozen Llama model.
<p align="center"> <img src="images/dual_FPMT.png" width="800"> </p> <p align="center"> <b>Figure 1.</b> Overall architecture of the Dual-Controller PrefixMemory-Tuning(E2A). </p>The shared CEFR embedding is
$$ e = \mathrm{Embedding}(c) $$
where
Each stream produces its own modulation tensor
\mathrm{MLP}_{\mathrm{lex}}(e) $$
\mathrm{MLP}_{\mathrm{syn}}(e) $$
where
$$ S \in \mathbb{R}^{32 \times 64} $$
For each transformer layer,
XW^{\mathrm{down}}_{\mathrm{lex}} $$
XW^{\mathrm{down}}_{\mathrm{syn}} $$
where
\mathrm{ELU} \left( Q_{\mathrm{lex}} \odot S_{\mathrm{lex}} \right) $$
\mathrm{ELU} \left( Q_{\mathrm{syn}} \odot S_{\mathrm{syn}} \right) $$
\Phi_{\mathrm{lex}} W^{\mathrm{up}}_{\mathrm{lex}} $$
\Phi_{\mathrm{syn}} W^{\mathrm{up}}_{\mathrm{syn}} $$
where
$$ W^{\mathrm{up}} \in \mathbb{R}^{64 \times 4096} $$
The two retrieved PrefixMemory vectors are combined using learnable scaling parameters
\alpha_{\mathrm{lex}} B_{\mathrm{lex}} + \alpha_{\mathrm{syn}} B_{\mathrm{syn}} $$
Finally,
H^{\mathrm{attn}} + B $$
The frozen Llama model is optimized using the standard causal language modeling objective.
\sum_t \log P(x_t \mid x_{<t}) $$
Only the Dual-FPMT controller parameters are updated.
Training uses a balanced subset derived from the EFCAMDAT learner corpus.
Original dataset:
https://ef-lab.mmll.cam.ac.uk/EFCAMDAT.html
Due to the EFCAMDAT data usage policy, the processed training subset cannot be redistributed.
| Column | Description |
|---|---|
topic_title | Writing prompt |
clean_text | Human learner response |
cefr | Target CEFR level |
Balanced across all CEFR levels.
| CEFR | Samples |
|---|---|
| A1 | 1,000 |
| A2 | 1,000 |
| B1 | 1,000 |
| B2 | 1,000 |
| C1 | 1,000 |
| C2 | 1,000 |
Total: 6,000 samples
Best checkpoint: Epoch 2
[
{
"epoch": 1,
"train_loss": 2.0800,
"val_loss": 1.9935,
"val_ppl": 7.34
},
{
"epoch": 2,
"train_loss": 1.7559,
"val_loss": 1.9554,
"val_ppl": 7.07
},
{
"epoch": 3,
"train_loss": 1.2714,
"val_loss": 2.1301,
"val_ppl": 8.42
},
{
"epoch": 4,
"train_loss": 0.7539,
"val_loss": 2.5029,
"val_ppl": 12.22
},
{
"epoch": 5,
"train_loss": 0.4018,
"val_loss": 2.9270,
"val_ppl": 18.67
}
]
A balanced benchmark constructed from held-out EFCAMDAT prompts.
| Column |
|---|
topic_id |
topic_title |
cefr |
| CEFR | Samples |
|---|---|
| A1 | 117 |
| A2 | 117 |
| B1 | 117 |
| B2 | 117 |
| C1 | 117 |
| C2 | 117 |
Total: 702 prompts
The benchmark demonstrates a substantial improvement over the frozen Meta-Llama-3.1-8B-Instruct baseline, increasing strict CEFR accuracy from 24.64% to 62.54%, while reducing the mean absolute CEFR prediction error (MAE) from 1.33 to 0.72. Adjacent accuracy also improves from 63.19% to 80.20%, indicating significantly stronger controllability across neighboring proficiency levels.
============================================================
📊 FINAL COMPILED MACRO-STATISTICS (DUAL-FPMT E2A ABLATION)
============================================================
Total Processed : 702
Strict Accuracy : 62.54%
Adjacent Accuracy : 80.20%
Mean Abs Error (MAE) : 0.7194
Avg MDD Score : 2.03
Avg Reading Ease : 80.78
Avg Sentence Drift : 2.37 levels
Avg Perplexity : (Computed externally)
============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================
strict_accuracy avg_mdd avg_flesch avg_drift
target_cefr
-------------------------------------------------------------
A1 69.23 1.52 83.22 1.11
A2 70.94 1.69 83.54 1.56
B1 75.21 1.88 80.77 2.18
B2 66.67 1.82 85.75 3.10
C1 45.30 1.94 75.34 2.79
C2 47.86 3.32 76.05 3.47
============================================================
📝 DETAILED CLASSIFICATION REPORT
============================================================
precision recall f1-score support
A1 0.65 0.69 0.67 117
A2 0.61 0.71 0.65 117
B1 0.56 0.75 0.64 117
B2 0.56 0.67 0.61 117
C1 0.72 0.45 0.55 117
C2 0.80 0.48 0.60 117
-------------------------------------------------------------
Accuracy 0.63 702
Macro Avg 0.65 0.63 0.62 702
Weighted Avg 0.65 0.63 0.62 702
============================================================
📊 FINAL COMPILED MACRO-STATISTICS (BASE LLM)
============================================================
Total Processed : 690
Strict Accuracy : 24.64%
Adjacent Accuracy : 63.19%
Mean Abs Error (MAE) : 1.3275
Avg MDD Score : 2.00
Avg Reading Ease : 49.47
Avg Sentence Drift : 3.04 levels
Avg Perplexity : 4.14
============================================================
📈 PERFORMANCE BREAKDOWN BY CEFR TARGET LEVEL
============================================================
strict_accuracy avg_mdd avg_flesch avg_drift avg_perplexity
target_cefr
-------------------------------------------------------------------------
A1 16.24 1.73 82.56 3.25 5.16
A2 23.08 1.64 85.44 3.27 5.12
B1 17.39 2.03 57.80 3.11 3.86
B2 41.74 2.15 34.18 2.97 3.32
C1 33.63 2.24 18.95 2.94 3.59
C2 15.93 2.25 15.54 2.69 3.74
============================================================
📝 DETAILED CLASSIFICATION REPORT
============================================================
precision recall f1-score support
A1 0.28 0.16 0.20 117
A2 0.38 0.23 0.29 117
B1 0.21 0.17 0.19 115
B2 0.22 0.42 0.29 115
C1 0.24 0.34 0.28 113
C2 0.23 0.16 0.19 113
-------------------------------------------------------------
Accuracy 0.25 690
Macro Avg 0.26 0.25 0.24 690
Weighted Avg 0.26 0.25 0.24 690
Dataset: MohammadKhosravi/ielts-cefr-benchmark-prompts
A manually curated benchmark containing IELTS-style writing prompts designed to evaluate prompt generalization beyond the EFCAMDAT training domain.
Dataset Repository
https://huggingface.co/datasets/MohammadKhosravi/ielts-cefr-benchmark-prompts
| Column |
|---|
prompt |
cefr |
Balanced across all CEFR proficiency levels.
| CEFR | Samples |
|---|---|
| A1 | 50 |
| A2 | 50 |
| B1 | 50 |
| B2 | 50 |
| C1 | 50 |
| C2 | 50 |
Total: 300 prompts
The E2A Dual-FPMT controller generalizes beyond the EFCAMDAT domain, improving strict CEFR accuracy from 18.33% to 44.67% on unseen IELTS-style prompts while substantially reducing the mean absolute error (MAE) from 1.57 to 1.08. These results indicate that the dual-stream PrefixMemory architecture learns transferable proficiency control rather than memorizing in-domain prompt distributions.
| Metric | PrefixMemory | Base Llama |
|---|---|---|
| Strict Accuracy | 44.67% | 18.33% |
| Adjacent Accuracy | 67.00% | 54.33% |
| MAE | 1.08 | 1.5733 |
| Average MDD | 1.94 | 2.07 |
| Reading Ease | 72.30 | 45.18 |
| Sentence Drift | 2.57 | 2.53 |
This model is intended for research on:
If you use this model in your research, please cite the following resources:
This repository contains the E2A architectural ablation of the proposed Dual-Stream Factorized PrefixMemory-Tuning (Dual-FPMT) framework and is intended to facilitate reproducible research on parameter-efficient controllable language generation.