Downloads · 30 days
0
MohammadKhosravi/llama3.1-8b-pure-pmt-cefr-gating-no-cefr-cues
llama3.1-8b-pure-pmt-cefr-gating-no-cefr-cues is a machine learning model from MohammadKhosravi. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as apache-2.0.
This repository contains the Prompt-Cue Ablation of CEFR-Gated PrefixMemory-Tuning (PMT) used in the master's thesis:
Downloads · 30 days
0
Access
Public
Updated Sep 13, 2026
Repo size
1.1 GB
Likes
0
Public
Click a slice to open those files.
.pt1.1 GB · 100%
From the Hugging Face model README
This repository contains the Prompt-Cue Ablation of CEFR-Gated PrefixMemory-Tuning (PMT) used in the master's thesis:
Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation
The model uses a frozen meta-llama/Llama-3.1-8B-Instruct backbone together with:
4096 × 4096 trainable memory matrix for each of the 32 transformer layersalphaThis ablation removes the explicit textual proficiency cues used by the full CEFR-Gated PMT configuration while leaving the controller architecture unchanged.
The experiment therefore measures how much CEFR control remains when proficiency information is communicated primarily through the architectural conditioning mechanism rather than through explicit prompt cues.
The full CEFR-Gated PMT prompt contains general proficiency guidance such as:
This ablation removes those explicit proficiency cues.
The prompt still:
but does not include explicit lower-level simplicity cues or higher-level advanced-language cues.
The CEFR-Gated PMT architecture itself remains unchanged.
This creates the following controlled comparison:
Full CEFR-Gated PMT
Architectural Conditioning
+
Textual Proficiency Cues
vs.
Prompt-Cue Ablation
Architectural Conditioning
+
No Explicit Proficiency Cues
The purpose is to estimate the contribution of textual proficiency guidance independently from the contribution of the CEFR-conditioned architecture.
The controller contains three trainable components:
Each of the 32 transformer layers receives one full hidden-dimension trainable memory matrix:
32 × 4096 × 4096
This contributes:
536,870,912 trainable parameters
For layer (l):
$$ M^{(l)} \in \mathbb{R}^{4096 \times 4096} $$
with:
$$ l \in {0,\ldots,31} $$
One trainable embedding is learned for each of the six CEFR levels:
6 × 4096
This contributes:
24,576 trainable parameters
The six target classes are:
A1
A2
B1
B2
C1
C2
For a target CEFR level (c):
$$ e_c \in \mathbb{R}^{4096} $$
A single scalar alpha controls the magnitude of the injected memory contribution:
1 parameter
The scalar is optimized jointly with the CEFR embeddings and memory matrices.
| Component | Parameters |
|---|---|
| PMT memory matrices | 536,870,912 |
| CEFR embeddings | 24,576 |
Learnable alpha | 1 |
| Total | 536,895,489 |
The meta-llama/Llama-3.1-8B-Instruct backbone remains frozen.
For each requested CEFR level, the corresponding trainable embedding is selected.
The controller applies the following transformation:
Hidden Representation
│
▼
ELU
│
▼
Element-Wise Multiplication
with CEFR Embedding
│
▼
Layer-Specific Memory Matrix
│
▼
Memory-Derived Bias
│
▼
Scaled Residual Injection
│
▼
Attention Output
Conceptually:
gated_X = ELU(hidden_states) * cefr_embedding
memory_bias = gated_X @ M_l
output = attention_output + alpha * memory_bias
For transformer layer (l), let:
$$ X^{(l)} \in \mathbb{R}^{B \times T \times 4096} $$
denote the hidden representation.
The nonlinear feature representation is:
\operatorname{ELU}\left(X^{(l)}\right) $$
For target CEFR class (c), the corresponding embedding is:
$$ e_c \in \mathbb{R}^{4096} $$
The CEFR-conditioned representation is:
\phi\left(X^{(l)}\right) \odot e_c $$
where (\odot) denotes element-wise multiplication, with the CEFR embedding broadcast across the batch and sequence dimensions.
The memory-derived bias is:
\widetilde{X}^{(l)} M^{(l)} $$
where:
$$ M^{(l)} \in \mathbb{R}^{4096 \times 4096} $$
The resulting attention output is:
H_{\mathrm{attn}}^{(l)} + \alpha B_{\mathrm{mem}}^{(l)} $$
The CEFR conditioning therefore directly modulates the hidden representation before external memory retrieval.
Unlike CEFR Multi-Prefix Tuning, this architecture does not select a dedicated virtual prefix bank.
Instead:
Target CEFR
│
▼
cefr_id
│
▼
4096-Dimensional
CEFR Embedding
│
▼
Feature-Wise Multiplicative
Gating
│
▼
Full-Rank PMT Memory
│
▼
Memory Bias
The selected CEFR embedding multiplicatively modulates the transformed hidden representation before the layer-specific memory matrix is applied.
This creates an explicit feature-wise architectural CEFR control signal.
The defining difference between this model and the full CEFR-Gated PMT configuration is the textual prompt.
In the full model, the prompt contains explicit proficiency cues.
In this ablation, those cues are removed.
Importantly, the target CEFR class is still supplied to the controller as:
cefr_id
so architectural CEFR conditioning remains fully active.
The comparison can therefore be summarized as:
Full Model
│
┌───────────┴───────────┐
│ │
▼ ▼
Architectural CEFR Textual Proficiency
Conditioning Cues
│ │
└───────────┬───────────┘
│
▼
CEFR Generation
Prompt Ablation
│
▼
Architectural CEFR
Conditioning
│
▼
CEFR Generation
This allows the experiment to estimate whether the full model's CEFR performance arises primarily from:
Training uses the Balanced CEFR Steering Subset derived from the EFCAMDAT Steering Training Dataset.
The dataset contains exactly 5,568 examples, balanced equally across the six CEFR classes:
| CEFR Level | Samples |
|---|---|
| A1 | 928 |
| A2 | 928 |
| B1 | 928 |
| B2 | 928 |
| C1 | 928 |
| C2 | 928 |
| Total | 5,568 |
A stratified 90/10 train-validation split is used with random_state=42.
| Partition | Samples |
|---|---|
| Training | 5,011 |
| Validation | 557 |
| Total | 5,568 |
Training uses the standard autoregressive causal language-modeling cross-entropy objective.
The textual instruction is concatenated with the reference response, while prompt tokens are excluded from the supervised target region.
Only the CEFR-Gated PMT controller is trained.
The Llama-3.1-8B-Instruct backbone remains frozen.
The target-only objective can be written as:
-\sum_{t \in \mathcal{T}{\mathrm{target}}} \log p{\theta} \left( y_t \mid y_{<t},x,c \right) $$
where:
| Setting | Value |
|---|---|
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
| Backbone state | Frozen |
| Epochs | 3 |
| Optimizer | AdamW |
| Learning rate | 2e-4 |
| Training batch size | 8 |
| Validation batch size | 8 |
| Random seed | 42 |
| CEFR classes | 6 |
| Hidden dimension | 4096 |
| Transformer layers | 32 |
The recorded training trajectory is:
| Epoch | Train Loss | Validation Loss | Validation Perplexity |
|---|---|---|---|
| 1 | 2.1370 | 1.8798 | 6.55 |
| 2 | 1.5536 | 1.8392 | 6.29 |
| 3 | 1.0606 | 2.0055 | 7.43 |
The best validation checkpoint is obtained at Epoch 2.
Validation loss follows:
1.8798 → 1.8392 → 2.0055
while validation perplexity follows:
6.55 → 6.29 → 7.43
Although training loss continues to decrease during Epoch 3, validation performance degrades.
Therefore, the Epoch 2 checkpoint is retained.
The trained prompt-cue ablation is evaluated on the final In-Domain Evaluation Prompt Matrix used throughout the thesis.
The benchmark contains:
117 topics × 6 CEFR levels = 702 generation conditions
During generation:
cefr_id.Conceptually:
Input Topic
│
▼
Prompt Without Explicit
Proficiency Cues
│
├────────────────────────────┐
│ │
▼ ▼
Text Tokens Target CEFR
│
▼
cefr_id
│
▼
CEFR Embedding
│
▼
CEFR-Gated PMT
│
┌────────────────────────────┘
▼
Frozen Llama-3.1-8B
│
▼
Generated Response
Generated texts are evaluated using the primary thesis CEFR evaluator:
MohammadKhosravi/roberta-large-cefr-classifier-JointLoss
The evaluator is independent of the generation controller and is not involved in training.
The prompt-cue ablation obtains the following results over all 702 in-domain generation conditions:
| Metric | Result |
|---|---|
| Total generations | 702 |
| Strict Accuracy | 63.11% |
| Adjacent Accuracy | 78.06% |
| Mean Absolute Error (MAE) | 0.7407 |
| Mean Dependency Distance (MDD) | 1.78 |
| Mean Flesch Reading Ease | 80.62 |
| Mean Sentence-Level CEFR Drift | 2.36 levels |
Strict Accuracy requires the predicted CEFR class to exactly match the requested target level.
For example:
Target B1:
B1 → Correct
A2 / B2 → Incorrect
A1/C1/C2 → Incorrect
Adjacent Accuracy also accepts predictions one neighboring CEFR level away from the target.
For example:
Target B1:
B1 → Strict + Adjacent correct
A2 / B2 → Adjacent correct
A1 / C1 / C2 → Incorrect
CEFR levels are represented as ordered classes:
A1 = 0
A2 = 1
B1 = 2
B2 = 3
C1 = 4
C2 = 5
MAE measures the average ordinal distance between requested and predicted levels:
\frac{1}{N} \sum_{i=1}^{N} \left| \hat{y}_i-y_i \right| $$
Lower MAE indicates stronger CEFR control.
The evaluation additionally reports:
These diagnostics supplement the primary CEFR-control metrics.
The main purpose of this artifact is the direct prompt-cue ablation comparison.
| Configuration | Strict Accuracy | Adjacent Accuracy | MAE |
|---|---|---|---|
| Full CEFR-Gated PMT | 68.80% | 81.34% | 0.6368 |
| Prompt-Cue Ablation | 63.11% | 78.06% | 0.7407 |
| Difference | −5.69 pp | −3.28 pp | +0.1039 |
Removing the explicit proficiency cues reduces Strict Accuracy from:
68.80% → 63.11%
corresponding to:
−5.69 percentage points
Adjacent Accuracy decreases from:
81.34% → 78.06%
corresponding to:
−3.28 percentage points
MAE increases from:
0.6368 → 0.7407
indicating larger average ordinal CEFR error.
Despite removing the explicit proficiency cues, the model retains:
63.11% Strict Accuracy
compared with:
68.80%
for the full model.
The ablated model therefore retains approximately:
$$ \frac{63.11}{68.80} \approx 0.9173 $$
or roughly 91.7% of the full model's Strict Accuracy.
This result suggests that textual proficiency cues contribute meaningfully to the final performance, but they do not account for the majority of the model's CEFR-control capability.
The architectural CEFR-conditioning mechanism remains capable of substantial proficiency control even when the explicit linguistic prompt cues are removed.
Conceptually:
Full CEFR-Gated PMT
│
├── Architectural CEFR Conditioning
│
└── Textual Proficiency Cues
│
▼
68.80%
Strict
Prompt-Cue Ablation
│
└── Architectural CEFR Conditioning
│
▼
63.11%
Strict
Within this experimental setup, the difference between the two configurations provides an estimate of the additional contribution of explicit textual guidance.
This experiment serves as a prompt-cue ablation for the proposed CEFR-Gated PMT method.
It separates:
Architectural CEFR Conditioning
+
Textual Proficiency Guidance
from:
Architectural CEFR Conditioning Only
This complements the other controlled experiments in the thesis that distinguish the effects of:
The experiment is particularly important because it tests whether strong CEFR controllability persists when the textual prompt itself provides substantially less direct guidance about linguistic complexity.
The complete training and evaluation notebook is available at:
notebooks/06_cefr_gated_pmt/prompt_cue_ablation.ipynb
The main CEFR-Gated PMT experiment is available at:
notebooks/06_cefr_gated_pmt/cefr_gated_pmt_train_and_eval.ipynb
The balanced training subset is constructed in:
notebooks/01_data_preparation/balanced_cefr_steering_subset.ipynb
The in-domain evaluation matrix is constructed in:
notebooks/01_data_preparation/efcamdat_preprocessing_and_partitioning.ipynb
The notebook retains the original experimental outputs, including:
This model is intended for research on:
Important: This model is an experimental research artifact and is not intended to serve as a standalone CEFR assessment model.
If you use this model or the accompanying experimental resources, please cite the associated master's thesis and repository.
@mastersthesis{khosravi2026beyond,
author = {Mohammad Khosravi},
title = {Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation},
school = {University of Padova},
year = {2026}
}