Downloads · 30 days
0
MohammadKhosravi/llama3.1-8b-pure-pmt-6k
llama3.1-8b-pure-pmt-6k is a text generation model from MohammadKhosravi. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
This repository contains the Vanilla PrefixMemory-Tuning (PMT) baseline used in the master's thesis:
Downloads · 30 days
0
Access
Public
Updated Sep 13, 2026
Repo size
2.2 GB
Likes
0
Public
Click a slice to open those files.
.pt2.1 GB · 99%
From the Hugging Face model README
This repository contains the Vanilla PrefixMemory-Tuning (PMT) baseline used in the master's thesis:
Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation
The model uses meta-llama/Llama-3.1-8B-Instruct as a frozen backbone and augments each of its 32 transformer layers with a trainable external memory matrix.
Unlike the CEFR-conditioned PMT variants investigated in the thesis, this baseline contains no CEFR-specific embedding, gating mechanism, or architectural conditioning variable. The same PMT controller is shared across all six CEFR proficiency levels:
A1, A2, B1, B2, C1, and C2.
CEFR information is supplied only through the textual generation instruction.
Vanilla PMT introduces one full hidden-dimension trainable memory matrix for each transformer layer.
For Llama-3.1-8B-Instruct:
Number of transformer layers = 32
Hidden dimension = 4096
Memory per layer = 4096 × 4096
The complete trainable memory tensor therefore has shape:
[32, 4096, 4096]
corresponding to:
536,870,912 trainable parameters (536.87M)
The frozen attention query representation is transformed using an ELU feature map and multiplied by the corresponding layer-specific memory matrix.
Conceptually:
Hidden Representation
│
▼
Frozen Attention q_proj
│
▼
Query Representation Q
│
▼
ELU(Q)
│
▼
Layer Memory Matrix M_l
│
▼
ELU(Q) @ M_l
│
▼
Additive Memory-Derived Bias
│
▼
Self-Attention Output
The mechanism is applied to all 32 self-attention layers.
| Setting | Value |
|---|---|
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
| Backbone parameters | Frozen |
| Transformer layers | 32 |
| Hidden dimension | 4096 |
| Memory matrix per layer | 4096 × 4096 |
| Complete memory tensor | [32, 4096, 4096] |
| Trainable parameters | 536,870,912 |
| Trainable parameters (millions) | 536.87M |
| Feature map | ELU |
| CEFR-specific embeddings | None |
| CEFR-specific gating | None |
| Separate scaling scalar | None |
The trainable memory tensor can be represented as:
$$ M \in \mathbb{R}^{32 \times 4096 \times 4096} $$
with a layer-specific memory matrix:
$$ M^{(l)} \in \mathbb{R}^{4096 \times 4096}, \qquad l \in {0,\ldots,31} $$
The memory matrices are initialized using dimension-scaled random values.
For transformer layer (l), let the hidden representation be:
$$ H^{(l)} \in \mathbb{R}^{B \times T \times 4096} $$
The frozen attention query projection produces:
H^{(l)}W_{q}^{(l)} $$
where (W_q^{(l)}) belongs to the frozen Llama backbone.
An ELU feature map is then applied:
\operatorname{ELU}\left(Q^{(l)}\right) $$
The layer-specific external memory is retrieved through:
\phi\left(Q^{(l)}\right)M^{(l)} $$
where:
$$ M^{(l)} \in \mathbb{R}^{4096 \times 4096} $$
is the trainable PMT memory matrix for layer (l).
The resulting memory-derived representation is injected additively into the attention pathway.
Importantly, the memory matrix is shared across all CEFR targets. There is no CEFR-dependent architectural variable controlling the memory retrieval operation.
Vanilla PMT has no dedicated architectural mechanism for selecting a CEFR level.
Instead, the requested proficiency level is included in the textual instruction supplied to the frozen Llama backbone.
The experimental comparison can therefore be summarized as:
Shared PMT Adaptation
+
Textual CEFR Instruction
│
│
▼
vs.
│
▼
Explicit CEFR-Conditioned
PMT Architecture
This distinction makes Vanilla PMT useful for testing whether a large trainable external memory alone is sufficient for proficiency control, or whether the memory pathway must be explicitly conditioned on the requested CEFR level.
Training uses the Balanced CEFR Steering Subset derived from the EFCAMDAT Steering Training Dataset.
The dataset contains exactly 5,568 examples, balanced equally across the six CEFR levels:
| CEFR Level | Samples |
|---|---|
| A1 | 928 |
| A2 | 928 |
| B1 | 928 |
| B2 | 928 |
| C1 | 928 |
| C2 | 928 |
| Total | 5,568 |
A stratified 90/10 train-validation split is used with random_state=42:
| Partition | Samples |
|---|---|
| Training | 5,011 |
| Validation | 557 |
| Total | 5,568 |
Each training example contains:
Because Vanilla PMT has no CEFR-specific architectural control mechanism, the target CEFR level is explicitly included in the textual instruction.
The formatted prompt is concatenated with the reference response for causal language-model training.
Prompt and padding tokens are excluded from the loss using the standard -100 ignore index, so optimization is applied only to the target response tokens.
Training uses the standard autoregressive causal language-modeling cross-entropy objective.
Only the PMT memory matrices are optimized. The parameters of Llama-3.1-8B-Instruct remain frozen.
The target-only objective can be written as:
-\sum_{t \in \mathcal{T}{\text{target}}} \log p{\theta}(y_t \mid y_{<t},x) $$
where (\mathcal{T}_{\text{target}}) contains the response-token positions included in the loss.
| Setting | Value |
|---|---|
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
| Epochs | 3 |
| Optimizer | AdamW |
| Learning rate | 2e-4 |
| Weight decay | 0.01 |
| LR schedule | Cosine |
| Warmup ratio | 5% |
| Training batch size | 8 |
| Gradient accumulation steps | 2 |
| Effective training batch size | 16 |
| Validation batch size | 16 |
| Maximum sequence length | 512 |
| Controller precision | bfloat16 |
| Checkpoint criterion | Minimum validation loss |
The recorded training trajectory is:
| Epoch | Train Loss | Validation Loss | Validation Perplexity |
|---|---|---|---|
| 1 | 6.1827 | 5.5490 | 256.97 |
| 2 | 5.3515 | 5.2795 | 196.26 |
| 3 | 5.1446 | 5.2255 | 185.96 |
Validation loss improves at every epoch:
5.5490 → 5.2795 → 5.2255
Validation perplexity similarly decreases:
256.97 → 196.26 → 185.96
The Epoch 3 checkpoint is therefore retained as the final trained controller.
Training produced the following recorded profiling statistics:
| Metric | Value |
|---|---|
| Total training time | 1816.52 s |
| Total training time | 0.50 h |
| Peak GPU memory allocation | 46.30 GB |
| Average GPU utilization | 97.2% |
These measurements correspond to the complete three-epoch Vanilla PMT training run.
The trained Vanilla PMT controller is evaluated on the final In-Domain Evaluation Prompt Matrix used throughout the thesis.
The benchmark contains:
117 topics × 6 CEFR levels = 702 generation conditions
The controller is loaded together with the frozen Llama-3.1-8B-Instruct backbone and attached to the same 32 self-attention layers used during training.
| Setting | Value |
|---|---|
| Maximum new tokens | 200 |
| Temperature | 0.6 |
| Sampling | Enabled |
| Backbone | meta-llama/Llama-3.1-8B-Instruct |
Generated texts are evaluated using the primary CEFR evaluator:
MohammadKhosravi/roberta-large-cefr-classifier-JointLoss
The Vanilla PMT baseline obtains the following results over the complete 702-condition in-domain evaluation matrix:
| Metric | Result |
|---|---|
| Total generations | 702 |
| Strict Accuracy | 15.67% |
| Adjacent Accuracy | 47.58% |
| Mean Absolute Error (MAE) | 1.7308 |
| Mean Dependency Distance (MDD) | 1.73 |
| Mean Flesch Reading Ease | 98.00 |
| Mean Sentence-Level CEFR Drift | 2.24 levels |
Strict Accuracy requires the predicted CEFR level to exactly match the requested target level.
Adjacent Accuracy additionally treats predictions one CEFR level away from the requested target as correct.
For example:
Target B1:
B1 → Strict + Adjacent correct
A2 / B2 → Adjacent correct
A1 / C1 / C2 → Incorrect
CEFR levels are represented as ordered class indices:
A1 = 0
A2 = 1
B1 = 2
B2 = 3
C1 = 4
C2 = 5
MAE measures the average ordinal distance between the requested and predicted CEFR levels:
\frac{1}{N} \sum_{i=1}^{N} \left| \hat{y}_i-y_i \right| $$
Lower values indicate smaller proficiency-control errors.
The evaluation additionally reports:
These diagnostics supplement the primary CEFR-control metrics.
Vanilla PMT serves as the unconditioned PMT baseline.
It deliberately excludes the CEFR-specific components introduced by the proposed controller:
The same large PMT memory tensor is therefore shared across all target proficiency levels.
This comparison is used to investigate whether simply increasing adaptation capacity is sufficient for CEFR control, or whether explicit CEFR conditioning is necessary.
The Vanilla PMT controller contains approximately:
536.87M trainable parameters
This is substantially larger than generic parameter-efficient baselines such as LoRA and Standard Prefix-Tuning.
Despite this large adaptation capacity, Vanilla PMT achieves only:
15.67% Strict Accuracy
on the in-domain CEFR benchmark.
This makes Vanilla PMT an important experimental control.
Conceptually, the result separates two potential explanations for improved controllability:
CEFR Control Performance
│
┌───────────┴───────────┐
│ │
▼ ▼
Controller Capacity Explicit Conditioning
(More Parameters) (CEFR-Aware Control)
│ │
▼ ▼
Vanilla PMT tests Conditioned PMT tests
this explanation this explanation
The baseline therefore helps determine whether performance improvements originate from raw controller capacity or from the introduction of an explicit CEFR-conditioned control mechanism.
The complete training and evaluation notebook is available in the thesis GitHub repository at:
notebooks/04_generic_peft_baselines/vanilla_pmt_train_and_eval.ipynb
The balanced training subset is constructed in:
notebooks/01_data_preparation/balanced_cefr_steering_subset.ipynb
The in-domain evaluation benchmark is constructed in:
notebooks/01_data_preparation/efcamdat_preprocessing_and_partitioning.ipynb
The experimental notebook retains the original outputs, including:
This model is intended for research on:
Important: This repository contains an experimental research artifact and is not intended to serve as a standalone CEFR assessment model.
If you use this model or the accompanying experimental resources, please cite the associated master's thesis and repository.
@mastersthesis{khosravi2026beyond,
author = {Mohammad Khosravi},
title = {Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation},
school = {University of Padova},
year = {2026}
}