llama3.1-8b-cefr-pt-537m-param-matched

Llama-3.1-8B CEFR Multi-Prefix Tuning — 537M PMT-Matched Variant

This repository contains the ~537M CEFR Multi-Prefix Tuning model used in the master's thesis:

Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation

The model uses meta-llama/Llama-3.1-8B-Instruct as a frozen backbone and introduces six CEFR-specific prefix banks, corresponding to:

A1, A2, B1, B2, C1, and C2.

This configuration is designed as a parameter-matched control against Vanilla PrefixMemory-Tuning (PMT).

Its purpose is to compare two controllers with nearly identical trainable parameter budgets while differing fundamentally in whether CEFR conditioning is explicitly encoded in the architecture.

---

Experimental Purpose

The key parameter comparison is:

The trainable parameter difference is below 0.1%.

This creates a tightly controlled comparison between:

Vanilla PMT
536.87M parameters
       │
       ▼
Shared Unconditioned Memory
+ Textual CEFR Instruction

              vs.

CEFR Multi-Prefix
536.70M parameters
       │
       ▼
CEFR-Specific Prefix Banks
+ Architectural CEFR Conditioning

Because the trainable capacities are effectively matched, the experiment tests whether explicit CEFR conditioning provides stronger proficiency control than raw controller capacity alone.

---

Model Architecture

The controller contains six independent CEFR-specific prefix embedding banks.

Each CEFR level contains:

30 virtual tokens

giving:

6 CEFR levels × 30 virtual tokens
                =
180 CEFR-specific prefix embeddings

Each virtual-token embedding has dimension:

4096

The selected prefix embeddings are processed by a high-capacity shared projection network:

4096 → 7696 → 65536

with a Tanh activation between the two linear transformations.

The resulting representation is reshaped into layer-wise key/value prefix states for the frozen 32-layer Llama-3.1-8B-Instruct backbone.

---

High-Level Architecture

                         Target CEFR
                             │
                             ▼
              ┌──────────────────────────┐
              │ CEFR-Specific Prefix Bank│
              │   A1 A2 B1 B2 C1 C2     │
              └────────────┬─────────────┘
                           │
                           ▼
                  30 Virtual Tokens
                           │
                           ▼
              ┌──────────────────────────┐
              │ Shared Projection MLP    │
              │ 4096 → 7696 → 65536     │
              │        Tanh              │
              └────────────┬─────────────┘
                           │
                           ▼
                 Per-Layer Prefix
                  Key/Value States
                           │
                           ▼
              ┌──────────────────────────┐
              │ Frozen Llama-3.1-8B     │
              │      32 Layers          │
              └──────────────────────────┘

---

Architecture Summary

The training notebook explicitly verifies:

CEFR Multi-Prefix : 536,698,384 parameters
Vanilla PMT       : 536,870,912 parameters
Difference        :    -172,528 parameters

---

CEFR Conditioning

The model uses CEFR-specific prefix selection.

Each requested target proficiency level selects one of six independent prefix banks:

A1
A2
B1
B2
C1
C2

Each bank contains 30 learned virtual tokens.

The control flow is:

Target CEFR
    │
    ▼
  cefr_id
    │
    ▼
Select CEFR-Specific
Prefix Bank
    │
    ▼
30 Level-Specific
Virtual Tokens
    │
    ▼
Shared Projection MLP
    │
    ▼
Layer-Wise Prefix
Key/Value States
    │
    ▼
Frozen Llama-3.1-8B

Importantly, this model does not use the multiplicative CEFR-gating mechanism introduced by the later CEFR-gated PMT architecture.

The CEFR control mechanism here is prefix-bank selection.

---

Textual Prompt

The specific target CEFR class is not directly inserted into the textual prompt.

Instead:

Target CEFR
     │
     ▼
   cefr_id
     │
     ▼
CEFR-Specific Prefix Bank
     │
     ▼
Architectural Conditioning

The textual prompt still contains general proficiency-oriented guidance, but the requested class itself—such as A1, B2, or C2—is supplied through the CEFR-specific prefix controller.

This separates the target CEFR signal from direct textual label insertion.

---

Training Dataset

Training uses the Balanced CEFR Steering Subset derived from the EFCAMDAT Steering Training Dataset.

The dataset contains exactly 5,568 examples, balanced equally across all six CEFR levels:

A stratified 90/10 train-validation split is used with random_state=42.

---

Training Objective

Training uses the standard autoregressive causal language-modeling cross-entropy objective.

Prompt and padding tokens are excluded from the loss using the standard -100 ignore index.

Only the CEFR Multi-Prefix controller is optimized.

The Llama-3.1-8B-Instruct backbone remains frozen throughout training.

The objective can be expressed as:

$$ \mathcal{L}_{\mathrm{CLM}}

-\sum_{t \in \mathcal{T}{\mathrm{target}}} \log p{\theta} \left( y_t \mid y_{<t},x,c \right) $$

where:

• (x) is the textual prompt;
• (c) represents the CEFR-specific prefix selection;
• (y_t) is a target response token;
• (\mathcal{T}_{\mathrm{target}}) contains only target-response positions included in the loss.

---

Training Configuration

---

Training Progression

The recorded training trajectory is:

Validation performance improves across all three epochs.

Validation loss follows:

2.6245 → 2.4497 → 2.4371

while validation perplexity follows:

13.80 → 11.58 → 11.44

The Epoch 3 checkpoint is therefore retained as the final model.

---

Hardware Profiling

The complete three-epoch training run produced the following hardware measurements:

These values correspond to the complete training run.

---

Model Checkpoint

The selected controller checkpoint is stored as:

cefr_prefix_tuning_537m_best_weights.pt

This checkpoint corresponds to the lowest validation loss observed during training.

---

Generation Evaluation

The trained controller is evaluated on the final In-Domain Evaluation Prompt Matrix used throughout the thesis.

The benchmark contains:

117 topics × 6 CEFR levels = 702 generation conditions

The evaluation pipeline is:

Input Topic
    │
    ▼
CEFR-Blind Textual Prompt
    │
    ├────────────────────────────┐
    │                            │
    ▼                            ▼
Text Tokens                  Target CEFR
                                 │
                                 ▼
                              cefr_id
                                 │
                                 ▼
                       CEFR-Specific Prefix
                                 │
    ┌────────────────────────────┘
    ▼
Frozen Llama-3.1-8B
    │
    ▼
Generated Response
    │
    ▼
CEFR Evaluator

For each generation condition:

1. The topic prompt is constructed.
2. The requested CEFR level is mapped to a class ID.
3. The corresponding CEFR-specific prefix bank is selected.
4. The controller constructs layer-wise key/value prefix states.
5. The frozen Llama-3.1-8B-Instruct backbone generates the response.

---

Generation Configuration

---

CEFR Evaluator

Generated texts are evaluated using the primary thesis CEFR evaluator:

MohammadKhosravi/roberta-large-cefr-classifier-JointLoss

The evaluator is independent of the generation controller and is not used to train the CEFR Multi-Prefix model.

---

Evaluation Results

The ~537M CEFR Multi-Prefix model obtains the following results over all 702 in-domain generation conditions:

---

Metric Interpretation

Strict Accuracy

Strict Accuracy requires the predicted CEFR class to exactly match the requested target level.

For example:

Target B1:

B1       → Correct
A2 / B2  → Incorrect
A1/C1/C2 → Incorrect

---

Adjacent Accuracy

Adjacent Accuracy additionally accepts predictions one neighboring CEFR level away from the target.

For example:

Target B1:

B1           → Strict + Adjacent correct
A2 / B2      → Adjacent correct
A1 / C1 / C2 → Incorrect

---

Mean Absolute Error

CEFR classes are represented as ordered indices:

A1 = 0
A2 = 1
B1 = 2
B2 = 3
C1 = 4
C2 = 5

MAE measures the average ordinal distance between requested and predicted levels:

$$ \operatorname{MAE}

\frac{1}{N} \sum_{i=1}^{N} \left| \hat{y}_i-y_i \right| $$

Lower values indicate smaller CEFR-control errors.

---

Linguistic Diagnostics

The evaluation additionally records:

• Mean Dependency Distance (MDD)
• Flesch Reading Ease
• Sentence-Level CEFR Drift
• Per-level CEFR performance
• Classification reports
• Confusion matrices

These diagnostics complement the primary CEFR-control metrics.

---

Capacity-Scaling Results

This model is the highest-capacity CEFR Multi-Prefix configuration in the controller-scaling study.

The progression is:

~68M
45.44%
   │
   ▼
~286M
52.42%
   │
   ▼
~537M
64.10%

From ~68M to ~286M:

+6.98 percentage points

From ~286M to ~537M:

+11.68 percentage points

From ~68M to ~537M:

45.44% → 64.10%

corresponding to an absolute improvement of:

+18.66 percentage points

and a relative increase of approximately:

+41.1%

Within this experimental setup, CEFR-control performance therefore increases across the three controller-capacity configurations.

---

PMT-Matched Comparison

The central controlled comparison is:

The trainable parameter difference is only:

−172,528 parameters (−0.0321%)

Yet Strict Accuracy changes from:

15.67% → 64.10%

an absolute increase of:

+48.43 percentage points

The CEFR Multi-Prefix model achieves approximately:

4.09× the Strict Accuracy

of Vanilla PMT.

Adjacent Accuracy improves by:

47.58% → 81.48%

or:

+33.90 percentage points

MAE decreases from:

1.7308 → 0.6966

corresponding to a reduction of approximately:

59.8%

Because the two controllers have nearly identical trainable parameter budgets, this comparison provides strong evidence within this experimental setting that the performance difference cannot be explained by parameter count alone.

Instead, the result supports the importance of explicit CEFR-conditioned architectural control.

---

Comparison with Other Baselines

These comparisons should be interpreted within the common experimental setup used throughout the thesis.

---

Role in the Thesis

This model serves two experimental purposes.

1. Highest-Capacity Multi-Prefix Configuration

It completes the controller-capacity scaling experiment:

CEFR Multi-Prefix — ~68M
        │
        ▼
CEFR Multi-Prefix — ~286M
        │
        ▼
CEFR Multi-Prefix — ~537M

This tests how CEFR-control performance changes as controller capacity increases while the underlying CEFR-specific prefix-selection mechanism is preserved.

2. Parameter-Matched Control Against Vanilla PMT

The model also enables a near-exact capacity-controlled comparison:

                 ≈537M Parameters
                        │
            ┌───────────┴───────────┐
            │                       │
            ▼                       ▼
       Vanilla PMT           CEFR Multi-Prefix
            │                       │
            ▼                       ▼
 Shared Unconditioned        CEFR-Specific
 External Memory             Prefix Selection
            │                       │
            ▼                       ▼
      15.67% Strict           64.10% Strict

This comparison isolates the effect of explicit CEFR conditioning from the effect of trainable capacity.

The model therefore plays a central role in distinguishing:

1. Controller capacity
2. Adaptation architecture
3. Explicit CEFR conditioning

---

Reproducibility

The complete training and evaluation notebook is available in the thesis GitHub repository at:

notebooks/05_cefr_prefix_tuning/cefr_prefix_tuning_537m_train_and_eval.ipynb

The balanced training subset is constructed in:

notebooks/01_data_preparation/balanced_cefr_steering_subset.ipynb

The in-domain evaluation matrix is constructed in:

notebooks/01_data_preparation/efcamdat_preprocessing_and_partitioning.ipynb

The notebook retains the original experimental outputs, including:

• Parameter-matching verification
• Train/validation split statistics
• Epoch-level training and validation losses
• Validation perplexity
• Hardware-profiling statistics
• Generated responses
• CEFR-control metrics
• Per-level results
• Classification reports
• Confusion matrices

---

Intended Use

This model is intended for research on:

• CEFR-controlled text generation
• Prefix-Tuning
• Multi-prefix conditioning
• Parameter-matched control experiments
• Controller-capacity scaling
• Language-proficiency-conditioned generation
• Controllable language generation

Important: This model is an experimental research artifact and is not intended to serve as a standalone CEFR assessment model.

---

Citation

If you use this model or the accompanying experimental resources, please cite the associated master's thesis and repository.

@mastersthesis{khosravi2026beyond,
  author = {Mohammad Khosravi},
  title = {Beyond Prompting: Resource-Efficient Explicit Control for CEFR-Aligned Language Generation},
  school = {University of Padova},
  year = {2026}
}