semsimula-parflm-multixi-structured-vtheta

Multi-Xi PARFLM with Structured V_theta (SQ3 Mixture of Quadratic Wells)

A structured variant of the Multi-Xi PARFLM conservative language model in which the MLP scalar potential \(V_\theta\) is replaced by a mixture of K=8 diagonal quadratic wells (SQ3), while retaining the full MLP-based pairwise potential \(V_\phi\). This replacement yields:

• Near-zero PPL cost — 12.27 PPL vs the MLP baseline's 12.10 PPL (only 0.17 PPL gap, 0.6% excess cross-entropy). The pairwise \(V_\phi\) absorbs nearly all of the nonlinear structure that the quadratic wells cannot express.
• Full analytical gradients for \(V_\theta\) — no torch.autograd.grad needed for the scalar potential force, eliminating the second-order computation graph for the \(V_\theta\) component
• Explicit attractor centres — the 8 semantic attractors \(\mu_k(\xi)\) are readable directly from the model parameters, with no gradient-descent extraction required
• Compressed \(V_\theta\) landscape — with \(V_\phi\) carrying the bulk of the force budget, \(V_\theta\) collapses to a near-flat bias field (mean 0.02, range 31.4 vs 644.9 for SPLM)

This model is from the Semantic Simulation framework.

Table of Contents

• When to Use This Model
• Architecture
• Structured V_theta with Pairwise V_phi: Why the Gap Nearly Vanishes
• How to Get Started
• Training Details
• Evaluation Results
• SPLM Family Overview
• Bias, Risks, and Limitations
• Citation

When to Use This Model

Choose this structured variant over the MLP-based Multi-Xi PARFLM when:

Bottom line: for PARFLM, structured \(V_\theta\) is the recommended default — the 0.17 PPL gap is negligible, while the interpretability and speed gains are substantial.

Architecture

Input tokens x_1, ..., x_T
       |
   Embedding E[x] + positional encoding
       |
   For each of L=8 integration steps:
       |
       +-- K-EMA channels: xi^(k)_t = causal_ema(h, alpha_k)   [K_xi=4 channels]
       |
       +-- Structured V_theta (SQ3):
       |     xi_flat = flatten(xi_1..xi_K)                      [K_xi * d = 1024]
       |     V = -tau * logsumexp_k(-E_k/tau + log pi_k)        [K_mix=8 wells]
       |     f_theta = -analytical_grad_h V                     [closed-form]
       |
       +-- Pairwise V_phi (competitive structural MLP):
       |     scores = score_net(h_t, h_s)                       [for all s <= t]
       |     top-k selection via Gumbel-softmax                 [k=8 neighbours]
       |     f_phi = -grad_h V_phi(h_t, h_s)                   [autograd, sparse]
       |
       +-- Total force: f = f_theta + f_phi
       |
       +-- Damped Stormer-Verlet step: h += (h-h_prev)/(1+dt*gamma) + dt^2*f/(m*(1+dt*gamma))
       |
       +-- LayerNorm(h)
       |
   Logits = h @ E^T                                            [tied embeddings]

Effective damping. The nominal gamma = 0.30 overstates the true dissipation. The LayerNorm applied after each integration step rescales the hidden state, absorbing most of the velocity decay. The dynamics are therefore heavily underdamped even at this nominal value. Gamma-sweep experiments on the OpenWebText-scale Fock-PARFLM variant confirm that the effective damping is much smaller than the nominal coefficient.

Structured V_theta with Pairwise V_phi: Why the Gap Nearly Vanishes

In the standalone Multi-Xi SPLM, replacing the MLP \(V_\theta\) with SQ3 incurs a 1.82 PPL gap (13.33 vs 11.51). In PARFLM, the same replacement incurs only a 0.17 PPL gap (12.27 vs 12.10) --- an order of magnitude smaller.

The reason is landscape compression: with \(V_\phi\) carrying the bulk of the force budget, \(V_\theta\) collapses to a near-flat bias field:

The MLP's expressivity advantage is irrelevant when \(V_\theta\)'s dynamic range is negligible. The pairwise \(V_\phi\) absorbs the nonlinear structure that quadratic wells cannot express, making structured \(V_\theta\) essentially free in PARFLM.

The force from the structured \(V_\theta\) is computed in closed form:

$$f_\theta = -\nabla_h V_\theta = -\sum_{k=1}^{K} q_k(\xi, h) \cdot a_k(\xi) \odot (h - \mu_k(\xi))$$

where \(q_k\) are the softmax responsibilities over the 8 quadratic wells. The \(V_\phi\) force still uses autograd (sparse, over top-k=8 neighbours only).

For full derivations, all four structured variants (SQ1--SQ4), landscape compression analysis, attractor basin decoding, and hyperparameter selection strategies, see the companion note: Structured_VTheta_Design_and_Theory.md.

How to Get Started

import torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from parf.model_parf_multixi import MultiXiPARFLM, MultiXiPARFConfig
from parf.model_structured_vtheta import MixtureQuadraticVTheta
from parf.model_structured_vtheta_multixi import StructuredVThetaMultiXiAdapter

# -- Build base model --
config = MultiXiPARFConfig(
    vocab_size=50257, d=256, n_layers=8,
    v_hidden=1024, v_depth=3,
    max_len=1024, block_size=512,
    gamma=0.30, xi_channels=4,
    xi_alpha_inits=[0.25, 0.5, 0.75, 0.95],
    xi_learnable=True, mass_mode="logfreq",
    logfreq_path="logfreq_surprisal_tinystories.npy",
    v_phi_kind="structural_competitive",
    v_phi_phi_hidden=128, v_phi_theta_hidden=128,
    top_k=8, score_head_hidden=32,
    gumbel_tau_init=1.0, gumbel_tau_min=0.3,
    gumbel_noise=True,
    use_gathered_v_phi=True,
    use_layer_checkpoint=True,
    ln_before_distance=True,
    per_layer_v_phi_scale=True,
)
model = MultiXiPARFLM(config)

# -- Swap in structured V_theta --
K_xi, d = 4, 256
inner = MixtureQuadraticVTheta(d=K_xi * d, K=8, tau=1.0)
model.V_theta = StructuredVThetaMultiXiAdapter(inner, K=K_xi, d=d)

# -- Load checkpoint --
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
    repo_id="dimitarpg13/semsimula-parflm-multixi-structured-vtheta",
    filename="checkpoint/ckpt_best.pt",
)
state = torch.load(ckpt_path, map_location="cpu")
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# -- Read attractor centres directly --
x = torch.randint(0, 50257, (1, 64))
with torch.no_grad():
    h = model._embed(x)
    xis = model._compute_xis(h)                        # (1, 64, 4, 256)
    centres = model.V_theta.attractor_centres(xis)      # (1, 64, 8, 1024)
    print(f"Attractor centres shape: {centres.shape}")   # 8 basins per token

Available Artifacts

Training Details

Training Data

TinyStories --- a synthetic corpus of short children's stories generated by GPT-3.5/4, tokenized with GPT-2 BPE (vocab size 50,257). Training cap: 5M tokens; validation: ~140k tokens.

Training Procedure

The base model architecture is identical to the Multi-Xi PARFLM. The only modification is the \(V_\theta\) replacement: the 3-layer MLP is swapped for the SQ3 mixture at model construction time, before training begins from scratch. The pairwise \(V_\phi\) (competitive structural MLP, hidden=128, top-k=8) is unchanged.

Training Script

notebooks/conservative_arch/scaleup/colab_parf_multixi_structured_vtheta.ipynb --- Colab notebook with structured V_theta arms, GDrive output, checkpointing, and live progress display.

Evaluation Results

TinyStories Validation Perplexity

V_theta Landscape Statistics

Learned Xi-Channel Decay Rates

The final learned alpha values \([\alpha_1, \ldots, \alpha_4] = [0.11, 0.55, 0.81, 0.97]\) are nearly identical to both the SPLM structured V_theta and the MLP PARFLM baseline, confirming that the causal EMA context structure is robust to the \(V_\theta\) parameterisation change.

SPLM Family Overview

This model is part of the Semantic Simulation SPLM family:

Collection: Semantic Simulation SPLM Model Family

Bias, Risks, and Limitations

• Research checkpoint only. This model is a proof-of-concept for structured scalar potentials in pairwise-force architectures, not a production system.
• TinyStories only. Trained exclusively on synthetic children's stories (~5M tokens). Not suitable for general-purpose language generation.
• English only. No multilingual capability.
• Small scale. 17.3M parameters, 256-dim hidden states.
• No safety training. No RLHF, DPO, or safety filtering has been applied.
• V_phi still uses autograd. Only the \(V_\theta\) gradient is analytical; the \(V_\phi\) pairwise force still requires torch.autograd.grad. The overall training speedup is therefore partial (\(V_\phi\) dominates the cost).

Citation

@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v15 (Jun 7, 2026).
               Companion code repository (DOI 10.5281/zenodo.20579561):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}

Environmental Impact

• Hardware: NVIDIA A100 40GB (Google Colab)
• Training time: ~2.5 hours (16,000 steps, A2 arm)
• Carbon footprint: Estimated less than 1.5 kg CO2

Correction (2026-09-27) — integrator name. This model's layer step was described here as a damped Euler step. It is not. The update is h_new = h + (h - h_prev)/(1 + dt*gamma) + dt^2*f/(m*(1 + dt*gamma)), which carries no velocity state at all: the velocity is the position difference h - h_prev. Undamped this is h_{n+1} = 2*h_n - h_{n-1} + dt^2*f/m, i.e. Störmer–Verlet in position form. It is also not velocity-Verlet, which carries an explicit velocity through half-kick/drift/half-kick. The distinction matters because the \(\omega \cdot dt < 2\) stability wall is the Störmer/leapfrog bound and applies to this lineage; the semi-implicit Euler models in the same collection were never subject to it. Only the name was wrong — no measurement on this card changes.