Downloads · 30 days
0
Artvv/lewm-qwen3-32b-token-similarity
lewm-qwen3-32b-token-similarity is a feature extraction model from Artvv. Use it when you need embeddings to search or compare text. It is set up for pytorch. The card lists the license as other.
This repository contains a LeWorldModel-style JEPA checkpoint trained on activation-derived observations from Qwen/Qwen3-32B.
Downloads · 30 days
0
Access
Public
Updated Apr 12, 2026
Repo size
53.6 MB
Likes
0
Public
Click a slice to open those files.
.pt40.2 MB · 75%
From the Hugging Face model README
This repository contains a LeWorldModel-style JEPA checkpoint trained on
activation-derived observations from Qwen/Qwen3-32B.
The model was trained on the dataset
Artvv/qwen3-32b-token-similarity-activation-triplets,
which contains 50,000 transition triplets:
obs_t : [3, 64, 64] float32 token-token similarity activation image
action_t : [256] float32 PCA projection of generated token embedding
obs_t+1 : [3, 64, 64] float32 next activation image
teacher : Qwen/Qwen3-32B
layers : [10, 20, 30]
encoding : token_similarity_v1
checkpoints/latest.pt: epoch-20 training checkpoint with model and optimizer state dicts.checkpoints/lewm_qwen3_epoch_20_object.ckpt: serialized epoch-20 model object.configs/qwen3_reasoning.yaml: training config.reports/train_summary.json: full epoch-by-epoch training metrics.reports/voe.json: first VoE report.reports/voe_calibrated.json: calibrated VoE report with temporal, intra-domain, and inter-domain controls.tensorboard/: TensorBoard event file from the run.Run setup:
examples : 50,000
train / val : 47,500 / 2,500
device : A100 80GB
epochs : 20
elapsed : 368.4 seconds
Final epoch metrics:
train_pred_loss = 0.03288
train_sigreg_loss = 3.64351
latent_model_mse = 0.02749
latent_identity_mse = 0.16838
latent_improvement_ratio = 6.12520
probe_auc_ovr = 0.99939
Decision at epoch 5:
continue_to_epoch_20
latent_improvement_ratio = 2.379 > 1.5
The calibrated violation-of-expectation evaluation compares the true temporal transition against mismatched next observations from the same domain and from different domains.
coherent_latent_mse_mean = 0.02708
intra_domain_latent_mse_mean = 1.62214
inter_domain_latent_mse_mean = 1.93429
intra_domain_voe_ratio = 59.90
inter_domain_voe_ratio = 71.43
coherent_image_mse_mean = 0.00518
intra_domain_image_ratio = 6.27
inter_domain_image_ratio = 6.94
The latent VoE ratios are much larger than the raw image-distance ratios, suggesting the model learned a transition-compatibility function beyond pixel distance alone. Same-domain mismatches remain highly surprising, so the current checkpoint should be interpreted as learning local trajectory compatibility, not yet architecture- or domain-level semantic invariance.
Clone the research code:
git clone https://github.com/ArthurVigier/CODE-JEPA.git
cd CODE-JEPA
python -m pip install -e .
Load the checkpoint with the LeWM-Qwen adapter code in third_party/le-wm:
from pathlib import Path
import sys
import torch
import yaml
root = Path("third_party/le-wm").resolve()
sys.path.insert(0, str(root))
from train_qwen3_reasoning import build_lewm_model
cfg = yaml.safe_load(Path("third_party/le-wm/config/train/qwen3_reasoning.yaml").read_text())
model = build_lewm_model(cfg, obs_channels=3, action_dim=256)
ckpt = torch.load("checkpoints/latest.pt", map_location="cpu")
model.load_state_dict(ckpt["model_state_dict"])
model.eval()
Qwen/Qwen3-32B; users must comply with the upstream model license and terms.If this artifact is useful, cite the CODE-JEPA / Activation-as-View repository and the upstream LeWorldModel work.