Downloads · 30 days
0
simlingo-predictive-fusion
nacefmbarek/simlingo-predictive-fusion
simlingo-predictive-fusion is a robotics model from nacefmbarek. Use it for the robotics task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
InternVL2-1B (InternViT-300M + Qwen2-0.5B, LoRA r=32 α=64) camera-VLA driving policy for CARLA, with a two-stage cross-attention fusion where InternViT vision tokens are the query:
Access
Public
Updated Aug 25, 2026
Repo size
8.8 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.ckpt8.8 GB · 100%
At a glance
- Task
- Robotics
- License
- mit
- Access
- Public
- Created
- Aug 25, 2026
- Updated
- Aug 25, 2026
- SHA
- 36f1a5e7
Model card
From the Hugging Face model README
SimLingo — Predictive Fusion (fdrop03)
InternVL2-1B (InternViT-300M + Qwen2-0.5B, LoRA r=32 α=64) camera-VLA driving policy for CARLA, with a two-stage cross-attention fusion where InternViT vision tokens are the query:
- Stage 1 — KV = 360° LiDAR → BEV occupancy histogram (current scene).
- Stage 2 — KV = DeltaTok-forecast future embedding — a frozen DINOv3 ViT-B/16 tokenizer plus a trained Predictor that forecasts the next delta token without seeing the future frame.
Enriched tokens replace <IMG_CONTEXT> in place. Training code, config, and the data-collection
pipeline: github.com/nacef112/kaut-project. Built on
carla_garage and
SimLingo.
Recipe
route_as | target_point_command — GPS target point and the language command |
predict_route_as_wps | True |
speed_wps_mode | 2d |
fusion_dropout_prob | 0.3 — withholds BEV and forecast TOGETHER, per sample, train-only. Forces the target-point/command path to carry gradient instead of only the fusion shortcut. |
| forecast | single horizon (not multi-horizon) |
num_context_frames / delta_frame_gap | 4 / 1 |
| DeltaTok tokenizer | included in this repo (deltatok_best_b2dheldout.ckpt) |
Files
| file | size | notes |
|---|---|---|
fdrop03_step050000_weights.ckpt | 3.2 GB | epoch=2, step=50000 — the warm-start every downstream checkpoint in this project line continued from. Use this one unless you specifically want the later epoch. |
fdrop03_final_epoch003_weights.ckpt | 3.2 GB | epoch=3, step=82840 — further trained, same recipe |
deltatok_best_b2dheldout.ckpt | 2.0 GB | DeltaTok forecast tokenizer (single-horizon) that Stage-2 was trained against |
SHA256SUMS.txt | — | integrity check |
Both weight files load state_dict only (optimizer/scheduler state stripped), verified to load
strict=True into a freshly constructed model — 0 missing, 0 unexpected keys, 1317.6M params.
Load
import torch
sd = torch.load("fdrop03_step050000_weights.ckpt", map_location="cpu", weights_only=False)
state_dict = sd.get("state_dict", sd)
# model = hydra.utils.instantiate(cfg.model, ...) -- see the training repo for cfg construction
model.load_state_dict(state_dict, strict=True)
The model class (DrivingModel) and its Hydra config live in the training repo linked above —
this is weights only, not a standalone inference package. DINOV3_BACKBONE env var overrides
the (gated) facebook/dinov3-vitb16-pretrain-lvd1689m backbone path if you have a local copy;
see the training repo's README for requesting access.
What this is not
Not a multi-horizon, trimodal-fusion, LoRA-only-SFT, or CoT/adaptive-CoT checkpoint — those are separate lines built on top of this same architecture, not included here.
- Task
- Robotics
- License
- mit
- Created
- Aug 25, 2026
- Updated
- Aug 25, 2026