Downloads · 30 days
7
15% of all-time downloads
m-vakili75/railway-lisa-7b-semantic-clip
railway-lisa-7b-semantic-clip is a image segmentation model from m-vakili75. Use it for the image segmentation task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as other.
This is the merged BF16 checkpoint for the Rail-finetuned LISA — semantic only row in Table 1 of the Railway Perception Foundation Model repository. It adapts LISA with three-class railway semantic supervision and ser…
Downloads · 30 days
7
15% of all-time downloads
All-time downloads
46
Public
Repo size
16.7 GB
Likes
0
Public
Click a slice to open those files.
.bin16.7 GB · 100%
From the Hugging Face model README
This is the merged BF16 checkpoint for the Rail-finetuned LISA — semantic only row in Table 1 of the Railway Perception Foundation Model repository. It adapts LISA with three-class railway semantic supervision and serves as the semantic-only comparison for the joint reasoning model.
This checkpoint requires the project repository and container. It is not a drop-in model for the generic Hugging Face pipeline() or AutoModel APIs.
Scores are percentages on the 2,000-image held-out evaluation described in the repository. CIoU pools intersection and union over each subset; GIoU is mean image-level IoU.
| Prompt | Scene subset | CIoU | GIoU |
|---|---|---|---|
| Reasoning-oriented | Switch-independent | 66.23 | 66.15 |
| Reasoning-oriented | Switch-dependent | 56.95 | 57.97 |
| Generic | Switch-independent | 65.92 | 66.03 |
| Generic | Switch-dependent | 57.21 | 58.18 |
Prompts:
By examining rail continuity and switch geometry, segment the active ego-route the train is following in this image.Segment the track bed in this image.This checkpoint received semantic supervision only and is not intended to produce the structured switch rationales supported by the joint model.
The root contains the merged LISA model, tokenizer, configuration, and weight shards. The tuned CLIP model and processor are stored separately under vision_tower/; the entire folder is required for inference.
Use chat_batch.py through the supplied demo_LISA.sbatch. The launcher accepts either one image or a flat image directory through TEST_IMAGE_SUBDIR.
Download the complete snapshot into a checkpoint directory whose model-folder name contains clip, then run:
# Replace the repository ID and host paths first.
hf download m-vakili75/railway-lisa-7b-semantic-clip \
--local-dir /path/to/checkpoints/railway-lisa-7b-semantic-clip
PROC_CKPT=/path/to/checkpoints \
PROC_CODE=/path/to/Railway_Perception_FoundationModel \
PROC_DATA=/path/to/data \
PROC_OUTPUT=/path/to/outputs \
MODEL_NAME=railway-lisa-7b-semantic-clip \
TEST_IMAGE_SUBDIR=test/images \
LISA_PROMPT='Segment the track bed in this image.' \
sbatch demo_LISA.sbatch
The launcher explicitly supplies both the model root and its sibling vision_tower/. Masks are saved under the selected output run; generated text is written to the Slurm log.
clip. Keep clip in the downloaded model path and use demo_LISA.sbatch, which passes that path explicitly.generation_config.json contains legacy BOS/EOS metadata. The supported chat_batch.py path replaces it at runtime from the bundled tokenizer under Transformers 4.31, yielding BOS 1, EOS 2, and PAD 0. Direct generation through a different loader or Transformers version is not validated.chat_batch.py mask inference.The checkpoint is a research artifact for railway ego-route segmentation and controlled comparison with the joint semantic-and-reasoning checkpoint. It may fail under domain shift, poor visibility, unusual track layouts, occlusion, or ambiguous switch geometry. Do not use its outputs for safety-critical railway control or operational decision-making.
This railway checkpoint was fine-tuned from LISA-7B-v1. The LISA documentation identifies LLaVA-Lightning-7B-v1-1 as the base used for its 7B model. That LLaVA model belongs to the original LLaMA/Vicuna/LLaVA lineage: its upstream model card describes a LLaMA/Vicuna-based model whose delta weights must be applied to the original LLaMA weights. LISA uses Segment Anything (SAM) as its segmentation component and OpenAI CLIP ViT-L/14 as its vision encoder.
RailSem19 imagery and semantic annotations were used as research/training data for the railway adaptation. This Hugging Face model repository contains trained checkpoint files and reported model results only. It does not redistribute RailSem19 images, annotations, sparse metadata, dense metadata, or meta-images. This model release does not claim ownership of, or relicense, any RailSem19 material. RailSem19 remains governed by its official license agreement.
LICENSES/LLaMA-LICENSE.txt. Its restrictions continue to apply to this derivative checkpoint.See THIRD_PARTY_NOTICES.md for component attribution, exact source links, and the required original LLaMA notice.
This checkpoint is released for non-commercial research use only and remains subject to the applicable licenses and terms of its upstream components, including the original LLaMA/Vicuna/LLaVA lineage used by LISA-7B-v1. The checkpoint itself is not licensed under Apache-2.0. Commercial use is not granted by this model card.
Where stated, the Railway Perception Foundation Model project licenses its source code under Apache License 2.0. That code license applies only to source code covered by it; it does not license these model weights as Apache-2.0 and does not override upstream model or dataset terms. Users are responsible for complying with all applicable licenses and terms when using or redistributing this checkpoint.
This model card summarizes the known upstream licensing and provenance of the checkpoint and is not legal advice. Users are responsible for ensuring that their use and redistribution comply with all applicable upstream licenses and terms.
@article{ghorbanalivakili2026reasoning,
author = {Ghorbanalivakili, Mohammadjavad and Varghese, Ashley and Sohn, Gunho},
title = {Reasoning-guided Ego-path Segmentation for Autonomous Trains using Vision-language Models},
journal = {The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences},
volume = {XLIX-B3-2026},
pages = {89--96},
year = {2026},
doi = {10.5194/isprs-archives-XLIX-B3-2026-89-2026}
}