multimodal-fusion-demo

Multimodal Fusion

A multi-modal fusion model combining image and text features.

Model Details

• Model Type: multimodal-fusion
• Task: multimodal-fusion
• Framework: PyTorch Lightning → Hugging Face Transformers

Usage

from transformers import AutoModel
import torch

# Load model
model = AutoModel.from_pretrained("golyuval/multimodal-fusion-demo", trust_remote_code=True)
model.eval()

# Prepare inputs
image_features = torch.randn(1, 3, 224, 224)
text_features = torch.randint(0, 1000, (1, 32))

# Inference
with torch.no_grad():
    outputs = model(image_features=image_features, text_features=text_features)
    
print(outputs)

Configuration

{
  "image_dim": 512,
  "text_dim": 256,
  "output_dim": 3,
  "lr": 0.001,
  "training": true,
  "model_type": "multimodal-fusion",
  "auto_map": {
    "AutoConfig": "modeling_multimodal_fusion.MultimodalFusionConfig",
    "AutoModel": "modeling_multimodal_fusion.MultimodalFusionForHF",
    "AutoModelForSequenceClassification": "modeling_multimodal_fusion.MultimodalFusionForHF"
  }
}

Model Info

• Total Parameters: 115,459
• Trainable Parameters: 115,459

Preprocessing

• Image Dim: 512
• Text Dim: 256
• Output Dim: 3

Citation

@misc{multimodal_fusion,
  title = {Multimodal Fusion},
  author = {Your Name},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/golyuval/multimodal-fusion-demo}}
}