MGEMMV-Llama-11b

MGEMMV: A Multimodal LLM Framework for GEMM Verilog Generation from Circuit Diagrams

<img src="image/overview.jpg" style="zoom:80%;" />

MGEMMV is an open-source multimodal framework designed for Verilog generation of GEMM modules.
It provides:

• Hierarchical Multimodal Dataset: automatically generated circuit-diagram–Verilog pairs, covering both basic logic and GEMM-level modules with diverse hardware optimization techniques (HOTs).
• Automated Evaluation Framework: scalable pipeline for syntax and functionality correctness verification of LLM-generated designs.

Experiments show syntax correctness up to 94.6% and functionality correctness up to 90.0%, significantly surpassing existing baselines.

Models and Datasets

Quick Start

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForVision2Seq

# Load the model and processor
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model_name = "/home/python/transformers/GEMMV-MultiModal-8b"
processor = AutoProcessor.from_pretrained(model_name)
model = AutoModelForVision2Seq.from_pretrained(model_name).to(device)

# Load input image (circuit diagram)
image = Image.open("example_gemm_diagram.jpg").convert("RGB")

# Define the prompt/question
prompt = "Generate Verilog code for the circuit diagram."

# Preprocess input
inputs = processor(text=prompt, images=image, return_tensors="pt").to(device)

# Generate Verilog code
outputs = model.generate(
    **inputs,
    max_length=4096,
    temperature=0.8,
    top_p=0.9,
    do_sample=True
)

# Decode output and ensure Verilog ends properly
response = processor.batch_decode(outputs, skip_special_tokens=True)[0]
if not response.strip().endswith("endmodule"):
    response += "\nendmodule"

print("Generated Verilog:\n", response)

@ARTICLE{3648843,
  author={Zhang, Gaoche and Wang, Meiqi and Wang, Zhongfeng},
  journal={IEEE Transactions on Circuits and Systems I: Regular Papers},
  title={MGEMMV: A Multimodal LLM Framework for GEMM Verilog Generation From Circuit Diagrams},
  year={2026},
  volume={},
  number={},
  pages={},
  keywords={Circuits;Hardware design languages;Hardware;Codes;Benchmark testing;Syntactics;AI accelerators;Optimization;Logic;Integrated circuit modeling;Electronic design automation (EDA);MLLMs;fine-tuning;GEMM;multimodal dataset},
  doi={10.1109/TCSI.2025.3648843}
}