Llama-3.1-8B-ReLU

Llama-3.1-8B-ReLU

This repository contains a ReLU-fied variant of Llama 3.1 8B released as part of the ReLU-Tune project.

The model was created by replacing the transformer MLP activation with ReLU and then recovering performance through staged LoRA fine-tuning. The weights in this repository are provided as a merged checkpoint for straightforward loading and evaluation.

What This Model Is

• Base model: meta-llama/Llama-3.1-8B
• Architecture change: transformer MLP activations changed to ReLU
• Training method: staged LoRA fine-tuning
• Model format: merged dense model
• Extra metadata: activation_config.json documents the activation modification

Intended Use

This model is intended for:

• research on activation functions in large language models
• experiments on sparse activations and efficiency tradeoffs
• evaluation and fine-tuning workflows built on top of ReLU-Tune

This is primarily a research release, not a production-optimized instruction model.

Files

Typical files in this repository include:

• model.safetensors
• config.json
• generation_config.json
• tokenizer.json
• tokenizer_config.json
• activation_config.json

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bishmoy/Llama-3.1-8B-ReLU"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

Notes

• activation_config.json is included for reproducibility and to document how activations were modified.
• This release follows the base Llama 3.1 licensing and usage constraints. Users should review and comply with the original model terms.

Project

This model was produced with ReLU-Tune, a toolkit for:

• full or partial ReLU-fication
• staged LoRA fine-tuning
• benchmark and perplexity evaluation
• activation sparsity measurement

Limitations

This is an experimental model release. ReLU-fication changes the base architecture and can alter behavior in ways that differ from the original dense model. Downstream performance and robustness should be validated for any intended use.