magic-bert-50m-roformer-classification

Magic-BERT 50M RoFormer Classification

A RoFormer-based transformer model fine-tuned for binary file type classification. This model achieves 93.7% classification accuracy across 106 MIME types, making it the recommended choice for production file type detection.

Why Not Just Use libmagic?

For intact files starting at byte 0, libmagic works well. But libmagic matches signatures at fixed offsets. Magic-BERT learns structural patterns throughout the file, enabling use cases where you don't have clean file boundaries:

• Network streams: Classifying packet payloads mid-connection, before headers arrive
• Disk forensics: Identifying file types during carving, when scanning raw disk images without filesystem metadata
• Fragment analysis: Working with partial files, slack space, or corrupted data
• Adversarial contexts: Detecting file types when magic bytes are stripped, spoofed, or deliberately misleading

Model Description

This model extends magic-bert-50m-roformer-mlm with contrastive learning fine-tuning. It uses Rotary Position Embeddings (RoPE) and produces highly discriminative embeddings for file type classification.

Tokenizer

The tokenizer uses the Binary BPE methodology introduced in Bommarito (2025). The original Binary BPE tokenizers (available at mjbommar/binary-tokenizer-001-64k) were trained exclusively on executable binaries (ELF, PE, Mach-O). This tokenizer uses the same BPE training approach but was trained on a diverse corpus spanning 106 file types.

Intended Uses

Primary use cases:

• Production file type classification
• MIME type detection from binary content
• Embedding-based file similarity search
• Security analysis and content filtering

This is the recommended model for file classification tasks due to its combination of high accuracy (93.7%) and parameter efficiency (42M parameters).

Detailed Use Cases

Network Traffic Analysis

When inspecting packet payloads, you often see file data mid-stream—TCP reassembly may give you bytes 1500-3000 of a PDF before you ever see byte 0. Traditional signature matching fails here. Classification embeddings can identify file types from interior content.

Disk Forensics & File Carving

During disk image analysis, you scan raw bytes looking for file boundaries. Tools like Scalpel rely on header/footer signatures, but many files lack clear footers. This model can score byte ranges for file type probability, helping identify carved fragments or validate carving results.

Incident Response

Malware often strips or modifies magic bytes to evade detection. Polyglot files (valid as multiple types) exploit signature-based tools. Learning structural patterns provides a second opinion that doesn't rely solely on the first few bytes.

Similarity Search

The embedding space (256-dimensional, L2-normalized) enables similarity search across file collections: "find files structurally similar to this sample" for malware clustering, duplicate detection, or content-based retrieval.

Architecture: RoPE vs Absolute Position Embeddings

This model uses Rotary Position Embeddings (RoPE), which encode position through rotation matrices in attention. This differs from the Magic-BERT variant which uses absolute position embeddings.

This model achieves higher classification accuracy with fewer parameters, making it the preferred choice for production deployment when only classification is needed.

MLM vs Classification: Two-Phase Training

This is the Phase 2 (Classification) model built on RoFormer. The training pipeline has two phases:

Two-Phase Training

Phase 2 specifics:

• Frozen: Embeddings + first 4 transformer layers
• Learning rate: 100x lower than Phase 1
• Result: Significantly improved embedding quality for classification

Evaluation Results

Classification Performance

Embedding Quality

The silhouette score of 0.663 indicates well-separated clusters, suitable for embedding-based retrieval and similarity search.

Phase 1 → Phase 2 Improvement

Supported MIME Types (106 Classes)

The model classifies files into 106 MIME types across these categories:

<details> <summary>Click to expand full MIME type list</summary>

application/ (41 types):

• application/SIMH-tape-data, application/encrypted, application/gzip
• application/javascript, application/json, application/msword
• application/mxf, application/octet-stream, application/pdf
• application/pgp-keys, application/postscript
• application/vnd.microsoft.portable-executable, application/vnd.ms-excel
• application/vnd.ms-opentype, application/vnd.ms-powerpoint
• application/vnd.oasis.opendocument.spreadsheet
• application/vnd.openxmlformats-officedocument.* (3 variants)
• application/vnd.rn-realmedia, application/vnd.wordperfect
• application/wasm, application/x-7z-compressed, application/x-archive
• application/x-bzip2, application/x-coff, application/x-dbf
• application/x-dosexec, application/x-executable
• application/x-gettext-translation, application/x-ms-ne-executable
• application/x-ndjson, application/x-object, application/x-ole-storage
• application/x-sharedlib, application/x-shockwave-flash
• application/x-tar, application/x-wine-extension-ini
• application/zip, application/zlib, application/zstd

text/ (24 types):

• text/csv, text/html, text/plain, text/rtf, text/troff
• text/x-Algol68, text/x-asm, text/x-c, text/x-c++
• text/x-diff, text/x-file, text/x-fortran, text/x-java
• text/x-m4, text/x-makefile, text/x-msdos-batch, text/x-perl
• text/x-php, text/x-po, text/x-ruby, text/x-script.python
• text/x-shellscript, text/x-tex, text/xml

image/ (18 types):

• image/bmp, image/fits, image/gif, image/heif, image/jpeg
• image/png, image/svg+xml, image/tiff, image/vnd.adobe.photoshop
• image/vnd.microsoft.icon, image/webp, image/x-eps, image/x-exr
• image/x-jp2-codestream, image/x-portable-bitmap
• image/x-portable-greymap, image/x-tga, image/x-xpixmap

video/ (9 types):

• video/3gpp, video/mp4, video/mpeg, video/quicktime, video/webm
• video/x-ivf, video/x-matroska, video/x-ms-asf, video/x-msvideo

audio/ (8 types):

• audio/amr, audio/flac, audio/mpeg, audio/ogg, audio/x-ape
• audio/x-hx-aac-adts, audio/x-m4a, audio/x-wav

font/ (3 types):

• font/sfnt, font/woff, font/woff2

other (3 types):

• biosig/atf, inode/x-empty, message/rfc822

</details>

How to Use

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model = AutoModelForSequenceClassification.from_pretrained(
    "mjbommar/magic-bert-50m-roformer-classification", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("mjbommar/magic-bert-50m-roformer-classification")

model.eval()

# Classify a file
with open("example.pdf", "rb") as f:
    data = f.read(512)

# Decode bytes to string using latin-1 (preserves all byte values 0-255)
text = data.decode("latin-1")
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)
    predicted_id = outputs.logits.argmax(-1).item()
    confidence = torch.softmax(outputs.logits, dim=-1).max().item()

print(f"Predicted class: {predicted_id}")
print(f"Confidence: {confidence:.2%}")

Embedding-Based Similarity Search

# Get normalized embeddings (256-dim, L2-normalized)
with torch.no_grad():
    embeddings = model.get_embeddings(inputs["input_ids"], inputs["attention_mask"])
    # embeddings shape: [batch_size, 256]

# Compute cosine similarity
similarity = torch.mm(embeddings1, embeddings2.T)

Loading MIME Type Labels

from huggingface_hub import hf_hub_download
import json

mime_path = hf_hub_download("mjbommar/magic-bert-50m-roformer-classification", "mime_type_mapping.json")
with open(mime_path) as f:
    id_to_mime = {int(k): v for k, v in json.load(f).items()}

print(f"Predicted MIME type: {id_to_mime[predicted_id]}")

Limitations

1. MLM capability sacrificed: Fill-mask accuracy drops to 14.5% after classification fine-tuning. Use the MLM variant if byte prediction is needed.

2. Position bias: Still present (~46% accuracy drop at offset 1000), though less relevant for classification than for fill-mask tasks.

3. Ambiguous formats: ZIP-based formats (DOCX, XLSX, JAR, APK) share similar structure and may be confused.

4. Rare types: Lower accuracy on underrepresented file types in training data.

Model Selection Guide

Related Models

• magic-bert-50m-roformer-mlm: Base model before classification fine-tuning
• magic-bert-50m-mlm: Absolute position embedding variant (MLM)
• magic-bert-50m-classification: Magic-BERT variant that retains better fill-mask capability (89.7% accuracy)

Related Work

This model builds on the Binary BPE tokenization approach:

• Binary BPE Paper: Bommarito (2025) introduced byte-level BPE tokenization for binary analysis, demonstrating 2-3x compression over raw bytes for executable content.
• Binary BPE Tokenizers: Pre-trained tokenizers for executables are available at mjbommar/binary-tokenizer-001-64k.

Key difference: The original Binary BPE work focused on executable binaries (ELF, PE, Mach-O). Magic-BERT extends this to general file type understanding across 106 diverse formats, using a tokenizer trained on the broader dataset.

Citation

A paper describing Magic-BERT, the training methodology, and the dataset is forthcoming.

@article{bommarito2025binarybpe,
  title={Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis},
  author={Bommarito, Michael J., II},
  journal={arXiv preprint arXiv:2511.17573},
  year={2025}
}