anime-frame-interesting-classifier-vit-v2

Anime Frame Interesting Classifier (ViT v2.0)

Model Details

Architecture: MobileViT-Small (Transformer)
Framework: Hugging Face Transformers
Input Size: 224x224 RGB images
Output: Binary classification (Boring/Interesting)

Performance

Evaluated on v2.0 test set (433 frames):

• F1 Score: 94.92%
• Accuracy: 94.92%
• Precision: 95.05%
• Recall: 94.92%

Training data: 3,999 frames (4,432 total with test holdout)

Intended Use

What it does: Classifies anime frames as either "interesting" (depicting meaningful character/scene details) or "boring" (back-of-head shots, non-descript backgrounds, montages).

Strengths:

• Transformer-based semantic understanding
• Better generalization to style variations
• Good for ensemble voting with CNN model
• Complementary confidence to CNN predictions

When to use:

• Ensemble voting with CNN model for higher confidence
• Applications preferring transformer-based features
• Fine-tuning for downstream anime tasks

When NOT to use:

• Real-world photos or non-anime content
• Frames smaller than 224x224
• Speed-critical deployments (slower than CNN)

Labels

• Class 0 (Boring): Frames lacking interesting visual details or character focus
• Class 1 (Interesting): Frames with clear character/scene details suitable for downstream tasks

Model Size & Speed

• Model Size: 19 MB (SafeTensors format)
• Inference Speed: ~25ms per image on GPU
• VRAM Required: ~2 GB (including activations)
• Speed vs CNN: ~25% slower but more semantically aware

Training Data Composition

• 900 manually curated frames (hand-labeled)
• 1,655 frames filtered via dual-agreement with garbage classifier
• 1,877 frames from curated anime site scraper
• Total: 4,432 frames (90% train, 10% test holdout)

All frames are 224x224 RGB anime screenshots.

How to Use

Recommended: HuggingFace Transformers (SafeTensors)

from transformers import AutoImageProcessor, AutoModelForImageClassification
from PIL import Image

# Load model (automatically uses SafeTensors)
processor = AutoImageProcessor.from_pretrained(
    'hf_models/anime-frame-interesting-classifier-vit-v2'
)
model = AutoModelForImageClassification.from_pretrained(
    'hf_models/anime-frame-interesting-classifier-vit-v2',
    trust_remote_code=False  # Safe: SafeTensors prevents code execution
)
model.eval()

image = Image.open('frame.png')
inputs = processor(image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits
    prediction = logits.argmax(-1).item()
    confidence = logits.softmax(-1)[0][prediction].item()

print(f"Prediction: {'Interesting' if prediction == 1 else 'Boring'}")
print(f"Confidence: {confidence:.2%}")

Direct Load with SafeTensors

from transformers import MobileViTForImageClassification, AutoImageProcessor
from safetensors.torch import load_file
from PIL import Image
import torch

# Load with SafeTensors (secure)
model = MobileViTForImageClassification.from_pretrained(
    'apple/mobilevit-small',
    num_labels=2
)
state_dict = load_file('model.safetensors')
model.load_state_dict(state_dict)
model.eval()

processor = AutoImageProcessor.from_pretrained('apple/mobilevit-small')
image = Image.open('frame.png')
inputs = processor(image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    prediction = outputs.logits.argmax(-1).item()

print(f"Prediction: {'Interesting' if prediction == 1 else 'Boring'}")

Security Note: This model uses SafeTensors format (not pickle). SafeTensors is a secure serialization format that cannot execute arbitrary code during loading, unlike pickle-based .bin files.

Comparison with CNN Model

See anime-frame-interesting-classifier-cnn-v2 for CNN alternative:

Recommended: Use ensemble voting for maximum confidence:

1. Classify with both models
2. Flag disagreements for manual review
3. Trust when both models agree

Limitations

• Anime-only: Trained exclusively on anime content
• Speed: Slower inference than CNN (1-2 fps vs 2-3 fps)
• Dataset bias: Training data skewed toward popular anime styles
• Resolution: Trained on 224x224; extreme aspect ratios need preprocessing
• Edge cases: Minimal training on hard-to-classify borderline frames

Training Details

• Dataset: v2.0 (4,432 frames)
• Base Model: apple/mobilevit-small (5.6M parameters)
• Train/Test Split: 90/10 (3,999 train, 433 test)
• Epochs: 20
• Batch Size: 64
• Optimizer: AdamW (lr=1e-4)
• Loss: CrossEntropyLoss
• Augmentation: None (data quality sufficient at this scale)

Version History

• v2.0 (current): 94.92% F1, retrained on expanded 4,432-frame dataset
• v1.0: 86% F1, 900-frame dataset (legacy, deprecated)

Citation

If you use this model, please reference:

• Dataset: Anime Frame Interesting v2.0 (4,432 curated frames)
• Architecture: MobileViT-Small (apple/mobilevit-small)
• Framework: Hugging Face Transformers
• Training: PyTorch, 2026

Ensemble Strategy

For best results, use both CNN and ViT models together:

def ensemble_classify(image_path, cnn_model, vit_model):
    """Classify with both models, flag disagreements"""
    # CNN prediction
    cnn_pred = classify_with_cnn(image_path, cnn_model)
    
    # ViT prediction
    vit_pred = classify_with_vit(image_path, vit_model)
    
    if cnn_pred['class'] == vit_pred['class']:
        # Agreement: high confidence
        confidence = (cnn_pred['conf'] + vit_pred['conf']) / 2
        return {
            'prediction': cnn_pred['class'],
            'confidence': confidence,
            'agreement': 'both'
        }
    else:
        # Disagreement: flag for manual review
        return {
            'cnn': cnn_pred,
            'vit': vit_pred,
            'agreement': 'none',
            'recommendation': 'manual_review'
        }

Future Improvements

• Collect 1,000+ edge-case frames for hard-negatives
• Experiment with larger ViT variants (if available)
• Fine-tune for specific anime styles
• Distill to smaller model for faster inference