Florence-2-large-no-flash-attn-Transformers-Fix

Model Card: Florence-2-large-no-flash-attn (Transformers 4.57.3 Compatible)

Model Overview

This is the Florence-2-large-no-flash-attn model from Microsoft Corporation, made compatible with transformers==4.57.3.

Original Model Information

• Model Name: Florence-2-large-no-flash-attn
• Original Authors: Microsoft
• Original Repository: microsoft/Florence-2-large-no-flash-attn
• Model Type: Vision-Language Model (Encoder-Decoder) - No Flash Attention Variant
• Purpose: Vision-Language Understanding without Flash Attention optimization

License & Ownership

Original License: MIT License
Original Copyright: © 2024 Microsoft Corporation

This modified version maintains the original MIT license and respects all original copyright notices. All original code and model weights remain the intellectual property of Microsoft Corporation.

Modifications Made

Issue Addressed

Fixed compatibility issues with transformers==4.57.3 where AttributeError: 'NoneType' object has no attribute 'shape' occurred during model inference and generation.

Root Cause

In transformers 4.57.3, the past_key_values structure can contain None elements at various nesting levels, particularly during early generation stages. The original code only checked for top-level None but not nested None values.

Changes Applied

1. Attention Layer Fixes (Florence2Attention and Florence2SdpaAttention)

• Location: Lines ~852, ~994, ~1182 in modeling_florence2.py
• Change: Added past_key_value[0] is not None check before accessing .shape[2]
• Pattern:

  # Before
  if (is_cross_attention and past_key_value is not None 
      and past_key_value[0].shape[2] == key_value_states.shape[1]):
  
  # After
  if (is_cross_attention and past_key_value is not None 
      and past_key_value[0] is not None
      and past_key_value[0].shape[2] == key_value_states.shape[1]):
  

2. KV Sequence Length Calculation

• Location: Line ~1026 in modeling_florence2.py
• Change: Added past_key_value[0] is not None check before shape access
• Pattern:

  # Before
  if past_key_value is not None:
      kv_seq_len += past_key_value[0].shape[-2]
  
  # After
  if past_key_value is not None and past_key_value[0] is not None:
      kv_seq_len += past_key_value[0].shape[-2]
  

3. Generation Input Preparation

• Location: Lines ~2228, ~2855 in modeling_florence2.py
• Change: Wrapped past_key_values access in try-except blocks with fallbacks
• Pattern:

  # Before
  past_length = past_key_values[0][0].shape[2]
  
  # After
  try:
      past_length = past_key_values[0][0].shape[2]
  except (AttributeError, TypeError, IndexError):
      past_length = 0
  

4. Positional Embedding Index Validation

• Location: Florence2LearnedPositionalEmbedding.forward near top of modeling_florence2.py
• Change: Added explicit bounds check to catch out-of-range positions before embedding lookup.
• Pattern:

  positions = positions + self.offset
  if torch.any(positions < 0) or torch.any(positions >= self.num_embeddings):
      raise ValueError(...)
  return super().forward(positions)
  

Files Modified

• modeling_florence2.py - Multiple locations for robustness

Backward Compatibility

✅ Fully backward compatible with older transformers versions
✅ Forward compatible with future transformers versions using DynamicCache

Performance Considerations

No Flash Attention Variant

This model variant does not use Flash Attention optimization, making it suitable for:

• Systems where Flash Attention is not available
• Compatibility testing and debugging
• Resource-constrained environments
• Detailed attention analysis

Usage

from transformers import AutoModel, AutoProcessor
import torch

# Load the no-flash-attn model
model = AutoModel.from_pretrained(
    "path/to/Florence-2-large-no-flash-attn",
    trust_remote_code=True
)
processor = AutoProcessor.from_pretrained(
    "path/to/Florence-2-large-no-flash-attn"
)

# Generate with compatibility fixes
model.eval()
with torch.no_grad():
    inputs = processor(text="Describe this image", return_tensors="pt")
    outputs = model.generate(**inputs, max_new_tokens=50)

Technical Details

Tested With

• Transformers: 4.57.3
• PyTorch: 2.0+
• Python: 3.8+

Memory and Speed Tradeoffs

• Advantages: Broader compatibility, no CUDA dependencies for optimizations
• Disadvantages: Potentially slower inference compared to Flash Attention variants
• Use Case: Environments where stability and compatibility are prioritized over speed

Testing Recommendations

Test the following scenarios to ensure compatibility:

1. Single image caption generation
2. Batch processing multiple images
3. Long sequence generation with many tokens
4. Memory profiling and optimization
5. Attention weight inspection

Acknowledgments

• Original Model Developers: Microsoft Corporation
• Modifications: Compatibility layer for transformers 4.57.3
• License Preservation: Original MIT license maintained

References

• Original Model: https://huggingface.co/microsoft/Florence-2-large-no-flash-attn
• Florence Paper: [Reference provided in original repository]
• Transformers Library: https://github.com/huggingface/transformers

---

Version: 1.0 (Transformers 4.57.3 Compatible)
Last Updated: March 2026
Status: Production Ready