jina-embedding-v3-gte

Running Jina Embedding V3 on Text-Embedding-Inference

• See branch: TEI-support

• Changes Made to GTE styled architecture:

1. Removed the "roberta" prefix from all tensor names.
2. Renamed "mixer" to "attention" in encoder layers.
3. Converted "Wqkv" to "qkv_proj" for combined query, key, value projections.
4. Renamed "mlp.fc1" to "mlp.up_proj" and "mlp.fc2" to "mlp.down_proj".
5. Created "mlp.up_gate_proj" by duplicating and expanding "mlp.up_proj".
6. Renamed "norm1" to "attn_ln" and "norm2" to "mlp_ln" in encoder layers.
7. Changed "emb_ln" to "embeddings.LayerNorm".
8. Renamed "weight" to "gamma" and "bias" to "beta" for layer normalization layers.
9. Removed LoRA-related tensors.

Features:

1. Structural Compatibility: The renamed model now closely matches the expected GTE architecture, allowing it to load without "tensor not found" errors.
2. Preservation of Core Weights: Most of the original model's weights are preserved, maintaining some of the learned features.
3. Adaptability: The script can handle various naming conventions and structures, making it somewhat flexible for future adjustments.
4. Transparency: The script provides a clear view of the tensor names and shapes after conversion, aiding in debugging.

Limitations:

1. Approximated Architecture: The conversion is an approximation of the GTE architecture, not an exact match. This may affect model performance.
2. Loss of LoRA Adaptations: By removing LoRA-related tensors, we've lost the fine-tuning adaptations, potentially impacting the model's specialized capabilities.
3. Up-Gate Projection Approximation: The "up_gate_proj" is created by duplicating weights, which may not accurately represent the intended GTE architecture.
4. Potential Performance Impact: The structural changes, especially in the MLP layers, may affect the model's performance and output quality.
5. Lack of Positional Embeddings Handling: We haven't specifically addressed positional embeddings, which might be different between XLM-RoBERTa and GTE models.
6. Possible Missing Specialized Layers: There might be specialized layers or components in the GTE architecture that we haven't accounted for.
7. No Guarantee of Functional Equivalence: While the model now loads, there's no guarantee it will function identically to a true GTE model.
8. Config File Mismatch: We haven't addressed potential mismatches in the config.json file, which might cause issues during model initialization or inference.