Downloads · 30 days
0
HCAE-21M-Instruct
HeavensHackDev/HCAE-21M-Instruct
HCAE-21M-Instruct is a feature extraction model from HeavensHackDev. Use it when you need embeddings to search or compare text. It is set up for pytorch. The card lists the license as apache-2.0.
HCAE-21M is a mid-scale (21 Million parameters) text embedding model combining Depthwise Separable Convolutions and Self-Attention layers. It achieves high performance on Semantic Textual Similarity and Retrieval task…
Access
Public
Updated Apr 8, 2026
Parameters
23.9M
191 MB on disk
Likes
2
Public
Repo makeup
Click a slice to open those files.
.bin95.7 MB · 50%
At a glance
- Task
- Feature Extraction
- Library
- pytorch
- License
- apache-2.0
- Access
- Public
- Created
- Apr 8, 2026
- Updated
- Apr 8, 2026
- SHA
- d975cb7f
Model card
From the Hugging Face model README
HCAE-21M (Hybrid Convolutional-Attention Encoder)
HCAE-21M is a mid-scale (21 Million parameters) text embedding model combining Depthwise Separable Convolutions and Self-Attention layers. It achieves high performance on Semantic Textual Similarity and Retrieval tasks while remaining extremely memory-efficient.
<img src="https://cdn-uploads.huggingface.co/production/uploads/680c9127408ea47e6c1dd6e8/0KKsVpqg2Id01nxh8zRjO.png" width="400">Architecture Description
- Size: ~21M parameters (d_model=384)
- Lower Layers: 5 layers of Depthwise Separable Conv1d + FFN.
- Upper Layers: 3 layers of Multihead Self-Attention.
- Pooling Strategy: Global Mean Pooling.
Benchmark Comparison (MTEB)
This table delineates the performance disparities between architectural iterations:
| Model Revision | STSBenchmark (Spearman) | SciFact (Recall@10) | Description |
|---|---|---|---|
| HCAE-21M-Base | 0.507 | 0.324 | Baseline configuration trained extensively on the MS MARCO dataset. |
| HCAE-21M-Instruct | 0.591 | 0.393 | Multi-stage tuning incorporating ArXiv, STS-B, and SQuAD instruction tuning paradigms. |
Utilization Guidelines (Instruction Format)
For optimal retrieval performance, prepend the instruction mapping to the query text:
Instruction: Retrieve the exact document that answers the following question. Query: [Your Query]
- Task
- Feature Extraction
- Library
- pytorch
- License
- apache-2.0
- Created
- Apr 8, 2026
- Updated
- Apr 8, 2026