cat-emb-2-128

bwang0911/cat-emb-2-128

cat-emb-2-128 is a machine learning model from bwang0911. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.

A set of embedding model trained for study embedding quality vs model architecture (width/depth) given a size constraint (12M params).

Downloads · 30 days

24% of all-time downloads

All-time downloads

Public

Repo size

26.3 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.bin17.6 MB · 99%

At a glance

Library: transformers
License: apache-2.0
Model type: bert
Access: Public
Created: Jun 13, 2024
Updated: Jun 14, 2024
SHA: 555e21a8

Model card

From the Hugging Face model README

Cat Embeddings

A set of embedding model trained for study embedding quality vs model architecture (width/depth) given a size constraint (12M params).

cat-emb-2-128: 2 layers/hidden size 128/4.4m
cat-emb-4-128: 4 layers/H 128/4.8m
cat-emb-8-128: 8 layers/H 128/5.6m
cat-emb-12-128: 12 layers/H 128/6.4m
cat-emb-2-256: 2 layers/H 256/9.7m
cat-emb-4-256: 4 layers/H 256/11.3m

Training

stage 1: seq 192, batch size 2048, 50k steps, sentence pairs.
stage 2: seq 512, batch size 64, 5k steps, sentence triplets.

Perf

MRL dim\Task	BIOSSES	SICK-R	STS12	STS13	STS14	STS15	STS16	STSB	SummEval
128	0.7107	0.7126	0.6815	0.7343	0.7038	0.8163	0.7495	0.7652	0.2958
64	0.713	0.7123	0.6829	0.7348	0.7008	0.813	0.7475	0.7609	0.2861
32	0.6714	0.7094	0.6847	0.7345	0.6911	0.7989	0.7385	0.7545	0.3106
16	0.6637	0.697	0.669	0.7096	0.6665	0.7589	0.7183	0.7307	0.3164

Library: transformers
Type: bert
License: apache-2.0
Created: Jun 13, 2024
Updated: Jun 14, 2024