bge-small-en-v1.5-ultrafineweb-vs-pile-classifier

bge-small-en-v1.5-ultrafineweb-vs-pile-classifier

[!IMPORTANT]
Note: This model is provided for reference and reproducibility, not for standalone use.

This model is a fine-tuned version of BAAI/bge-small-en-v1.5 to classify text as high quality or low quality for AI training.

• Trained on 100k samples from openbmb/Ultra-FineWeb (high quality) and 100k from EleutherAI/the_pile_deduplicated (low quality)
• 80% training / 20% validation split

On the validation set:

• Loss: 0.2926
• Accuracy: 0.9061
• Combined Score: 2.1448
• Tokens processed: 102,184,960

Example

from transformers import pipeline

classifier = pipeline("text-classification", model="agentlans/bge-small-en-v1.5-ultrafineweb-vs-pile-classifier")
classifier("Your text here.")

Limitations

• Tends to be overly strict, labelling most texts outside training data as low quality
• English only
• May be biased against some text types such as source code and personal blogs

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 5.0

Training results

Framework versions

• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0