Downloads · 30 days
0
wsdm23_pretrain
lixsh6/wsdm23_pretrain
wsdm23_pretrain is a machine learning model from lixsh6. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
- This repo contains the checkpoints of our competition in WSDM Cup 2023: Pre-training for Web Search and Unbiased Learning for Web Search.
Access
Public
Updated Jul 28, 2023
Repo size
7.5 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin7.5 GB · 99%
At a glance
- Access
- Public
- Created
- Jan 13, 2023
- Updated
- Jul 28, 2023
- SHA
- b92fcc1e
Model card
From the Hugging Face model README
WSDM Cup 2023 BERT Checkpoints:
- This repo contains the checkpoints of our competition in WSDM Cup 2023: Pre-training for Web Search and Unbiased Learning for Web Search.
Paper released
Please refer to our paper for details in this competition:
- Task1 Unbiased Learning to rank: Multi-Feature Integration for Perception-Dependent Examination-Bias Estimation
- Task2 Pretraining for web search: Pretraining De-Biased Language Model with Large-scale Click Logs for Document Ranking
Method Overview
- Pre-training BERT with MLM and CTR prediction loss (or multi-task CTR prediction loss).
- Finetuning BERT with pairwise ranking loss.
- Obtain prediction scores from different BERTs.
- Ensemble learning to combine BERT features and sparse features.
Details will be updated in the submission paper.
BERT features:
1) Model details: Checkpoints Download Here
| Index | Model Flag | Method | Pretrain step | Finetune step | DCG on leaderboard |
|---|---|---|---|---|---|
| 1 | large_group2_wwm_from_unw4625K | M1 | 1700K | 5130 | 11.96214 |
| 2 | large_group2_wwm_from_unw4625K | M1 | 1700K | 5130 | NAN |
| 3 | base_group2_wwm | M2 | 2150K | 5130 | ~11.32363 |
| 4 | large_group2_wwm_from_unw4625K | M1 | 590K | 5130 | 11.94845 |
| 5 | large_group2_wwm_from_unw4625K | M1 | 1700K | 4180 | NAN |
| 6 | large_group2_mt_pretrain | M3 | 1940K | 5130 | NAN |
2) Method details
| Method | Model Layers | Details |
|---|---|---|
| M1 | 24 | WWM & CTR prediction as pretraining tasks |
| M2 | 12 | WWM & CTR prediction as pretraining tasks |
| M3 | 24 | WWM & Multi-task CTR prediction as pretraining tasks |
Contacts
- Xiangsheng Li: [email protected].
- Xiaoshu Chen: [email protected]
- Created
- Jan 13, 2023
- Updated
- Jul 28, 2023