Downloads · 30 days
4
2% of all-time downloads
bert-base-uncased-teacher-preparation-pretrain
lvkaokao/bert-base-uncased-teacher-preparation-pretrain
bert-base-uncased-teacher-preparation-pretrain is a machine learning model from lvkaokao. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as other.
All-time downloads
233
Public
Repo size
441 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin441 MB · 100%
At a glance
- Library
- transformers
- License
- other
- Model type
- bert
- Access
- Public
- Created
- Sep 27, 2022
- Updated
- Nov 4, 2022
- SHA
- 7b2b2c33
Model card
From the Hugging Face model README
#!/bin/bash
# Apache v2 license
# Copyright (C) 2021 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
# Teacher Preparation
# Notes:
# Auto mixed precision can be used by adding --fp16
# Distributed training can be used with the torch.distributed.lauch app
TEACHER_PATH=./bert-base-uncased-teacher-preparation-pretrain
OUTPUT_DIR=$TEACHER_PATH
DATA_CACHE_DIR=/root/kaokao/Model-Compression-Research-Package/examples/transformers/language-modeling/wikipedia_processed_for_pretrain
python -m torch.distributed.launch \
--nproc_per_node=8 \
../../examples/transformers/language-modeling/run_mlm.py \
--model_name_or_path bert-base-uncased \
--datasets_name_config wikipedia:20200501.en \
--data_process_type segment_pair_nsp \
--dataset_cache_dir $DATA_CACHE_DIR \
--do_train \
--learning_rate 5e-5 \
--max_steps 100000 \
--warmup_ratio 0.01 \
--weight_decay 0.01 \
--per_device_train_batch_size 8 \
--gradient_accumulation_steps 4 \
--logging_steps 10 \
--save_steps 5000 \
--save_total_limit 2 \
--output_dir $OUTPUT_DIR \
--run_name pofa-teacher-prepare-pretrain
- Library
- transformers
- Type
- bert
- License
- other
- Created
- Sep 27, 2022
- Updated
- Nov 4, 2022