unmasker

Approach:

Pretrain Hugging Face's BERT MaskedLM model using WordNet3.0 synsets and definitions

Training details:

Dataset: WordNet3.0 synsets and first definition associated with each word

Pre-training sample size: 5000 words

Optimizer: Pytorch SGD

Batch size: 16

Epochs: 5

Activation function: softmax

Input for the model:

Concatenate definition with masked version of word using [SEP] tokens and replace given _ with [MASK] e.g.: "A royal figure, typically male [SEP] K [MASK] [MASK] G"

Results:

Average loss per batch plateaued at around 0.2029 for all five epochs Softmax and argmax return tensors of all 0 and thus no useful predictions.