PLDR-LLM-v51G-106M-test

PLDR-LLM-v51G-106M-test

Model Description

PLDR-LLM-v51G-106M-test is a large language model from power law decoder representations with KV-cache and G-cache support. It is trained with a model configuration similar to PLDRv51G-106M-2 and by using the default Huggingface Transformers implementation of RoPE embeddings for Llama when reference_rope=False in PLDR-LLM model configuration.

Training data

PLDR-LLM-v51G-106M-test was pretrained on the RefinedWeb, a publicly available English web dataset with extensive filtering and deduplication.

Training procedure

This model was trained for ~8B tokens on RefinedWeb over 250k steps per rank. It was trained autoregressively with cross-entropy loss.

Intended Use and Limitations

This model is made available to be used for testing and development of PLDR-LLM model implementation into Huggingface Transformers library. The repository with added PLDR-LLM model support can be found here. Given text as input prompt, it carries out next token prediction to generate continuation text. The context length for this model is 1024 tokens.

Limitations and Biases

Large Language Models may generate text that is profane, lewd, socially unacceptable or offensive based on the contents of the dataset it was pretrained. RefinedWeb is a dataset that is as toxic and biased as the Pile. Please see the papers for RefinedWeb and the Pile for more information. Moreover, large language models are also susceptible to hallucinations and may generate text that contains incorrect, irrelevant or misleading information. Since it is very hard to expect the contents of generated text ahead of time, the output of the large language models need to be heavily moderated and curated to avoid undesired content to appear without warning.