GPT-NEO-1.3B-wiki

Model Card for GPT-NEO-1.3B-wiki

Model Details

Model Description

This model is based on EleutherAI/gpt-neo-1.3B and has been fine-tuned on the Wikipedia dataset. It is designed for text generation tasks such as summarization, question answering, and text completion in English. The model is fine-tuned to improve the fluency and factual accuracy of the generated content.

• Developed by: Kimargin
• Model type: Fine-tuned model
• Language(s): English
• License: Apache 2.0
• Finetuned from model: EleutherAI/gpt-neo-1.3B

Model Sources

• Repository: Kimargin/GPT-NEO-1.3B-wiki

Uses

Direct Use

This model can be used for tasks like text generation, summarization, and question-answering. It is useful for generating coherent and factual text based on English-language prompts.

Downstream Use

The model can be fine-tuned further for domain-specific applications such as legal or medical text generation, creating specialized question-answering systems, or generating structured content from prompts.

Out-of-Scope Use

The model should not be used in critical applications (e.g., legal, medical, or financial advice) as it may generate biased, inaccurate, or misleading information. It is also not suited for real-time decision-making.

Bias, Risks, and Limitations

Since the model was trained on Wikipedia data, it may inherit biases present in the dataset. Users should be cautious when using the model for generating sensitive or potentially biased content. The model may produce inaccurate or misleading text if given ambiguous or misleading prompts.

Recommendations

Users should verify the outputs of the model, especially in critical use cases, and should not rely solely on the model for factual accuracy without human verification.

How to Get Started with the Model

To use the model, you can load it as follows:

tokenizer = AutoTokenizer.from_pretrained("Kimargin/GPT-NEO-1.3B-wiki") model = AutoModelForCausalLM.from_pretrained("Kimargin/GPT-NEO-1.3B-wiki")

input_text = "What happened during World War II?" inputs = tokenizer(input_text, return_tensors="pt") outputs = model.generate(inputs["input_ids"], max_length=100) print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

The model was fine-tuned on a subset of the Wikipedia dataset, which contains a broad range of general knowledge topics. This dataset was chosen to improve the model's capability to generate accurate, general-domain knowledge.

Training Procedure

The model was fine-tuned using mixed precision (float16) on multiple GPUs for three epochs. The training was done to minimize perplexity and improve fluency in generated text.

Training Hyperparameters

• Learning rate: 5e-5
• Batch size: 16
• Epochs: 3
• Precision: float16 (mixed precision)

Evaluation

Testing Data

The model was evaluated using a validation subset of the Wikipedia dataset to measure its performance on general text generation tasks.

Metrics

• Perplexity: The model achieved a perplexity of 25.3 on the validation set.
• Accuracy: The accuracy of the model in generating factual answers was evaluated qualitatively.

Results

The model demonstrates good performance in generating coherent and contextually relevant text, but it may still struggle with niche or specialized topics that are underrepresented in its training data.

Environmental Impact

Training large models like GPT-Neo has a significant carbon footprint due to the computational resources required. The estimated environmental impact of fine-tuning this model is as follows:

• Hardware Type: NVIDIA A100 GPUs
• Hours used: 20 hours
• Cloud Provider: Google Cloud
• Compute Region: US-Central
• Carbon Emitted: ~50 kg CO2 (estimated using the ML Impact calculator)

Technical Specifications

Model Architecture and Objective

The model is a causal language model with 1.3 billion parameters, based on the GPT-Neo architecture. It generates text by predicting the next word in a sequence, making it suitable for text completion and generation tasks.

Compute Infrastructure

The model was trained on NVIDIA A100 GPUs using Google Cloud infrastructure.

Citation

If you use this model, please cite the original GPT-Neo model as follows:

@article{gpt-neo,
  author = {EleutherAI},
  title = {GPT-Neo: Large Scale Autoregressive Language Model},
  year = {2021},
  url = {https://github.com/EleutherAI/gpt-neo}
}