nano-phi-192M-v0.1

Model Card for nano-phi-192M-v0.1

This is a continual effort from kenhktsui/nano-phi-115M-v0.1.
The model is not aligned.

Major differences:

• bigger tokenizer's vocab size
• addition of HuggingFaceTB/cosmopedia as training dataset
• training token: 19B vs 7B

How to use

To use the model, you will need transformer version >= 4.37.2

pip install transformers>=4.37.2

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kenhktsui/nano-phi-192M-v0.1")
pipe("I am a machine learning researcher. I work on", max_new_tokens=50, repetition_penalty=10.0)

Some metrics

• model
  • hidden_size: 768
  • num_key_value_heads: 8 (grouped query attention)
  • num_attention_heads: 24
  • num_hidden_layers: 6
  • context length: 1024
  • total params: 192M
• training:
  • global steps: 36,000

Open LLM Leaderboard Evaluation Results

Details:

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 8

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 25, batch_size: 8

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 10, batch_size: 8

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 8

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 5, batch_size: 8

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 5, batch_size: 8

hf-causal-experimental (pretrained=/content/lm-evaluation-harness/artifacts/model-9gh18vfl:v25,use_accelerate=false,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 5, batch_size: 8