llama31-8b-balitanlp-cpt

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

llama31-8b-balitanlp-cpt

This model is a continuously pretrained version of meta-llama/Llama-3.1-8B on an unknown dataset.

Model description

Continuous pretraining on Filipino news articles from https://huggingface.co/datasets/LanceBunag/BalitaNLP

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 10
• training_steps: 100

Training results

Example

Before continuous pretraining:

After:

📌 API Test Notes

1️⃣ Request 1

curl -X POST localhost:8080/v1/completions
-H "Content-Type: application/json"
-d '{ "model": "internetoftim/llama31-8b-balitanlp-cpt", "prompt": "Kumusta ka?", "max_tokens": 50, "temperature": 0.7 }'

Result 1 { "id": "cmpl-f10e26c5290a460b89156bc373b5d6f6", "object": "text_completion", "created": 1765715251, "model": "internetoftim/llama31-8b-balitanlp-cpt", "choices": [ { "index": 0, "text": " Ano ang isang magandang kanta na ang maaaring maging ang ikawang single na sisimulan mo ngayong taon? Sa halip na manood ka lang ng mga kanta ng ibang artista,", "logprobs": null, "finish_reason": "length", "stop_reason": null, "token_ids": null, "prompt_logprobs": null, "prompt_token_ids": null } ], "service_tier": null, "system_fingerprint": null, "usage": { "prompt_tokens": 6, "total_tokens": 56, "completion_tokens": 50, "prompt_tokens_details": null }, "kv_transfer_params": null }

2️⃣ Request 2

curl -X POST localhost:8080/v1/completions
-H "Content-Type: application/json"
-d '{ "model": "internetoftim/llama31-8b-balitanlp-cpt", "prompt": "P900-M RELIEF GOODS INIHANDA NG NDRRMC PARA SA BAGYO", "max_tokens": 100, "temperature": 0.7 }'

Result 2 { "id": "cmpl-106c32f72d464409a9f7d8597bded8b6", "object": "text_completion", "created": 1765715647, "model": "internetoftim/llama31-8b-balitanlp-cpt", "choices": [ { "index": 0, "text": " NA TUMATAAS SA HAWAAN NG BISAYA – NDRRMC (NATIONAL DISASTER RISK REDUCTION AND MANAGEMENT COUNCIL) ay inihanda na ang P900-Million worth ng relief goods na gagamitin para sa mga nasalanta ng bagyo sa Eastern Visayas, Central Visayas at Western Visayas, na tumataas sa Dagat sa Timog China at patungong Luzon sa loob ng Linggo,", "logprobs": null, "finish_reason": "length", "stop_reason": null, "token_ids": null, "prompt_logprobs": null, "prompt_token_ids": null } ], "service_tier": null, "system_fingerprint": null, "usage": { "prompt_tokens": 22, "total_tokens": 122, "completion_tokens": 100, "prompt_tokens_details": null }, "kv_transfer_params": null }

🔹 Request 3

curl -X POST localhost:8080/v1/completions
-H "Content-Type: application/json"
-d '{ "model": "internetoftim/llama31-8b-balitanlp-cpt", "prompt": "P900-M RELIEF GOODS INIHANDA NG NDRRMC PARA SA BAGYO", "max_tokens": 100, "temperature": 0.7 }'

🔹 Result

{ "id": "cmpl-106c32f72d464409a9f7d8597bded8b6", "object": "text_completion", "created": 1765715647, "model": "internetoftim/llama31-8b-balitanlp-cpt", "choices": [ { "index": 0, "text": " NA TUMATAAS SA HAWAAN NG BISAYA – NDRRMC (NATIONAL DISASTER RISK REDUCTION AND MANAGEMENT COUNCIL) ay inihanda na ang P900-Million worth ng relief goods na gagamitin para sa mga nasalanta ng bagyo sa Eastern Visayas, Central Visayas at Western Visayas, na tumataas sa Dagat sa Timog China at patungong Luzon sa loob ng Linggo,", "logprobs": null, "finish_reason": "length", "stop_reason": null, "token_ids": null, "prompt_logprobs": null, "prompt_token_ids": null } ], "service_tier": null, "system_fingerprint": null, "usage": { "prompt_tokens": 22, "total_tokens": 122, "completion_tokens": 100, "prompt_tokens_details": null }, "kv_transfer_params": null }

####TL;DR This is pre-trained base model, it's not quite useful but you can see it learned some Filipino. Next would be to do instruction tuning.

Framework versions

• Transformers 4.57.3
• Pytorch 2.10.0a0+b558c986e8.nv25.11
• Datasets 4.4.1
• Tokenizers 0.22.1