Downloads · 30 days
10
4% of all-time downloads
cemig-temp/qwen3.5-4b-regulacao-normativa
qwen3.5-4b-regulacao-normativa is a image-text-to-text model from cemig-temp. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for transformers.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
10
4% of all-time downloads
All-time downloads
268
Public
Parameters
4.5B
9.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors9.1 GB · 100%
From the Hugging Face model README
axolotl version: 0.16.2.dev0
base_model: ./outputs/qwen3.5-4b-regulacao-distribuicao-mcsynth-1e5
# Stage 2: full SFT refinement on domain data (regulacao + distribuicao).
# Requires stage 1 (qwen3.5-cemig.yml) to have finished and saved weights.
plugins:
- axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
strict: false
chat_template: qwen3_5
datasets:
- path: TokenLab/CemigConvoV1.1
split: regulacao
type: chat_template
field_messages: messages
roles_to_train: [assistant]
- path: /workspace/pipeline/data/distribuicao/norm_tecnicas_dist.jsonl
ds_type: json
type: chat_template
field_messages: messages
roles_to_train: [assistant]
val_set_size: 0.05
output_dir: ./outputs/qwen3.5-4b-regulacao-distribuicao-stage2-5e6
dataset_prepared_path: ./prepared_datasets/qwen3.5-4b-stage2
sequence_len: 4096
sample_packing: true
# Freeze vision encoder (same as stage 1)
unfrozen_parameters:
- model.language_model.*
- lm_head.*
wandb_project: qwen3.5-4b-regulacao-distribuicao
wandb_entity:
wandb_watch:
wandb_name: qwen3.5-4b-regulacao-distribuicao-stage2
wandb_log_model:
gradient_accumulation_steps: 4
micro_batch_size: 8
num_epochs: 1
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 5e-6
bf16: auto
tf32: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
resume_from_checkpoint:
logging_steps: 1
attn_implementation: flash_attention_2
warmup_ratio: 0.02
evals_per_epoch: 2
saves_per_epoch: 1
weight_decay: 0.01
special_tokens:
# fsdp_config:
# fsdp_version: 2
# offload_params: false
# cpu_ram_efficient_loading: false
# auto_wrap_policy: TRANSFORMER_BASED_WRAP
# transformer_layer_cls_to_wrap: Qwen3_5DecoderLayer
# state_dict_type: FULL_STATE_DICT
# sharding_strategy: FULL_SHARD
# reshard_after_forward: true
# activation_checkpointing: true
</details><br>
This model was trained from scratch on the TokenLab/CemigConvoV1.1 and the /workspace/pipeline/data/distribuicao/norm_tecnicas_dist.jsonl datasets. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss | Ppl | Active (gib) | Allocated (gib) | Reserved (gib) |
|---|---|---|---|---|---|---|---|
| No log | 0 | 0 | 0.6501 | 1.9157 | 12.6 | 12.6 | 12.67 |
| 0.6275 | 0.5004 | 518 | 0.6470 | 1.9098 | 28.28 | 28.28 | 47.0 |
| 0.6454 | 0.9998 | 1035 | 0.6464 | 1.9087 | 28.28 | 28.28 | 47.62 |