VDrontV1.5-Flash

cc0-1.0

Small description

VDrontV1.5-Flash is a 60M-parameter GPT-like model. Train in 1.5b tokens.

The model was trained on FineWeb-Edu + FineWeb-Edu-Instruct.

Stored in Hugging Face format and DrontQ(.dq) quantized format.

VDrontV1.5-Flash/ - HF format 32 bit

VDrontV1.5-Flash_quant/ .dq format, the folder contains DQuse.py for using .dq.

Formats

Details

Context length: 512

Tokenizer: gpt2

Base model: VdrontV1 (ours)

Fine-tuning on GPU using the Lion optimizer

Ratio of FW-Edu to FW-Edu-instruct training = 3:1

Use of Drontal Quant.

License : cc0-1.0.

Guide

To use the HF model, go to the files, download the VDrontV1.5-Flash/ folder and use.py, and place them together. use.py includes prompt formatting with <|user|> and <|assistant|> and is ready for immediate use.

To use Dront Quant models, go to the VDrontV1.5-Flash_quant/ folder. It contains 3 .dq models (8-bit, 12-bit, and 16-bit) as well as DQuse.py. Download the model you are interested in and place DQuse.py and the .dq model next to each other. At the end of the DQuse.py file, change the model name to the one you want to use. Note that the prompt formatting uses <|user|> and <|assistant|> tokens. (388 line: chat_with_dq_model("[path].dq", 256) )

Fine-tune

If you fine-tune the VDrontV1.5-Flash model, we will most likely add it to the VDrontV collection.

Python libraries:

DQuse.py : torch json os struct numpy collections transformers

use.py : torch transformers threading

DrontQuant : https://huggingface.co/VDC-team/DrontGPTQ

Hugging Face : VDC-team/