Downloads · 30 days
16
3% of all-time downloads
pymlex/qwen3-4b-ds-jobs-expert
qwen3-4b-ds-jobs-expert is a text generation model from pymlex. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as gpl-3.0.
Qwen/Qwen3-4B-Base is a 4B base model suitable for custom post-training. In this project, we apply continuous pretraining (CPT) on the pymlex/datasciencejobs-tg dataset to make the model more fluent on machine learnin…
Downloads · 30 days
16
3% of all-time downloads
All-time downloads
529
Public
Parameters
4.1B
8.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors8.3 GB · 100%
How the weights are stored.
BF164B · 99%
From the Hugging Face model README
Qwen/Qwen3-4B-Base is a 4B base model suitable for custom post-training. In this project, we apply continuous pretraining (CPT) on the pymlex/datasciencejobs-tg dataset to make the model more fluent on machine learning and data science job posts, vacancy patterns, salary and hiring phrasing, and domain vocabulary for further usage on platforms like HeadHunter or LinkedIn.
The dataset contains 1,784 Telegram job-post records. Each row includes:
iddateviewstextThe raw post text is used as plain language modeling data. Each sample is converted into the following format:
Date: YYYY-MM-DD HH:MM:SS
<job post text><eos>
Short texts under 50 tokens were removed after loading because they are usually not full job descriptions, but channel information, recruiter contacts, or references to earlier posts.
The final split is 90/5/5:
The token distribution is shown below:

Continued pretraining was performed with the following setup:
Training settings:
10246431e-4adamw_torch16320.05The best checkpoint is selected by validation loss.
The training and validation losses show a clear downward trend with normal oscillations for a small-domain CPT run.

Use these two cells for inference.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
base_model_id = "Qwen/Qwen3-4B-Base"
adapter_id = "pymlex/qwen3-4b-ds-jobs-expert"
tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
base_model = AutoModelForCausalLM.from_pretrained(
base_model_id,
device_map="auto",
torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()
def generate_continuation(model, tokenizer, prompt, max_new_tokens=260):
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
prompt_len = inputs.input_ids.shape[1]
outputs = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
temperature=0.7,
top_p=0.9,
do_sample=True,
repetition_penalty=1.08,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
)
decoded = tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True)
return decoded.strip()
sample_prompt = (
"#вакансия #ml #python #удаленно\n"
"Ищем Machine Learning Engineer в продуктовую команду. "
"Задачи: построение и запуск моделей, работа с данными, эксперименты, "
"поддержка ML-пайплайнов, участие в развитии продукта. "
)
output = generate_continuation(model, tokenizer, sample_prompt, max_new_tokens=1000)
print("Prompt:")
print(sample_prompt)
print("\nGenerated continuation:")
print(output)
We evaluated both the base model and the tuned model on the test split.
| Model | Perplexity | Loss |
|---|---|---|
| Base Model | 7.014 | 1.9479 |
| Tuned Model | 5.409 | 1.6881 |
The tuned model reduces perplexity by about 22.9% and loss by about 13.3%. This shows that CPT makes the model substantially more confident and accurate on domain-specific job-post text.