ATLAS-NIST-Measure

ATLAS-NIST-Measure: Welfare Risk Assessment SLM

Project Overview

This model is a specialized Small Language Model (SLM) designed for the Welfare and Public Service domain, developed as part of the ATLAS V3.0 'Brain Build' Super-Prompt initiative. It is fine-tuned to evaluate risk in welfare applications, specifically focusing on unemployment benefit scenarios, categorizing them into actionable decisions.

Anna Ko Milestone

This release marks the Anna Ko Milestone, incorporating specific requirements for balanced class distribution, integration of unstructured caseworker notes, and rigorous Human-in-the-Loop (HITL) validation logic. The dataset engineering ensures the model is robust against diverse input conditions while adhering to regulatory constraints.

Validation Results

The model was fine-tuned on the Unemployment HITL Dataset (3,000 samples) and evaluated on a held-out test set of 600 samples.

• Macro F1 Score: 0.8522
• Overall Accuracy: 85%

Class-wise Performance (F1-Score)

• auto_approve: 1.00 (Perfect)
• auto_deny: 0.89
• auto_review: 0.83
• escalate_to_human: 0.69

Deterministic Logic & Safety

A key insight from this milestone is the 100% precision and recall achieved for the auto_approve class. This validates the safety of automating low-risk approvals, as the model perfectly learned the deterministic logic required for standard cases. This allows agencies to confidently automate routine approvals while reserving human attention for complex (escalate_to_human) or ambiguous (auto_review) cases.

Usage

Python (Unsloth/Transformers)

from unsloth import FastLanguageModel
import json
import torch

# Load model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "nislam-mics/ATLAS-NIST-Measure",
    load_in_4bit = True
)
FastLanguageModel.for_inference(model)

# Define input
instruction = "Evaluate the unemployment benefit application risk."
input_data = {
    "structured_inputs": {"employment_status_declared": "unemployed", "income_verification": "verified"},
    "decision_context": {"case_age_days": 10}
}

# Format prompt
prompt = f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{json.dumps(input_data)}

### Response:
"""

# Generate
inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0])