inframind-0.5b-dapo

InfraMind-DAPO: Infrastructure-as-Code Model with Direct Advantage Policy Optimization

InfraMind-DAPO is a 0.5B parameter language model fine-tuned for Infrastructure-as-Code (IaC) generation using DAPO (Direct Advantage Policy Optimization) - an advanced reinforcement learning technique that builds upon GRPO.

Model Description

Training Pipeline

Qwen2.5-0.5B-Instruct → GRPO Training → inframind-grpo → DAPO Training → inframind-dapo
                        (Stage 1)                        (Stage 2 - This Model)

This model is the second stage of InfraMind training, starting from the GRPO-trained checkpoint and applying DAPO innovations for enhanced learning.

What is DAPO?

Direct Advantage Policy Optimization (DAPO) is an advanced RL algorithm that improves upon GRPO with four key innovations:

Why DAPO After GRPO?

Evaluation Results

Evaluated on InfraMind-Bench (110 held-out test samples) across:

• Terraform (AWS, GCP, Azure)
• Kubernetes (Deployments, Services, Ingress)
• Docker (Dockerfile, docker-compose)
• CI/CD (GitHub Actions, GitLab CI)

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load DAPO model
model = AutoModelForCausalLM.from_pretrained("srallabandi0225/inframind-0.5b-dapo")
tokenizer = AutoTokenizer.from_pretrained("srallabandi0225/inframind-0.5b-dapo")

# Generate Terraform
prompt = """### Instruction:
Create Terraform for AWS EC2 instance
### Input:
t3.micro instance type
### Response:
"""

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.pad_token_id
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Example Output

resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.micro"

  tags = {
    Name = "web-server"
  }
}

Supported IaC Categories

Training Details

DAPO Configuration

Training:
  epochs: 2
  batch_size: 16 (effective)
  learning_rate: 5e-6
  beta (KL): 0.0  # Pure DAPO - no KL penalty
  generations_per_prompt: 8

DAPO Innovations:
  clip_higher:
    epsilon_low: 0.2
    epsilon_high: 0.28
  dynamic_sampling: true
  token_level_loss: true
  overlong_punishment:
    enabled: true
    soft_penalty: true

LoRA:
  r: 16
  alpha: 32
  target_modules: [q_proj, k_proj, v_proj, o_proj]

Reward Function

Domain-specific reward for IaC quality:

Reward = α × Syntax + β × Correctness + γ × Format

Where:
- Syntax (α=0.4): Valid resource declarations
- Correctness (β=0.3): Correct resource types
- Format (γ=0.3): Proper structure

GRPO vs DAPO Comparison

Hardware Requirements

The 0.5B model is small enough to run on edge devices, making it suitable for:

• Air-gapped environments
• Local development
• CI/CD pipelines
• IoT/Edge infrastructure

Limitations

• IaC-specific: Optimized for infrastructure tasks, not general conversation
• English only: Training data is in English
• No execution: Generates code, does not execute or validate against real infrastructure
• Version-sensitive: Generated code may use older API versions
• Security: Always review generated code for security best practices

Out-of-Scope Uses

• Legal or medical advice
• General-purpose chatbot
• Executing infrastructure changes without human review
• Production deployment without validation

Intended Use

Primary Use Cases

• Generating Terraform configurations
• Creating Kubernetes manifests
• Writing Dockerfiles and docker-compose
• Building CI/CD pipelines
• Infrastructure automation scripting

Users

• DevOps engineers
• Platform engineers
• SREs
• Cloud architects
• Infrastructure developers

Training Data

InfraMind-Bench: 2000+ IaC tasks in Alpaca format

Ethical Considerations

• Model may generate insecure configurations if not prompted for security
• Generated infrastructure code should always be reviewed before deployment
• Model does not have access to real infrastructure or credentials
• Users are responsible for validating generated code against their security policies

Citation

@misc{rallabandi2024inframind,
  title={InfraMind: Fine-tuning Small Language Models for Infrastructure-as-Code Generation with Reinforcement Learning},
  author={Rallabandi, Sai Kiran},
  year={2024},
  publisher={HuggingFace},
  url={https://huggingface.co/srallabandi0225/inframind-0.5b-dapo}
}

Links

• GitHub: github.com/saikiranrallabandi/inframind
• GRPO Model: srallabandi0225/inframind-0.5b-grpo
• DAPO Model: srallabandi0225/inframind-0.5b-dapo

Acknowledgments

• Qwen Team for the base model
• DeepSeek for GRPO
• NVIDIA NeMo for DAPO reference
• TRL for training infrastructure

Model Card Contact

Author: Sai Kiran Rallabandi GitHub: @saikiranrallabandi