CodeBaitSmol-135M

Overview

CodeBaitSmol 135M is a lightweight transformer-based causal language model designed to detect software bugs and code anomalies. The model is trained on coder-focused datasets including bug-fix samples, static analysis violations, and open-source repositories. It is optimized for fast inference, developer tooling integration, and automated code review workflows.

Model Architecture

Base Architecture: LLaMA-style Decoder Transformer Parameter Size: ~135 Million Transformer Layers: 12 Hidden Size: 768 Attention Heads: 12 Key-Value Heads: 12 Head Dimension: 64 MLP Intermediate Size: 2048 Activation Function: SiLU Normalization: RMSNorm Vocabulary Size: 32000 tokens Max Context Length: 1024 tokens Positional Encoding: Rotary Positional Embeddings (RoPE) Precision: Float32

Configuration Summary

Model Type: LlamaForCausalLM

Attention Bias: Disabled

Attention Dropout: 0.0

MLP Bias: Disabled

Initializer Range: 0.02

Rope Theta: 10000.0

Token IDs: BOS=1, EOS=2

Training Details

The model is trained using causal language modeling with focus on bug detection. Training includes:

Pretraining

Multi-language code understanding

Syntax and structural learning

Bug-Focused Fine-Tuning

Buggy vs corrected code samples

Error localization

Secure coding pattern learning

Dataset Sources include:

Open-source repositories

Bug-fix commit diffs

Static analysis datasets

Synthetic bug injection samples

Competitive programming solutions

Capabilities

Syntax Bug Detection

Missing brackets

Invalid tokens

Indentation errors

Logical Bug Detection

Infinite loops

Null reference risks

Incorrect conditions

Variable misuse

Security Vulnerability Detection

SQL injection patterns

Command injection risks

Unsafe deserialization

Buffer overflow patterns

Performance Anti-Pattern Detection

Redundant computations

Memory inefficiencies

Blocking async operations

Supported Languages

Python

Java

JavaScript / TypeScript

C / C++

Go

Rust

Shell scripting

Inference Usage

Input: Source code or structured prompt requesting bug detection.

Output:

Bug identification

Explanation

Suggested fix

Example: Input: for i in range(len(arr)): print(arr[i+1])

Output: Bug: Index out of range risk Explanation: Accessing arr[i+1] exceeds array boundary Fix: Adjust loop range or indexing

Performance Characteristics

Strengths:

Low memory footprint

Fast inference

Strong pattern recognition for common coding errors

Limitations:

Context limited to 1024 tokens

Reduced accuracy for extremely large codebases

Not a formal verification system

Deployment Recommendations

Supported Deployment:

IDE plugins

CI/CD pipelines

Static analysis tools

Edge developer tooling

Packaging Options:

GGUF

ONNX

HuggingFace Transformers

Quantized INT4 / INT8 variants

Hardware Requirements

Minimum:

CPU: 4 cores

RAM: 4GB

GPU: Optional (4GB VRAM recommended)

Evaluation Metrics

Bug Detection Accuracy

Fix Suggestion Quality

Security Vulnerability Recall

Static Analyzer Agreement

Human Code Review Ratings

Safety Notice

The model assists in bug detection but does not guarantee correctness. Human review and static analysis tools are recommended for production-critical code validation.

CodeBaitLarge 7B is scheduled to release on 13/05/2026.