LLM Architect

LLM Architect

Purpose

Provides expert large language model system architecture for designing, deploying, and optimizing LLM applications at scale. Specializes in model selection, RAG (Retrieval Augmented Generation) pipelines, fine-tuning strategies, serving infrastructure, cost optimization, and safety guardrails for production LLM systems.

When to Use

• Designing end-to-end LLM systems from requirements to production
• Selecting models and serving infrastructure for specific use cases
• Implementing RAG (Retrieval Augmented Generation) pipelines
• Optimizing LLM costs while maintaining quality thresholds
• Building safety guardrails and compliance mechanisms
• Planning fine-tuning vs RAG vs prompt engineering strategies
• Scaling LLM inference for high-throughput applications

Quick Start

Invoke this skill when:

• Designing end-to-end LLM systems from requirements to production
• Selecting models and serving infrastructure for specific use cases
• Implementing RAG (Retrieval Augmented Generation) pipelines
• Optimizing LLM costs while maintaining quality thresholds
• Building safety guardrails and compliance mechanisms

Do NOT invoke when:

• Simple API integration exists (use backend-developer instead)
• Only prompt engineering needed without architecture decisions
• Training foundation models from scratch (almost always wrong approach)
• Generic ML tasks unrelated to language models (use ml-engineer)

Decision Framework

Model Selection Quick Guide

RAG vs Fine-Tuning Decision Tree

Need to customize LLM behavior?
│
├─ Need domain-specific knowledge?
│  ├─ Knowledge changes frequently?
│  │  └─ RAG (Retrieval Augmented Generation)
│  └─ Knowledge is static?
│     └─ Fine-tuning OR RAG (test both)
│
├─ Need specific output format/​style?
│  ├─ Can describe in prompt?
│  │  └─ Prompt engineering (try first)
│  └─ Format too complex for prompt?
│     └─ Fine-tuning
│
└─ Need latency <100ms?
   └─ Fine-tuned small model (7B-13B)

Architecture Pattern

[Client] → [API Gateway + Rate Limiting]
              ↓
         [Request Router]
          (Route by intent/​complexity)
              ↓
    ┌────────┴────────┐
    ↓                 ↓
[Fast Model]    [Powerful Model]
(Haiku/​Small)   (Sonnet/​Large)
    ↓                 ↓
[Cache Layer] ← [Response Aggregator]
    ↓
[Logging & Monitoring]
    ↓
[Response to Client]

Core Workflow: Design LLM System

1. Requirements Gathering

Ask these questions:

• Latency: What's the P95 response time requirement?
• Scale: Expected requests/​day and growth trajectory?
• Accuracy: What's the minimum acceptable quality? (measurable metric)
• Cost: Budget constraints? ($/​request or $/​month)
• Data: Existing datasets for evaluation? Sensitivity level?
• Compliance: Regulatory requirements? (HIPAA, GDPR, SOC2, etc.)

2. Model Selection

def select_model(requirements):
    if requirements.latency_p95 < 100:  # milliseconds
        if requirements.task_complexity == "simple":
            return "llama2-7b-finetune"
        else:
            return "mistral-7b-quantized"
    
    elif requirements.latency_p95 < 2000:
        if requirements.budget == "unlimited":
            return "claude-3-opus"
        elif requirements.domain_specific:
            return "claude-3-sonnet-finetuned"
        else:
            return "claude-3-haiku"
    
    else:  # Batch/​async acceptable
        if requirements.accuracy_critical:
            return "gpt-4-with-ensemble"
        else:
            return "batch-api-cheapest-tier"

3. Prototype & Evaluate

# Run benchmark on eval dataset
python scripts/​evaluate_model.py \
  --model claude-3-sonnet \
  --dataset data/​eval_1000_examples.jsonl \
  --metrics accuracy,latency,cost

# Expected output:
# Accuracy: 94.3%
# P95 Latency: 1,245ms
# Cost per 1K requests: $2.15

4. Iteration Checklist

• Latency P95 meets requirement? If no → optimize serving (quantization, caching)
• Accuracy meets threshold? If no → improve prompts, fine-tune, or upgrade model
• Cost within budget? If no → aggressive caching, smaller model routing, batching
• Safety guardrails tested? If no → add content filters, PII detection
• Monitoring dashboards live? If no → set up Prometheus + Grafana
• Runbook documented? If no → document common failures and fixes

Cost Optimization Strategies

Safety Checklist

• Content filtering tested against adversarial examples
• PII detection and redaction validated
• Prompt injection defenses in place
• Output validation rules implemented
• Audit logging configured for all requests
• Compliance requirements documented and validated

Red Flags - When to Escalate

Quick Reference: Performance Targets

Additional Resources

• Detailed Technical Reference: See REFERENCE.md

  • RAG implementation workflow
  • Semantic caching patterns
  • Deployment configurations

• Code Examples & Patterns: See EXAMPLES.md

  • Anti-patterns (fine-tuning when prompting suffices, no fallback)
  • Quality checklist for LLM systems
  • Resilient LLM call patterns