Finalize Your LLMOps Skill

Finalize Your LLMOps Skill

Throughout Part 8, you built specialized skills for each LLMOps capability:

Now you will consolidate these into one production-ready llmops-fine-tuner skill that composes all capabilities. This is Layer 3 work: transforming specialized knowledge into reusable intelligence.

Why Consolidation Matters

The Problem with 9 Separate Skills:

When you need to fine-tune a model for production, you currently must:

1. Invoke llmops-decision-framework to determine if fine-tuning is appropriate
2. Invoke llmops-compute-planner to budget VRAM
3. Invoke llmops-data-engineer to create datasets
4. Invoke llmops-fine-tuner for training
5. Invoke model-evaluation for quality gates
6. Invoke model-serving for deployment
7. Invoke agent-integration for framework connection

That is 7 separate skill invocations with manual handoffs between each. Errors compound. Context is lost.

The Solution: Unified Orchestration

A consolidated skill:

• Accepts a single specification
• Orchestrates all pipeline stages internally
• Maintains context across stages
• Produces a deployable Digital FTE

This transforms your skills from components into a product.

Skill Consolidation Architecture

The Composition Pattern

Your consolidated skill follows this architecture:

llmops-fine-tuner (Unified)
├── Decision Layer
│   └── When to fine-tune? (from Ch.61)
│
├── Planning Layer
│   └── Compute requirements (from Ch.62)
│
├── Data Layer
│   └── Dataset creation (from Ch.63)
│
├── Training Layer
│   ├── Fine-tuning (from Ch.64)
│   ├── Merging (from Ch.67)
│   └── Alignment (from Ch.68)
│
├── Validation Layer
│   └── Quality gates (from Ch.69)
│
└── Deployment Layer
    ├── Serving (from Ch.70)
    └── Integration (from Ch.71)

Cohesion Analysis

Not all skills should merge. Ask:

Analysis for LLMOps Skills:

Result: All skills merge into unified llmops-fine-tuner.

Creating Your Consolidated Skill

Step 1: Define the Unified SKILL.md

Create your consolidated skill:

mkdir -p .claude/​skills/​llmops-fine-tuner

SKILL.md Structure:

---
name: llmops-fine-tuner
description: "End-to-end LLMOps pipeline for fine-tuning, evaluating, and deploying custom models as Digital FTEs. Use when building production model deployments."
---

# LLMOps Fine-Tuner Skill

## Purpose

Complete pipeline from specification to deployed Digital FTE.

## When to Use

- Building custom models for specific domains
- Creating proprietary AI products
- Deploying fine-tuned models to production

## Pipeline Stages

### Stage 1: Decision (from Ch.61)
Evaluate whether fine-tuning is appropriate for this use case.

[Include decision framework here]

### Stage 2: Planning (from Ch.62)
Calculate VRAM budget and hardware requirements.

[Include compute planning here]

### Stage 3: Data (from Ch.63)
Create and validate training dataset.

[Include data engineering here]

### Stage 4: Training (from Ch.64, 67, 68)
Execute fine-tuning with optional merging and alignment.

[Include training workflows here]

### Stage 5: Evaluation (from Ch.69)
Run quality gates before deployment.

[Include evaluation criteria here]

### Stage 6: Deployment (from Ch.70, 71)
Deploy model and integrate with agent frameworks.

[Include deployment and integration here]

## Specification Template

When invoking this skill, provide:

- **Domain**: What expertise is this model encoding?
- **Base Model**: Which model to fine-tune
- **Dataset Source**: Where does training data come from
- **Quality Target**: What accuracy threshold for deployment
- **Deployment Target**: Where model will run (Ollama, API, etc.)

## Constraints

- Colab Free Tier compatible (T4 GPU)
- Total cost < $1
- Completion time < 4 hours

Step 2: Merge Skill Content

For each specialized skill, extract the core patterns and integrate:

From llmops-decision-framework:

# Decision criteria
def should_fine_tune(use_case):
    """
    Returns True if fine-tuning is appropriate.

    Fine-tune when:
    - Prompting consistently fails (>30% error rate)
    - Domain requires specialized vocabulary
    - Latency requirements mandate smaller model
    - Cost optimization requires local deployment

    Don't fine-tune when:
    - Prompting achieves 90%+ accuracy
    - Task is one-off or experimental
    - No clear evaluation criteria exist
    """
    pass

From llmops-compute-planner:

# VRAM budget calculation
def calculate_vram_budget(model_size_b, batch_size=1, lora_rank=16):
    """
    Estimate VRAM requirements for training.

    Formula: VRAM = model_params * 4 bytes + optimizer_state * 8 bytes + activations
    With LoRA: Reduce by ~80% (only training adapter weights)

    T4 GPU (16GB):
    - 1B params: Works with batch_size=4
    - 3B params: Works with batch_size=2
    - 7B params: Works with batch_size=1, gradient_checkpointing
    - 8B params: Works with 4-bit quantization
    """
    pass

From llmops-data-engineer:

# Dataset creation patterns
def create_training_dataset(examples, format="chat"):
    """
    Create JSONL dataset for fine-tuning.

    Format options:
    - chat: {"messages": [{"role": "user", "content": ...}, ...]}
    - instruct: {"instruction": ..., "input": ..., "output": ...}
    - completion: {"text": "Input: ... Output: ..."}
    """
    pass

Step 3: Define Handoff Protocols

Each stage produces artifacts consumed by the next:

Stage 1 (Decision) → decision_report.json
    └── Inputs: use_case_description
    └── Outputs: should_proceed, reasoning, constraints

Stage 2 (Planning) → compute_plan.json
    └── Inputs: decision_report, model_choice
    └── Outputs: vram_budget, batch_size, hardware_requirements

Stage 3 (Data) → dataset.jsonl + data_report.json
    └── Inputs: raw_examples, format_spec
    └── Outputs: training_data, validation_data, quality_metrics

Stage 4 (Training) → adapter_weights/ + training_report.json
    └── Inputs: base_model, dataset, compute_plan
    └── Outputs: trained_adapter, loss_curves, checkpoint_path

Stage 5 (Evaluation) → eval_report.json
    └── Inputs: trained_model, test_dataset, quality_thresholds
    └── Outputs: accuracy, pass_fail, regression_analysis

Stage 6 (Deployment) → deployed_model + integration_config.json
    └── Inputs: trained_model, deployment_target
    └── Outputs: model_endpoint, ollama_modelfile, agent_config

Step 4: Validate Consolidation

Test your consolidated skill end-to-end:

# Test with minimal example
claude --skill llmops-fine-tuner "
Create a Task API model:
- Domain: Task management tool-calling
- Base: Qwen 2.5 3B Instruct
- Data: 200 synthetic examples
- Target: 90% tool-calling accuracy
- Deploy: Local Ollama
"

Expected Output:

Stage 1: Decision PASS - Fine-tuning appropriate for tool-calling specialization
Stage 2: Planning COMPLETE - T4 compatible, batch_size=2, LoRA rank=16
Stage 3: Data COMPLETE - 200 examples generated, 80/​20 train/​val split
Stage 4: Training COMPLETE - Loss converged: 0.12, Epoch 3/​3
Stage 5: Evaluation PASS - Tool accuracy: 94%, threshold: 90%
Stage 6: Deployment COMPLETE - Ollama model registered: task-api-qwen3b

Digital FTE Ready: task-api-qwen3b

Capability Preservation Checklist

Verify your consolidated skill preserves all original capabilities:

Production Readiness Criteria

Your consolidated skill is production-ready when:

1. Single Invocation: Full pipeline from one specification
2. Stage Visibility: Clear progress reporting for each stage
3. Error Recovery: Graceful handling with stage-specific retry
4. Artifact Persistence: All intermediate outputs saved for debugging
5. Constraint Compliance: Stays within Colab Free Tier limits
6. Consistent Quality: Produces equivalent results to individual skills

What You Built

Your consolidated llmops-fine-tuner skill:

This is intelligence accumulation: 11 chapters of knowledge compressed into one reusable skill.

Try With AI

Prompt 1: Design Your Skill Structure

I'm consolidating these LLMOps skills into one unified skill:
- Decision framework (when to fine-tune)
- Compute planning (VRAM budgeting)
- Data engineering (dataset creation)
- Training (fine-tuning, merging, alignment)
- Evaluation (quality gates)
- Deployment (serving, integration)

Help me design the SKILL.md structure. What sections should it have?
How should stages hand off to each other? What specification template
should users provide when invoking this skill?

What you're learning: Skill architecture design—how to structure reusable intelligence for production use.

Prompt 2: Test Capability Preservation

I've consolidated my LLMOps skills. Help me create a test plan to verify
all capabilities are preserved. For each original skill, give me:
1. A test prompt that exercises that capability
2. Expected output that proves it works
3. A failure mode to watch for

What you're learning: Validation methodology—ensuring consolidation doesn't lose specialized capabilities.

Prompt 3: Connect to Your Domain

I'm building a consolidated LLMOps skill for [your domain: legal, medical,
finance, etc.]. My original skills handle [list your specialized skills].
Help me analyze: Which skills should merge? Which should stay separate?
Ask me questions about how I use each skill to determine the right
composition.

What you're learning: Cohesion analysis—understanding when to merge vs. separate based on usage patterns.

Safety Note

When consolidating skills, preserve all safety mechanisms from individual skills. A unified skill must maintain the same guardrails as its components—never optimize away safety checks for convenience.