ML Ops Skill

Entry point: /​faion-net — invoke this skill for automatic routing to the appropriate domain.

ML Ops Skill

Communication: User's language. Code: English.

Purpose

Handles ML model operations. Covers fine-tuning, evaluation, cost management, and observability.

Context Discovery

Auto-Investigation

Check these project signals before asking questions:

Discovery Questions

question: "What ML operation are you working on?"
header: "Operation Type"
multiSelect: false
options:
  - label: "Fine-tuning LLM"
    description: "Custom model training (OpenAI API, LoRA, QLoRA)"
  - label: "Model evaluation"
    description: "Benchmark performance, LLM-as-judge"
  - label: "Cost optimization"
    description: "Reduce API costs, prompt caching, batching"
  - label: "Observability/​monitoring"
    description: "Track LLM usage, traces, performance"

question: "For fine-tuning: dataset size and approach?"
header: "Fine-tuning Strategy"
multiSelect: false
options:
  - label: "<100 examples - use few-shot prompting instead"
    description: "Too small for fine-tuning, improve prompts"
  - label: "100-1000 examples - OpenAI fine-tuning"
    description: "Use OpenAI API fine-tuning endpoint"
  - label: ">1000 examples - LoRA/​QLoRA"
    description: "Efficient parameter fine-tuning"
  - label: "Not fine-tuning"
    description: "Skip this question"

question: "Which observability tools?"
header: "Monitoring Stack"
multiSelect: true
options:
  - label: "LangSmith (recommended)"
    description: "LangChain native tracing"
  - label: "Langfuse (open-source)"
    description: "Self-hosted observability"
  - label: "Custom logging"
    description: "Build custom tracking"
  - label: "None yet"
    description: "Starting from scratch"

Scope

Quick Start

Methodologies (15)

Fine-tuning (5):

• finetuning-basics: Fundamentals, when to fine-tune
• finetuning-datasets: Data preparation, quality
• fine-tuning-openai-basics: OpenAI API fine-tuning
• fine-tuning-openai-production: Production deployment
• lora-qlora: Efficient fine-tuning, parameter selection

Evaluation (3):

• evaluation-metrics: Accuracy, F1, perplexity, task metrics
• evaluation-framework: LLM-as-judge, human eval
• evaluation-benchmarks: MMLU, HumanEval, industry benchmarks

Cost Optimization (2):

• llm-cost-basics: Token counting, pricing models
• cost-reduction-strategies: Caching, compression, batching

Observability (5):

• llm-observability: Fundamentals, why monitor
• llm-observability-stack: Tools selection
• llm-observability-stack-2026: Latest tools (LangSmith, Langfuse)
• llm-management-observability: End-to-end management

Code Examples

OpenAI Fine-tuning

from openai import OpenAI

client = OpenAI()

# Upload training data
file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

# Create fine-tuning job
job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-4o-mini-2024-07-18",
    hyperparameters={"n_epochs": 3}
)

# Monitor
while True:
    job = client.fine_tuning.jobs.retrieve(job.id)
    if job.status == "succeeded":
        break

LoRA Fine-tuning

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/​Llama-3-8b")

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.1,
    bias="none"
)

model = get_peft_model(model, lora_config)

Cost Tracking

import tiktoken

def count_tokens(text, model="gpt-4o"):
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))

def estimate_cost(prompt, completion, model="gpt-4o"):
    prompt_tokens = count_tokens(prompt, model)
    completion_tokens = count_tokens(completion, model)

    # GPT-4o pricing
    prompt_cost = prompt_tokens * 0.000005
    completion_cost = completion_tokens * 0.000015

    return prompt_cost + completion_cost

LLM Observability with LangSmith

from langsmith import traceable

@traceable
def rag_pipeline(query: str) -> str:
    # Retrieval
    docs = retrieve(query)

    # Generation
    response = generate(query, docs)

    return response

Fine-tuning Decision Matrix

Cost Reduction Strategies

Evaluation Frameworks

Related Skills

---

ML Ops v1.0 | 15 methodologies