LangGraph Application Fine-Tuning Skill

LangGraph Application Fine-Tuning Skill

A skill for iteratively optimizing prompts and processing logic in each node of a LangGraph application based on evaluation criteria.

📋 Overview

This skill executes the following process to improve the performance of existing LangGraph applications:

1. Load Objectives: Retrieve optimization goals and evaluation criteria from .langgraph-architect/​fine-tune.md (if this file doesn't exist, help the user create it based on their requirements)
2. Identify Optimization Targets: Extract nodes containing LLM prompts using Serena MCP (if Serena MCP is unavailable, investigate the codebase using ls, read, etc.)
3. Baseline Evaluation: Measure current performance through multiple runs
4. Implement Improvements: Identify the most effective improvement areas and optimize prompts and processing logic
5. Re-evaluation: Measure performance after improvements
6. Iteration: Repeat steps 4-5 until goals are achieved

Important Constraint: Only optimize prompts and processing logic within each node without modifying the graph structure (nodes, edges configuration).

🎯 When to Use This Skill

Use this skill in the following situations:

1. When performance improvement of existing applications is needed

   • Want to improve LLM output quality
   • Want to improve response speed
   • Want to reduce error rate

2. When evaluation criteria are clear

   • Optimization goals are defined in .langgraph-architect/​fine-tune.md
   • Quantitative evaluation methods are established

3. When improvements through prompt engineering are expected

   • Improvements are likely with clearer LLM instructions
   • Adding few-shot examples would be effective
   • Output format adjustment is needed

📖 Fine-Tuning Workflow Overview

Phase 1: Preparation and Analysis

Purpose: Understand optimization targets and current state

Main Steps:

1. Load objective setting file (.langgraph-architect/​fine-tune.md)
2. Identify optimization targets (Serena MCP or manual code investigation)
3. Create optimization target list (evaluate improvement potential for each node)

→ See workflow.md for details

Phase 2: Baseline Evaluation

Purpose: Quantitatively measure current performance

Main Steps: 4. Prepare evaluation environment (test cases, evaluation scripts) 5. Baseline measurement (recommended: 3-5 runs) 6. Analyze baseline results (identify problems)

Important: When evaluation programs are needed, create evaluation code in a specific subdirectory (users may specify the directory).

→ See workflow.md and evaluation.md for details

Phase 3: Iterative Improvement

Purpose: Data-driven incremental improvement

Main Steps: 7. Prioritization (select the most impactful improvement area) 8. Implement improvements (prompt optimization, parameter tuning) 9. Post-improvement evaluation (re-evaluate under the same conditions) 10. Compare and analyze results (measure improvement effects) 11. Decide whether to continue iteration (repeat until goals are achieved)

→ See workflow.md and prompt_optimization.md for details

Phase 4: Completion and Documentation

Purpose: Record achievements and provide future recommendations

Main Steps: 12. Create final evaluation report (improvement content, results, recommendations) 13. Code commit and documentation update

→ See workflow.md for details

🔧 Tools and Technologies Used

MCP Server Utilization

• Serena MCP: Codebase analysis and optimization target identification

  • find_symbol: Search for LLM clients
  • find_referencing_symbols: Identify prompt construction locations
  • get_symbols_overview: Understand node structure

• Sequential MCP: Complex analysis and decision making

  • Determine improvement priorities
  • Analyze evaluation results
  • Plan next actions

Key Optimization Techniques

1. Few-Shot Examples: Accuracy +10-20%
2. Structured Output Format: Parsing errors -90%
3. Temperature/​Max Tokens Adjustment: Cost -20-40%
4. Model Selection Optimization: Cost -40-60%
5. Prompt Caching: Cost -50-90% (on cache hit)

→ See prompt_optimization.md for details

📚 Related Documentation

Detailed guidelines and best practices:

• workflow.md - Fine-tuning workflow details (execution procedures and code examples for each phase)
• evaluation.md - Evaluation methods and best practices (metric calculation, statistical analysis, test case design)
• prompt_optimization.md - Prompt optimization techniques (10 practical methods and priorities)
• examples.md - Practical examples collection (copy-and-paste ready code examples and template collection)

⚠️ Important Notes

1. Preserve Graph Structure

   • Do not add or remove nodes or edges
   • Do not change data flow between nodes
   • Maintain state schema

2. Evaluation Consistency

   • Use the same test cases
   • Measure with the same evaluation metrics
   • Run multiple times to confirm statistically significant improvements

3. Cost Management

   • Consider evaluation execution costs
   • Adjust sample size as needed
   • Be mindful of API rate limits

4. Version Control

   • Git commit each iteration's changes
   • Maintain rollback-capable state
   • Record evaluation results

🎓 Fine-Tuning Best Practices

1. Start Small: Optimize from the most impactful node
2. Measurement-Driven: Always perform quantitative evaluation before and after improvements
3. Incremental Improvement: Validate one change at a time, not multiple simultaneously
4. Documentation: Record reasons and results for each change
5. Iteration: Continuously improve until goals are achieved

🔗 Reference Links

• LangGraph Official Documentation
• Prompt Engineering Guide