ORCA6

ORCA6 v0.1-rc1

ORCA6 is an orchestration-advisor model project focused on AI tool selection, workflow architecture, RAG design, model-routing tradeoffs, and developer automation strategy.

Current best local adapter:

qwen3_14b_orca_refusal_smoke/

This checkpoint is a guarded local release candidate, not a broad public-quality model. It should be used with the source-packet runtime guard documented in the evaluation report, runbook, and local adapter CLI.

Intended Use

• Recommend orchestration patterns for AI developer workflows.
• Compare tools such as n8n, LangGraph, LiteLLM, Qdrant, Langfuse, Promptfoo, MCP servers, local inference stacks, and related infrastructure.
• Provide architecture tradeoffs, implementation plans, and conservative next steps.

Not Intended For

• Executing code or tools directly.
• Legal, medical, financial, or safety-critical decisions.
• General code-completion benchmarks unrelated to orchestration.

Training Data

• SFT train rows: 41
• SFT validation rows: 3
• DPO preference rows: 30
• Preference source: auto-graded bootstrap preferences
• Corpus source: GitHub documentation chunks from the ORCA6 pilot retrieval set.
• Final SFT source mix: {"graded_preference": 27, "grounded_sft_builder": 8, "refusal_sft_builder": 6}
• Final answer word count: min=102, max=239, avg=187.05
• Grounded SFT rows include retrieved-evidence citation examples and refusal hard negatives for empty evidence, unsupported claims, high-risk automation, medical-record access, and certification/compliance claims.

Training Dataset Audit

Models and Infrastructure Used

Training Rounds

Earlier 2048-token/r64 and 1024-token/r32 grounded-profile attempts hit CUDA OOM with the longer grounded examples. The release fit-check profile settled on 512 tokens and LoRA rank 16 on the RTX 3090.

Retrieval and Vector Database Stack

The model was trained and evaluated around a source-packet workflow rather than free-form citation generation.

Evaluation

Latest recorded retrieval metrics:

{
  "queries": 20,
  "calibrated_pass_at_1": 0.7,
  "calibrated_pass_at_3": 1.0,
  "calibrated_pass_at_5": 1.0,
  "hit_at_1": 1.0,
  "hit_at_3": 1.0,
  "hit_at_5": 1.0,
  "all_expected_at_3": 0.95,
  "all_expected_at_5": 1.0,
  "all_expected_at_10": 1.0
}

Latest guarded grounded-answer eval:

{
  "outputs": "evals/qwen3_14b_orca_refusal_guarded_eval_outputs.jsonl",
  "total": 12,
  "passed": 12,
  "pass_rate": 1.0,
  "by_type": {
    "source_packet": {
      "passed": 8,
      "total": 8
    },
    "hard_negative": {
      "passed": 4,
      "total": 4
    }
  }
}

Expanded v0.1-rc1 grounded release eval:

{
  "total": 54,
  "passed": 54,
  "pass_rate": 1.0,
  "by_type": {
    "source_packet": {
      "passed": 50,
      "total": 50
    },
    "hard_negative": {
      "passed": 4,
      "total": 4
    }
  },
  "outputs": "evals/qwen3_14b_orca_refusal_release_grounded_outputs.jsonl"
}

Evaluation Matrix

Unguarded refusal eval passed 11/12. The remaining unguarded failure was an empty-evidence, high-risk payment automation prompt where the model invented a source citation. The current gate therefore requires the runtime source-packet guard.

Artifact Statistics

Release artifact checksums are recorded in:

release/release_manifest.json

Limitations

• The current dataset is small and should be treated as a v0 bootstrap.
• Auto-graded preferences are useful for pipeline validation but should be replaced or supplemented with human preference labels.
• Recommendations are only as current as the indexed source corpus.
• The adapter is not intended for unguarded citation-heavy answering. Use a runtime prompt guard that forbids invented source IDs, URLs, integrations, certifications, guarantees, and high-risk actions without retrieved evidence.
• The model must not execute tools or approve irreversible actions.

Release Notes

• Generated: 2026-06-26
• Version: v0.1-rc1
• Base model target: Qwen/Qwen3-14B
• Adapter target: qwen3_14b_orca_refusal_smoke/

<!-- ORCA6_WEEKLY_TUNING_START -->

Weekly Fine-Tuning Updates

<!-- ORCA6_WEEKLY_TUNING_END -->