laya-issue-triage

laya-issue-triage

Laya fine-tuned to triage GitHub issues. It answers two typed questions about an issue in a single forward pass, with no text generation:

Results

Measured on 3868 issues from three repositories held out of training entirely (huggingface/transformers, facebook/react, microsoft/TypeScript), so these are generalization numbers, not memorization.

Per-class F1 on issue_type:

Per-repository accuracy on issue_type:

Median latency on a T4 during evaluation: 78.8 ms per issue.

Scope of these numbers

This model is evaluated only on GitHub issue triage. It has not been run on the LocalLLaMA/typed-decisions benchmark, so its scores are not comparable to numbers reported there for the base Laya checkpoints or for any other system. Different task, different label space, different data.

Usage

import laya

agent = laya.Agent("harikarthikmanyam/laya-issue-triage")
questions = {
    "issue_type": {
        "type": "choice",
        "instructions": "What kind of GitHub issue do `title` and `body` describe?",
        "criteria": {
            "bug": "something is broken: a crash, an error, wrong output, or a regression from behaviour that used to work",
            "feature": "a request for new functionality, an enhancement, or a proposal to change how something works",
            "question": "the author is asking how to use the project or why it behaves a certain way, not reporting a defect",
            "docs": "the documentation is missing, wrong, unclear, or needs an example"
        }
    },
    "needs_more_info": {
        "type": "noul",
        "instructions": "Must a maintainer ask the author of `body` for more information -- reproduction steps, a version number, logs, or a code sample -- before this issue can be worked on?"
    }
}

state = {"title": "Crash when opening a file with a BOM",
         "body": "v1.4.2 on Linux. Steps: open any UTF-8-BOM file, editor segfaults."}
result = agent.predict(state, questions)
print(result["answers"]["issue_type"]["choice"])

The question definitions above are part of the contract: this model was trained against those exact instructions and criteria strings, so changing them changes behaviour.

Training data

~7.7k closed issues carrying maintainer-applied labels, harvested from 14 public repositories and balanced across the four types. Labels come from the maintainers who triaged each issue, not from a teacher model.

Dataset, harvesting code and evaluation script: https://github.com/manyamkarthik/laya-issue-triage

Limitations

• English issue text only; use laya-multilingual as a base for other languages.
• Issue text is truncated to roughly 900 characters, so decisions rest on the title and opening paragraphs.
• needs_more_info is supervised by maintainer labels such as needs-repro, which projects apply inconsistently; it is a weaker signal than issue_type.
• Confidences are uncalibrated unless you fit temperatures (see the notebook).