collie-ent-direct-0.6b

COLLIE enterprise cataloger — 0.6B

COLLIE converts extracted document text into a six-facet JSON catalog card:

{"subject":["database_performance_tuning"],"type":"issue_report",
 "audience":"internal_team","time":"current","purpose":"request",
 "content_flags":["code_snippets","error_traceback"]}

This is the recommended anchor-free, direct-SFT checkpoint from the COLLIE project. It is a descriptive cataloger, not a sensitivity or importance classifier.

Usage

Use the project CLI for the checkpoint's prompt and decoding behavior:

git clone https://github.com/harshsinghal/COLLIE
cd COLLIE
python3.12 -m venv .venv
.venv/bin/pip install "torch>=2.6,<3" "transformers>=5.14.1,<6"
.venv/bin/python collie.py --file report.txt

For direct Transformers use, copy the CLI's SYSTEM prompt. A language-neutral output contract is published with the checkpoint as collie-card.schema.json.

Compatibility for this checkpoint:

• Python 3.10+
• Transformers 5.14.1 through the latest compatible 5.x release
• PyTorch 2.6+

Training data and procedure

• Base: Qwen/Qwen3-0.6B
• Direct supervised fine-tuning
• 5,480 public-corpus teacher inputs across email, extracted PDFs, public chat, public source code, Apache JIRA, and system logs
• 4,655 final training rows after parsing, schema, register-holdout, and compound-value filters
• Apache JIRA and LogHub were held out from final training
• Source documents are not redistributed because their upstream terms vary

See the repository for sampling code, prompt versions, experimental artifacts, and the negative-result journal.

Evaluation

On the original 3,705-document reference-free faithfulness evaluation, the published run reported:

• 79% precise emitted subjects
• 9% wrong emitted subjects
• 0.39 missed major subjects per document
• 92% correct artifact type

These values were produced by an LLM judge and should not be interpreted as human-ground-truth accuracy. The repository's evaluation-v2 tooling adds strict response schemas, per-document audit artifacts, source slices, document-clustered bootstrap intervals, a second-judge path, and a human-audit sample. The original split also contained 18 exact train/evaluation duplicate texts; evaluation v2 removes exact and near duplicates. All historical model outputs parsed as JSON, but 3,630/3,705 (98.0%) satisfy the stricter v1.1 card schema; v2 reports the remainder as model failures rather than dropping them.

Limitations

• English-dominant training data.
• Inputs are text. PDFs must be text-extracted first; the CLI can do this with its documents extra.
• The training excerpts were capped at 4,000 characters. The CLI chunks longer documents, but cross-chunk aggregation has not yet received a human benchmark.
• The 0.6B model sometimes fills all three subject slots and may add a merely related third subject. The next data revision explicitly penalizes quota filling and rebalances subject cardinality.
• Audience, purpose, and temporal orientation can require organizational context absent from the document.
• Document text is untrusted input. The CLI delimits it and instructs the model not to follow embedded directions, but small language models are not a security boundary.

Do not use COLLIE as the sole basis for legal, employment, access-control, or security decisions.