structure-completeness-distilbert-zh

structure-completeness-distilbert-zh

License: PolyForm Noncommercial 1.0.0 — personal learning / research / education / non-profit use only, no commercial use. This is a "source-available" model, not an OSI-approved open-source license (the restriction on commercial use is the reason). See the license link for the full text before using this model in any product or paid service.

What this model does

5-class classifier that scores the structural completeness of a Chinese interview answer (does it follow a clear situation/action/result-style structure vs. being disorganized rambling), as one dimension of an AI mock interview coaching app's answer-scoring pipeline. It is not a general Chinese text classifier and was not trained for any other task.

Output is one of 5 ordinal bands, each corresponding to a 0-10 structural completeness score range:

Training data

• 150 Chinese interview answers, each with a structure_completeness score (0-10) from a single human reviewer (no second independent rater — see "Known limitations" below).
• Answers span 3 question types (behavioral / technical / case analysis), 50 each, balanced.
• Split: 5-fold stratified cross-validation over a 128-sample pool (stratified jointly on question type x score band), plus a held-out test set of 22 samples (14.7%) that never participated in any training, validation, or early-stopping decision.
• The final checkpoint published here was trained on all 128 CV-pool samples combined (no validation split left for early stopping), for a fixed 14 epochs (no back-translation data augmentation — augmentation was tried and did not help this model, see below).

Final metrics

Held-out test set (n=22, one single predict() call, never touched during training):

For reference, the 5-fold cross-validation mean on the 128-sample CV pool (different data split, not directly comparable 1:1 to the 22-sample test set above) was macro_f1=0.885, qwk=0.950. The test-set macro_f1/exact_accuracy are noticeably lower than the CV mean, which is expected given n=22 is a small, high-variance sample; qwk and within1_accuracy stay close to the CV mean, and every miss on the test set was an adjacent-band miss (no error was off by 2+ bands).

Known limitations

• Test set is small (n=22). Point-estimate metrics on 22 samples have wide, unreported confidence intervals — don't treat these numbers as precise beyond +/-1 band roughly.
• Chinese only. All 150 training/eval samples are Chinese-language interview answers; there is no evaluation on English or code-switched input, and it will likely not behave sensibly on non-Chinese text.
• Single annotator. Every label came from one human reviewer with no second independent rater, so no inter-annotator agreement metric (e.g. Cohen's kappa) exists for the labels themselves — some fraction of any model "error" against these labels may be label noise, not model error.
• Back-translation augmentation did not help this model. A back- translation augmented variant (same architecture) scored slightly worse on every metric (macro_f1 0.885->0.878, qwk 0.950->0.942, exact_accuracy 0.883->0.876, within1_accuracy 0.985->0.977, all 5-fold CV means) than the plain (non-augmented) version this checkpoint is. The published model is the non-augmented one.
• Small overall dataset (150 labeled examples total). This is a small-data fine-tune, not a model trained on a large labeled corpus — expect it to be sensitive to answer styles/topics that differ a lot from the training distribution (3 question types: behavioral, technical, case analysis).
• One dimension of a multi-dimension rubric. Structural completeness is one of several scoring dimensions (alongside keyword coverage, logical coherence, specificity) in the source project's full scoring rubric — this model only covers structure, not overall answer quality.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "a1a1a1aaa/structure-completeness-distilbert-zh"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

answer = "首先我发现了这个问题，然后评估了几个方案，最后选择了风险最低的一个并推动落地。"
inputs = tokenizer(answer, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    logits = model(**inputs).logits
band_id = int(torch.argmax(logits, dim=-1))

band_labels = ['0-2', '3-4', '5-6', '7-8', '9-10']
print(f"predicted band: {band_labels[band_id]}")

Or with pipeline:

from transformers import pipeline

clf = pipeline("text-classification", model="a1a1a1aaa/structure-completeness-distilbert-zh")
print(clf("首先我发现了这个问题，然后评估了几个方案，最后选择了风险最低的一个并推动落地。"))

Source project

Trained as part of a 10-week solo AI mock interview coaching app project. Full training/eval code, data prep, cross-validation splits, and the decision log documenting model selection reasoning are in the project repo: https://github.com/Daifanqi/ai-interview-coach (ml/train.py, ml/common.py, docs/week7_finetuning_results.md, docs/decision_log.md decisions #33-#34).