scancode-required-phrases-deberta-bioes-crf-hardened

ScanCode Required Phrase Model

This model predicts candidate required phrases in ScanCode license rules.

Required phrases help prevent weak or partial rule matches from producing false positive license detections. Predictions must pass ScanCode validation and should be reviewed before they are added to rules.

Model

• Base model: microsoft/deberta-v3-large
• Base revision: 64a8c8eab3e352a784c658aef62be1662607476f
• Architecture: DeBERTa-v3-large with a word level CRF
• Labels: O, B-REQ, I-REQ, E-REQ, S-REQ
• Maximum input length: 512 tokens
• Parameters: 434,017,320
• Artifact revision: 11215925b0f9b64cfcfbbb5492b52d6aeb5a572b

Training data

The model was trained on required phrases extracted from ScanCode license rules.

The dataset uses a hybrid split. Common license expressions may appear in more than one split through different rule identifiers. Rare expressions remain in one split. This is not a fully unseen license evaluation.

The model was trained for eight epochs with FP32 precision, gradient accumulation, 8-bit AdamW, a fixed seed of 42, and best checkpoint selection using validation strict span F1.

Results

The test split was evaluated once after checkpoint selection and must not be used for further tuning.

Usage

Install the required phrases package from the reviewed commit:

python -m pip install \
  "scancode-required-phrases[inference] @ git+[https://github.com/aboutcode-org/scancode-required-phrases.git@d6456afe13407ca20b21d7957903f65c007393d4](https://github.com/aboutcode-org/scancode-required-phrases.git@d6456afe13407ca20b21d7957903f65c007393d4)"

Run read only prediction:

add-model-required-phrases --rule path/to/example.RULE --predict-only

Run interactive review:

add-model-required-phrases --rule path/to/example.RULE

After the package is published, the installation command will become:

python -m pip install "scancode-required-phrases[inference]"

Safety and limitations

• A wrong required phrase can prevent a valid license match.
• Model scores are not calibrated probabilities.
• Long rules may be truncated.
• The model may predict URLs or filenames that ScanCode blocks.
• Results depend on the ScanCode rules used for training.
• Predictions must pass ScanCode validation.
• Human review is recommended before applying changes.
• The model does not modify rules by itself.

Reproducibility

The repository includes the full weights, tokenizer, training configuration, artifact hashes, reproducibility manifest, and publication marker.

See run_manifest.json and SUCCESS.json for the exact configuration and file hashes.

License

The fine tuned model weights and model specific files are released under the Apache License 2.0.

The base microsoft/deberta-v3-large model is MIT licensed and remains subject to its original license and attribution.