Qiushi-Engine-Frontier-Advancement

Qiushi-Engine-Frontier-Advancement

Full research project on GitHub · Oxelra AI · Questions & feedback

Explore the English and Chinese reports, methods, training and evaluation code, controlled experiments, results and research notes in the GitHub project. We welcome questions, independent replication and discussion of the findings.

Stage I model in Qiushi Engine's BabyLM Strict-Small research program.

Frontier Advancement -> Principle Discovery -> Principle-Guided Frontier Advancement.

This English masked language model pairs source passages with concise rewrites, allowing more source passages within a fixed 10-million-word corpus. It uses a DeBERTa-v2 encoder with residual adapters: small trainable layers added to the encoder. A second adapter is added after pretraining and trained during a short continuation.

The two releases connect three research stages: training a limited-data language model, studying how it learns from paired texts and retains learned behavior, and using those findings to develop a continuation method tested against matched controls.

Companion model: Qiushi-Engine-Principle-Guided-Frontier-Advancement.

Model

Use

from transformers import AutoModelForMaskedLM, AutoModel, AutoTokenizer

repo = "leslie721007/Qiushi-Engine-Frontier-Advancement"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)
encoder = AutoModel.from_pretrained(repo, trust_remote_code=True)

The custom code is required: both adapter paths are part of the model. The encoder entry point supports downstream fine-tuning. The included requirements.txt records the tested model-loading environment. For a fixed experiment, use the same Hub commit for tokenizer, model and encoder.

Full Evaluation

Local results from the full BabyLM evaluation. SuperGLUE uses the custom encoder with both adapters and fine-tuning seed 42.

Overall is the arithmetic mean of the nine components. Age of Acquisition (AoA) was evaluated across 18 checkpoints and scored 0.

Research Comparison

Both continuation runs start from the same Stage I model. The final method improves Overall over ordinary continuation in each run; individual task scores vary. METHOD.md explains the training changes and what the comparisons measure.

Artifacts

• EVALUATION.json: exact local component scores and model identity.
• METHOD.md, TRAINING.md, DATA.md: design, recorded training and data provenance.
• CHECKPOINTS.json: shared intermediate checkpoints and actual exposure counts.
• evaluation/: complete final prediction files, AoA measurements and Fast metrics.
• submission/: full and full-plus-Fast official-format prediction bundles.
• VALIDATION.json: package loading and save/reload tests.

Intended uses are masked-token prediction, sentence scoring and encoder fine-tuning. The experiments cover English language modeling under the BabyLM Strict-Small budget.

License

Model weights and original code: Apache-2.0. Data sources retain their original licenses; see DATA.md.