QED-Base-v1

QED-Base-v1-300M

Model Description

QED-Base-v1-300M is a decoder-only Transformer language model trained entirely from scratch.

Unlike fine-tuned or continually pretrained models, QED-Base-v1 was not initialized from or adapted from any existing pretrained model. All model weights were learned through original autoregressive pretraining using an independently created dataset.

The model serves as the foundation for future instruction-tuned and task-specific QED models while remaining suitable for language modeling research and experimentation.

---

Model Details

---

Architecture

QED-Base-v1 retains a modern decoder-only Transformer architecture featuring:

• Rotary Positional Embeddings (RoPE)
• RMSNorm
• SwiGLU feed-forward networks
• Multi-head causal self-attention
• Weight-tied input and output embeddings

The architecture is designed as a general-purpose language foundation model suitable for pretraining, research, and downstream fine-tuning.

---

Training Data

QED-Base-v1 was pretrained using a fully original dataset created specifically for this project.

The training corpus was independently authored and prepared for large-scale autoregressive language modeling.

The data preparation process included:

• Original content creation
• Custom formatting
• Custom preprocessing
• SentencePiece tokenization
• Dataset preparation for autoregressive pretraining

No existing pretrained model weights were used during training.

---

Intended Use

QED-Base-v1 is intended for:

• Language model research
• Transformer architecture experimentation
• Fine-tuning into instruction-following models
• Domain adaptation
• Educational projects
• Creating derivative language models

This is a foundation model and has not been instruction-tuned.

---

Limitations

As a pretrained foundation model:

• It is not optimized for instruction following.
• It may generate incomplete or unexpected responses.
• It has not been aligned with human preferences.
• It may require instruction tuning for assistant-style applications.
• Outputs should be reviewed before use in production or high-risk scenarios.

---

License

MIT License.