wiola-120m

Wiola-120M

<div align="center">

Wiola-120M

A Novel Small Language Model (SLM) introducing Spiral Rotary Positional Encoding (SRPE), Gated Cross-Layer Attention (GCLA), Adaptive Token Merging (ATM), Dual-Stream Feed-Forward (DSFF), and WiolaRMSNorm.

Developed by OSCOWL-AI

</div>

---

Overview

Wiola-120M is an experimental decoder-only Small Language Model developed from first principles as part of the Wiola research project.

Unlike conventional transformer architectures, Wiola introduces several novel architectural components designed to improve parameter efficiency, contextual reasoning, and computational efficiency while maintaining compatibility with the Hugging Face ecosystem.

This repository contains the trained model weights.

The implementation of the architecture is provided separately through the Wiola Python package.

---

Architecture

Wiola introduces five core innovations:

Spiral Rotary Positional Encoding (SRPE)

A rotary positional encoding variant that augments angular rotations with spiral radial modulation for richer positional representations.

---

Gated Cross-Layer Attention (GCLA)

Allows each decoder layer to selectively attend to compressed contextual representations from previous layers using learnable gates.

---

Adaptive Token Merging (ATM)

During training, semantically redundant neighbouring tokens are dynamically merged to reduce computation while preserving information.

ATM is automatically disabled during inference.

---

Dual-Stream Feed Forward Network (DSFF)

Each decoder block contains two parallel feed-forward streams:

• Narrow stream
• Wide stream

whose outputs are fused to improve representation learning.

---

WiolaRMSNorm

A lightweight RMSNorm implementation used throughout the architecture.

---

Model Specifications

---

Training

This checkpoint was trained using the official Wiola training pipeline.

Dataset

• WikiText-103

Optimizer

• AdamW

Learning Rate Scheduler

• Cosine Decay
• Linear Warmup

Mixed Precision

• BF16 / FP16 supported

---

Evaluation

Final validation metrics obtained during training:

These values correspond to the released checkpoint and should be considered baseline research results.

---

Installation

Install the Wiola framework first.

pip install wiola

or install the latest development version:

pip install git+https://github.com/Wiola-OSCOWL-ai/wiola.git

---

Usage

from transformers import AutoTokenizer
from wiola import WiolaForCausalLM

repo = "oscowlai/wiola-120m"

tokenizer = AutoTokenizer.from_pretrained(repo)

model = WiolaForCausalLM.from_pretrained(repo)

prompt = "Artificial Intelligence is"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Repository

GitHub

https://github.com/Wiola-OSCOWL-ai/wiola

---

Intended Uses

Wiola is intended for:

• Language model research
• Architecture experimentation
• Fine-tuning research
• Educational purposes
• Small Language Model benchmarking

---

Limitations

This checkpoint is released primarily as a research model.

Current limitations include:

• Trained on a relatively small corpus compared to production LLMs.
• Not instruction tuned.
• No RLHF or preference optimization.
• May generate inaccurate or fabricated information.
• Should not be used for high-risk or safety-critical applications.

---

Citation

@software{wiola2026,
  title={WiolaSLM A Novel Small Language Model with Spiral Rotary Positional Encoding, Gated Cross-Layer Attention, Adaptive Token Merging and Dual-Stream Feed Forward Networks},
  author={OSCOWL-AI},
  year={2026},
  url={https://github.com/Wiola-OSCOWL-ai/wiola}
}

---

License

Apache License 2.0

---

Acknowledgements

Wiola is an independent research effort by OSCOWL-AI focused on advancing efficient Small Language Models through novel transformer architectures.

Contributions, issues, and feedback are welcome through the official GitHub repository.