Wiola13M

Wiola

Wiola is a novel decoder-only Small Language Model (SLM) developed by OSCOWL-AI. It introduces several architectural improvements over conventional Transformer decoder models, focusing on improving contextual reasoning, computational efficiency, and parameter utilization while remaining compatible with the Hugging Face ecosystem.

Note

Wiola uses a custom architecture. Before loading the model, install the Wiola package:

pip install git+https://github.com/Wiola-OSCOWL-ai/Wiola13M.git

or (once available)

pip install wiola

---

Model Overview

Wiola is designed as a research-focused decoder-only language model that explores new methods for improving attention, positional encoding, and feed-forward computation.

The architecture introduces five core innovations:

• Spiral Rotary Positional Encoding (SRPE) for enhanced positional representation.
• Gated Cross-Layer Attention (GCLA) for improved information flow across decoder layers.
• Adaptive Token Merging (ATM) for reducing redundant token computation during training.
• Dual-Stream Feed Forward Network (DSFF) for richer feature learning.
• WiolaRMSNorm, a lightweight normalization technique designed specifically for Wiola.

These components are integrated into a standard autoregressive language modeling framework and are compatible with Hugging Face Transformers.

---

Installation

Install directly from GitHub:

pip install git+https://github.com/Wiola-OSCOWL-ai/Wiola13M.git

or after the PyPI release:

pip install wiola

---

Quick Start

Create a new Python file (for example, test.py) and paste the following code into it:

from wiola13m import WiolaForCausalLM
from transformers import AutoTokenizer

model_path = "oscowlai/Wiola13M"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = WiolaForCausalLM.from_pretrained(model_path)

prompt = "Once upon a time"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    return_token_type_ids=False,
)

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_p=0.95
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Run the script from your terminal:

python test.py

The first time you run the script, the model and tokenizer will be downloaded from Hugging Face and stored in the local cache. This may take a few moments depending on your internet connection. Subsequent runs will use the cached files and start much faster.

Using your own prompts

To generate text for a different prompt, replace:

prompt = "Once upon a time"

with any text you would like the model to continue.

For example:

prompt = "Artificial Intelligence will"

prompt = "Write a short story about a robot."

prompt = "Explain gravity in simple terms."

Example output

Running the script prints the generated text to your terminal.

Example:

Once upon a time, a simple idea turned into a powerful innovation. It all started with coding ...

Since text generation uses sampling (do_sample=True), the generated output will be different each time you run the script.

---

Inputs

The model accepts tokenized text.

Input shape:

(batch_size, sequence_length)

Example:

inputs = tokenizer(
    "Hello, how are you?",
    return_tensors="pt"
)

---

Outputs

The model returns:

• Hidden representations
• Next-token logits
• Optional cached key/value states for autoregressive generation

For text generation, use:

outputs = model.generate(...)

The generated token IDs can be converted back into readable text using:

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Architecture

Wiola is a decoder-only Transformer architecture composed of:

• Token Embedding Layer
• Multiple Wiola Decoder Layers
• Final WiolaRMSNorm
• Language Modeling Head

Each decoder layer consists of:

• Gated Cross-Layer Attention
• Spiral Rotary Positional Encoding
• Adaptive Token Merging (training only)
• Dual-Stream Feed Forward Network
• Residual Connections
• WiolaRMSNorm

---

Model Characteristics

Initialization

The released checkpoints are trained from scratch using the Wiola architecture.

Architecture Type

• Decoder-only Transformer
• Autoregressive Language Model

Framework

• PyTorch
• Hugging Face Transformers

---

Training Data

The architecture supports training on standard causal language modeling corpora such as:

• OpenWebText
• BooksCorpus
• FineWeb
• Wikipedia
• Common Crawl derived datasets

The released checkpoints may use different datasets depending on the model version.

Training data is cleaned using standard preprocessing:

• UTF-8 normalization
• Deduplication
• Document concatenation
• Tokenization

---

Evaluation

The model is designed to be evaluated using common language modeling benchmarks including:

• WikiText
• HellaSwag
• ARC
• PIQA
• MMLU
• GSM8K
• HumanEval (code models)

Benchmark results will be released alongside future checkpoints.

---

Hardware Requirements

Training:

• NVIDIA GPUs
• CUDA-enabled PyTorch

Inference:

CPU:

• Supported
• Suitable for small checkpoints

GPU:

• Recommended
• Significantly faster generation

---

Known Limitations

Wiola is a research model and has several limitations:

• May generate incorrect or fabricated information.
• Does not possess factual understanding.
• Performance depends heavily on training data.
• Not intended for safety-critical or medical decision making.
• Should always be used with human verification.

---

Intended Use

Suitable for:

• Language model research
• NLP experimentation
• Education
• Fine-tuning
• Benchmarking novel Transformer architectures

Not recommended for:

• Medical diagnosis
• Legal advice
• Financial decision making
• Autonomous high-risk systems

---

Ethical Considerations

Like other large language models, Wiola may inherit biases from training data.

Developers should:

• Review generated outputs.
• Validate factual information.
• Monitor for harmful or biased generations.
• Apply appropriate safety filters in downstream applications.

---

Novel architecture

                    Input Tokens
                         │
                         ▼
                 Token Embeddings
                         │
                         ▼
        ┌─────────────────────────────────────┐
        │      Spiral Rotary Encoding      │
        │   Gated Cross-Layer Attention    │
        │     Adaptive Token Merging       │
        │        Dual-Stream FFN           │
        │          WiolaRMSNorm            │
        └─────────────────────────────────────┘
                         │
                         ▼
                 Repeated N Layers
                         │
                         ▼
                   Final RMSNorm
                         │
                         ▼
                       LM Head
                         │
                         ▼
                 Generated Tokens

---

Citation

If you use Wiola in your research, please cite:

@software{wiola2026,
  title={Wiola: A Novel Decoder-Only Small Language Model},
  author={OSCOWL-AI},
  year={2026},
  url={https://github.com/Wiola-OSCOWL-ai/wiola}
}

---

License

Apache License 2.0

---

Links

• GitHub: https://github.com/Wiola-OSCOWL-ai/Wiola13M
• PyPI: https://pypi.org/project/wiola/ (coming soon)
• Hugging Face: https://huggingface.co/OSCOWL-AI

---

Acknowledgements

Wiola is an open research project developed by OSCOWL-AI to explore efficient and scalable Small Language Model architectures. Contributions, feedback, and research collaborations are welcome.