Downloads · 30 days
0
bekhruzsuleyman/sinegated-lm-1m
sinegated-lm-1m is a machine learning model from bekhruzsuleyman. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Attention-conditioned sinusoidal gating for small language models.
Downloads · 30 days
0
Access
Public
Updated Aug 28, 2026
Repo size
141 KB
Likes
1
Public
Click a slice to open those files.
.txt6.3 MB · 97%
From the Hugging Face model README
Attention-conditioned sinusoidal gating for small language models.
SinGatedLM is an experimental language-model architecture built around a simple idea:
Instead of using attention only as an additive/residual transformation, use the attention output as a multiplicative sinusoidal control signal for another learned transformation.
The core operation is:
f(x, y) = (Wx + b) \odot (\alpha \sin(y))
where:
x is the current representationWx + b is a learned linear transformationy is an attention-derived representationsin(y) provides nonlinear, oscillatory modulationα controls the gating amplitude and can be learnedThe experimental language model uses the following structure:
Token IDs
│
▼
Token Embedding
│
+
Positional Embedding
│
▼
Linear
│
▼
Multi-Head Attention ←── with residual
│
▼
Linear
│
▼
SinGated Attention ←── no residual (gate must act)
│
▼
Linear
│
▼
Vocabulary Head
│
▼
Logits
The central mechanism is:
A = Attention(x)
g = \alpha \sin(A)
h = Wx + b
SinGated(x,A) = h \odot g
In other words, attention is used as a dynamic modulation signal rather than simply being added to the representation.
The core operation is implemented as:
class SinGatedLinear(nn.Module):
"""f(x, y) = (W @ x) * alpha * sin(y)"""
def forward(self, x, y):
Wx = self.linear(x)
gate = self.alpha * torch.sin(y)
return Wx * gate
The attention-conditioned block is:
class SinGatedAttention(nn.Module):
"""
A = Attention(x)
out = SinGatedLinear(x, A)
= (W @ x) * alpha * sin(A)
"""
The attention operation itself remains standard multi-head self-attention. The experimental change is what happens after attention produces its representation.
Experiments were performed on the Tiny Shakespeare character-level language-modeling task.
The purpose of these experiments is not to claim that the architecture is universally superior, but to investigate whether sinusoidal attention-conditioned modulation provides useful behavior under small parameter budgets.
The first experiment compares SinGatedLM against a parameter-matched PlainLM baseline.
Both models contain essentially the same number of parameters and use the same overall computational structure, with the SinGated mechanism replaced by standard learned transformations in the baseline.
| Model | Parameters | Final validation loss |
|---|---|---|
| PlainLM | 64,659 | 2.6931 |
| SinGatedLM | 64,660 | 2.5603 |
Difference:
2.6931 - 2.5603 = 0.1328
SinGatedLM achieved a 0.1328 lower validation loss while using exactly one additional parameter.
The advantage appeared throughout training rather than only at the final checkpoint.
| Iteration | SinGatedLM | PlainLM |
|---|---|---|
| 800 | 2.7418 | 2.8728 |
| 1300 | 2.6677 | 2.7664 |
| 1600 | 2.6673 | 2.7819 |
| 2000 | 2.6180 | 2.7158 |
| 2400 | 2.6016 | 2.7247 |
| 2999 | 2.5603 | 2.6931 |
This suggests that the observed difference is not limited to a single final checkpoint and may involve differences in optimization behavior.
SinGatedLM 21.7s
PlainLM 22.0s
The measured training times were also very similar in this run.
This experiment scales both models to approximately one million parameters with identical hidden dimensions (d_model = 292).
The parameter counts are effectively identical:
SinGatedLM 1,027,048 parameters
PlainLM 1,027,047 parameters
Difference 1 parameter
The only architectural difference is that SinGatedLM replaces the PlainLM second block (MHA + residual + Linear) with SinGatedAttention (attention-driven sinusoidal gating, no residual around the gate).
| Model | Parameters | Final train loss | Final validation loss | Time |
|---|---|---|---|---|
| SinGatedLM | 1,027,048 | 1.5151 | 1.7174 | 62.8s |
| PlainLM | 1,027,047 | 1.6763 | 1.8616 | 60.5s |
Validation-loss difference:
1.8616 - 1.7174 = 0.1442
Perplexity:
SinGatedLM exp(1.7174) ≈ 5.57
PlainLM exp(1.8616) ≈ 6.43
Both models were trained to convergence to observe their asymptotic behavior.
| Model | Parameters | Final train loss | Final validation loss | Time |
|---|---|---|---|---|
| SinGatedLM | 1,027,048 | 1.3863 | 1.5902 | 164s |
| PlainLM | 1,027,047 | 1.5421 | 1.7592 | 159s |
Validation-loss difference:
1.7592 - 1.5902 = 0.1690
Perplexity:
SinGatedLM exp(1.5902) ≈ 4.90
PlainLM exp(1.7592) ≈ 5.81
| Iteration | SinGatedLM Val | PlainLM Val | Gap (PL − SG) |
|---|---|---|---|
| 500 | 2.3239 | 2.3334 | −0.0095 |
| 1,000 | 2.0104 | 2.1320 | +0.1216 |
| 2,000 | 1.8057 | 1.9725 | +0.1668 |
| 3,000 | 1.7210 | 1.9033 | +0.1823 |
| 4,000 | 1.6853 | 1.8664 | +0.1811 |
| 5,000 | 1.6380 | 1.8224 | +0.1844 |
| 6,000 | 1.6167 | 1.8034 | +0.1867 |
| 7,000 | 1.5955 | 1.7670 | +0.1715 |
| 8,000 | 1.5902 | 1.7592 | +0.1690 |
Observations:
SinGatedLM:
Be prisons are love of the for our way,
The subtlend shallows, for half unto an enemy.
I as a little fight in my dutinner.
OsLIVIA. No dear a mountakes of my fear therer horse.
Fear. What live you you are my life, but when his prince.
FOR. Thou love. I have me we are man. Is it once
PlainLM:
Enter CAIUSTE SERVANT. Exeunt Hame, not ekent,
By stone, for hall when and to the tell as of any friendst,
And to guilmes that when for a kill.
We were will get begles; with hall's in cunnot,
They mince. With them and littles!
SinGatedLM produces more recognizable character names, dialogue structure, and grammatical fragments at the same parameter count.
An earlier experiment compared a ~1M SinGatedLM against a larger ~1.5M PlainLM. This is not parameter-matched and is included only for completeness.
| Model | Parameters | Final train loss | Final validation loss | Time |
|---|---|---|---|---|
| SinGatedLM | 1,027,048 | 1.6663 | 1.8818 | 58.3s |
| PlainLM | 1,535,483 | 1.7800 | 1.9914 | 68.3s |
SinGatedLM used approximately 33% fewer parameters while achieving a lower validation loss. This result should be interpreted as efficiency, not as proof of superiority at equal scale.
Validation Loss
~64K parameters (matched)
PlainLM 2.6931
SinGatedLM 2.5603
↓ 0.1328
~1M parameters (matched, 3,000 steps)
PlainLM 1.8616
SinGatedLM 1.7174
↓ 0.1442
~1M parameters (matched, 8,000 steps, convergence)
PlainLM 1.7592
SinGatedLM 1.5902
↓ 0.1690
Perplexity: 5.81 → 4.90 (1.18× better)
The parameter-matched experiments are the most informative:

The motivation is to explore neural transformations involving multiple interacting variables rather than repeatedly applying only:
y = Wx + b
SinGatedLM instead uses:
y = f(x, A)
where A is produced by attention.
Specifically:
A = Attention(x)
and:
f(x,A) = (Wx+b) \odot \alpha\sin(A)
This gives attention a second role:
Traditional use:
x → Attention(x) → representation
│
▼
output
SinGated use:
x ────────────────→ Linear(x) ──────┐
│
Attention(x) → sin(·) → gate ──────┤
▼
×
│
▼
output
The hypothesis is that this multiplicative nonlinear interaction can provide useful representational behavior that is different from simply adding attention outputs to the network's representation. The sinusoid is not merely a gate — its periodicity creates multiple active regions, effectively giving the model a soft routing mechanism without explicit mixture-of-experts parameters.
The project follows:
Hypothesis
↓
Implementation
↓
Controlled experiment
↓
Measurement
↓
Scaling
↓
Validation
The goal is to test the mechanism experimentally rather than assume that a mathematically unusual operation is automatically better.
α (fixed vs learnable)tanh, σ, identity)The current experiments use:
AdamW8000 (convergence runs)3212843e-4100The experiments use a fixed random seed for reproducibility.
Note that the exact training results can vary depending on hardware, PyTorch/CUDA versions, random-number-generator state, and other implementation details.
🚧 Experimental / Research
SinGatedLM is an experimental architecture.
The current results are encouraging:
However, these results are limited to a single dataset (Tiny Shakespeare), a single random seed, and a shallow architecture. They are not sufficient to establish general superiority over Transformer architectures.
The most important next steps are multiple-seed validation, alternative gating function ablations (sin vs tanh vs σ), and broader-dataset evaluation.
MIT License
Copyright (c) 2026 Bekhruz Suleyman
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Bekhruz Suleyman