MambaBitPet-NoRes-NoNorm-TinyShakespeare

MambaBit. Bit-level cursed model with vocab size=2

• 4 layers, vocab size=2, embedded size = 4096 float32 parm per bit.

• Training was done on first 8030848 bits of tiny Shakespeare in 10 hours on laptop with 16GB VRAM on 9 batches of 128*8 bit each. Training code included in trainer.ipynb

• To run the model run python mambabit.py "As sun raised over". Expected output

As sun raised over me.

LEONTES:
Now means means me not so much as my father,
In the good many lord, and my father come.

KING RICHARD III:
What is my father come and my father,
In the good lord, and my father come and before his father.

GLOUCESTER:
Now the goes of men, a

• Bytes are encoded with most significant bit fed first. Eg '7' = [0, 0, 1, 1, 0, 1, 1, 1], so MSB 0 is being fed first rather than last as if it was with [1, 1, 1, 0, 1, 1, 0, 0]. Intuition with that is that bits at the beginning change less frequent than in the end, so model will be like "I think I will produce a digit" then "I think I will produce 7" instead of "so I spat something. Should it be a number? a letter? dunno"

• I tried to use BF16 originally, but model went into nan (with default big LR) or gradients were so small weights didn't change(smaller LR). I switched back to F32, however some layers still initialize weight with factor x0.001 as I hoped it would stop model from going to nan.

---

Also next morning I realized that Mamba module doesn't come with built-in normalization and residual as I thought, which explains why BF16 attempts failed. Oops.