jsfakes-music-xlstm

JS Fakes Music xLSTM - An xLSTM model trained on Johann Sebastian Bach Style music

Say Hello on LinkedIn and X.

This is an xLSTM trained on music. The dataset that has been used is JS Fakes Garland 100K, which is based on a collection of musical samples extracted from the JS Fake Chorales dataset by Omar Peracha. The samples come in the prototypical Garland notation.

The dataset contains 100K samples and comes with a total token count of 80M.

The model size is 138.78K trainable parameters.

How to use

1. Clone this repository and follow the installation instructions: https://github.com/AI-Guru/helibrunna/
2. Open and run the notebook examples/music.ipynb.
3. Enjoy!

Training

Trained with Helibrunna by Dr. Tristan Behrens.

Configuration

training:
  model_name: jsfakes_garland_xlstm
  batch_size: 16
  lr: 0.001
  lr_warmup_steps: 312
  lr_decay_until_steps: 3125
  lr_decay_factor: 0.001
  weight_decay: 0.1
  amp_precision: bfloat16
  weight_precision: float32
  enable_mixed_precision: true
  num_epochs: 1
  output_dir: output/jsfakes_garland_xlstm
  save_every_step: 500
  log_every_step: 10
  wandb_project: jsfakes_garland_xlstm_2
  torch_compile: false
model:
  num_blocks: 4
  embedding_dim: 64
  mlstm_block:
    mlstm:
      num_heads: 4
  slstm_block:
    slstm:
      num_heads: 4
  slstm_at:
  - 2
  context_length: 2048
  vocab_size: 115
modelGPT:
  type: gpt2
  num_blocks: 4
  embedding_dim: 64
  decoder:
    num_heads: 4
  context_length: 2048
dataset:
  hugging_face_id: TristanBehrens/jsfakes_garland_2024-100K
tokenizer:
  type: whitespace
  fill_token: '[EOS]'