fela-streaming-asr

Streaming ASR (FNO CTC), on device, constant memory

This model transcribes speech to text as it arrives, on a CPU, with memory that stays flat no matter how long the stream runs. It is built for on device and edge use: real time transcription on a phone, a Pi, or a laptop, where the audio can be unbounded in length and there is no cloud round trip. It is a streaming CTC recognizer on a Fourier Neural Operator acoustic backbone.

What goes in, what comes out

• Input: an audio waveform at 16 kHz, fed in chunks for streaming. The model turns it into a log mel spectrogram (80 bins, 25 ms window, 10 ms hop) and subsamples it.
• Output: text. The decoder is greedy CTC over a 257 unit vocabulary (256 SentencePiece byte pair units plus a blank), no language model.
• This lets a user transcribe live or recorded speech on the device, with the transcript scrolling as the audio plays.

Why we built it this way

The acoustic backbone is a Fourier Neural Operator (FNO) sequence mixer, which mixes information across time in the frequency domain at about N log N cost instead of the N squared cost of full attention. There is no attention and no key/value cache. In streaming mode each block carries a bounded context, so memory is constant regardless of stream length, and the streaming output matches the offline forward exactly. That is the property Whisper (30 second offline chunks) and wav2vec2 (a fixed context window) do not have. The model is small and CPU native, so it runs on the device and keeps audio private.

Performance

The differentiated property is flat memory over an unbounded stream (0.3 to 0.4 MB added RSS over a 2.6 minute continuous stream) and faster than real time CPU inference.

Accuracy

LibriSpeech, greedy decode, no language model, measured on the full test sets. Reference baselines were rerun by us (n=200) where noted.

On clean WER this model is behind the self supervised and large weakly supervised baselines (13.4 versus about 3 to 6). WER scales the right way with data and model size (25.4 at 100h/64M, then 16.4 at 460h/64M, then 13.4 at 960h/206M).

The differentiated angle is streaming, flat memory, and CPU native serving, plus bonuses for any degraded and noisy audio enviroments where the dense transformer baselines break.

How to run it

It runs on CPU; no GPU is required. See quickstart/ for a runnable example. The given code uses the self contained modeling.py and config.json in this repo.

For an interactive playground, see the Hugging Face Space linked in this repo. For production serving, use the CPU native FELA server (https://github.com/Lowdown-Labs/fela_server).

Loading, exports, and verification

• Weights ship as model.safetensors (no pickle code execution risk). config.json holds the architecture hyperparameters; modeling.py is a self contained loader with the exact preprocessing (resample to 16 kHz, downmix to mono, log mel front end) and input validation that fails clearly on the wrong sample rate or unnormalized audio.

Formats

• fp32: reference and CPU.
• int8: on device deployment format. 3.3x smaller on the 64.4M model (258 MB to 78 MB).
• bf16: server and GPU inference only; most commodity ARM and microcontroller CPUs lack native bf16, so it is not the on device format.

Training data

• LibriSpeech (Panayotov et al. 2015): read English audiobooks. The streaming model was trained on train clean 100 plus train clean 360 (460 hours); the 206M byte CTC model on the full 960 hours (clean 100, clean 360, other 500). License: CC BY 4.0.
• Common Voice (Ardila et al. 2020): crowd sourced multilingual read speech. License: CC0.
• Multilingual LibriSpeech (Pratap et al. 2020): read speech from LibriVox audiobooks in multiple languages. License: CC BY 4.0.
• Targets are SentencePiece byte pair units (256 units plus a CTC blank). Training used SpecAugment. No language model is used at decode.

Training data splits and licensing

Datasets (for the two shipped/measured checkpoints, both LibriSpeech only):

• LibriSpeech (Panayotov, Chen, Povey, Khudanpur. "Librispeech: An ASR corpus based on public domain audio books." ICASSP 2015). Source: https://www.openslr.org/12/ (loaded via torchaudio.datasets.LIBRISPEECH). License: CC-BY-4.0 (verified from the LICENSE.TXT shipped in the corpus: "licensed under a Creative Commons Attribution 4.0 International License").
  • Common Voice (Ardila et al. 2020). License: CC0-1.0 (public domain dedication).
• Multilingual LibriSpeech (Pratap et al. 2020). License: CC-BY-4.0.

Split method and sizes:

• Evaluation is on the OFFICIAL LibriSpeech test sets, not a resplit: test clean = 2620 utterances, test other = 2939 utterances. Greedy CTC decode, no external LM. Full sets.
• Training subsets are the official LibriSpeech train folds concatenated (no held out validation carved from test). Audited WER: 206M byte CTC test clean 13.4 / test other 26.2.
• Split defined in train.py line 46 (official test clean) and line 47 (test other), loaded via torchaudio.datasets.LIBRISPEECH. Bare size assertions assert n_test == 2620 and assert n_other == 2939 are in the --smoke block at train.py lines 51 - 52.

Reproduced training script: /workspace/train_asr.py on the training pod produced the streaming 64.4M checkpoint (FNO, bpe256, 460 h). train.py in this repo adapts it faithfully; python train.py --smoke builds the split, asserts the official test sizes, and exits before training.

Intended use, limitations, and safety

What it is for: on device and edge streaming transcription where flat memory and faster than real time CPU inference matter more than the last few WER points: live captions, on device dictation, an offline transcription box. It runs on CPU; a GPU is optional and only helps speed.

What it is not for: it is not yet a drop in replacement for cloud Whisper on raw clean accuracy. Do not use the transcript as the sole record for a decision that carries legal, medical, or safety weight without human review, especially on noisy or accented audio where WER is higher.

Evaluated conditions and known failure modes:

• Trained on read English (LibriSpeech) for the measured baseline; expect higher WER on noisy, conversational, or accented audio for the current checkpoint.
• A 4 gram word language model gave no net gain at this acoustic quality (25.4 to 25.8 on the 100h model), so none is used.
• This on device build uses the FNO native mixer, which runs fully on CPU for streaming. A separate recall strong variant exists but is not part of this CPU build.
• Streams far longer than the roughly 16 second training clip regime use VAD style state resets (about 12 second windows) to avoid global context drift. Memory stays bounded regardless; the reset only avoids out of distribution drift on very long inputs.

Privacy: the model runs on the device or on your own CPU, so the audio does not have to leave the machine and there is no per call API.

Devices: the size and the streaming/RTF numbers were measured on an x86 server CPU. Given the RAM usage and capability on lower compute devices, this model would probably be a great fit for cellular usage, especially if trained further to lower WER.

How to cite

@misc{lowdownlabs_asr_streaming,
  title  = {Streaming ASR: an FNO CTC on device, constant memory speech recognizer},
  author = {Lowdown Labs},
  year   = {2026},
  note   = {Model card}
}

You should also cite the datasets, the reference baselines, and the libraries:

• LibriSpeech: Panayotov, Chen, Povey, Khudanpur. "Librispeech: An ASR corpus based on public domain audio books." ICASSP 2015.
• Common Voice: Ardila et al. "Common Voice: A Massively-Multilingual Speech Corpus." LREC 2020.
• Multilingual LibriSpeech: Pratap, Xu, Sriram, Synnaeve, Collobert. "MLS: A Large-Scale Multilingual Dataset for Speech Research." Interspeech 2020.
• wav2vec2: Baevski, Zhou, Mohamed, Auli. "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations." NeurIPS 2020.
• Whisper: Radford et al. "Robust Speech Recognition via Large-Scale Weak Supervision." 2022.
• Fourier Neural Operator: Li et al. "Fourier Neural Operator for Parametric Partial Differential Equations." ICLR 2021.
• PyTorch: Paszke et al. NeurIPS 2019.

Datasets

LibriSpeech (the two shipped/measured checkpoints 460 h and 960 h - are LibriSpeech-only). Panayotov, V., Chen, G., Povey, D., Khudanpur, S. (2015). "Librispeech: An ASR corpus based on public domain audio books." ICASSP 2015, pp. 5206-5210. DOI: https://doi.org/10.1109/ICASSP.2015.7178964. Source: https://www.openslr.org/12/ (loaded via torchaudio.datasets.LIBRISPEECH). License: CC-BY-4.0 - verified from the LICENSE.TXT shipped in the corpus (https://www.openslr.org/resources/12/LICENSE.TXT, "licensed under a Creative Commons Attribution 4.0 International License"). Commercial use: permitted with attribution.

@inproceedings{panayotov2015librispeech,
  title     = {Librispeech: An {ASR} corpus based on public domain audio books},
  author    = {Panayotov, Vassil and Chen, Guoguo and Povey, Daniel and Khudanpur, Sanjeev},
  booktitle = {2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages     = {5206--5210},
  year      = {2015},
  doi       = {10.1109/ICASSP.2015.7178964}
}

Common Voice Ardila, R., Branson, M., Davis, K., et al. (2020). "Common Voice: A Massively-Multilingual Speech Corpus." LREC 2020, pp. 4218-4222. arXiv:1912.06670 (https://arxiv.org/abs/1912.06670). Dataset: https://commonvoice.mozilla.org/en/datasets. License: CC0-1.0 (public domain dedication) - Mozilla releases Common Voice under CC0 (https://creativecommons.org/publicdomain/zero/1.0/). Commercial use: unrestricted.

Multilingual LibriSpeech / MLS Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., Collobert, R. (2020). "MLS: A Large-Scale Multilingual Dataset for Speech Research." Interspeech 2020. arXiv:2012.03411 (https://arxiv.org/abs/2012.03411). Source: https://www.openslr.org/94/. License: CC-BY-4.0 (https://creativecommons.org/licenses/by/4.0/). Commercial use: permitted with attribution.

Comparison baselines (not training data): wav2vec2 (Baevski et al., NeurIPS 2020, arXiv:2006.11477) and Whisper (Radford et al. 2022, arXiv:2212.04356).

Methods and code

The shipped on device streaming model is a pure Fourier Neural Operator sequence mixer (layer_pattern="FNO", gla_delta=false, no GLA recall layers - the build log records pattern=FNO, gla_layers=[]), a CTC head over a 257 unit SentencePiece byte pair vocabulary, and a log mel front end.

• PyTorch (training and inference framework). Paszke, A., et al. (2019). "PyTorch: An Imperative Style, High-Performance Deep Learning Library." NeurIPS 32. https://papers.nips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.
• Fourier Neural Operator (the acoustic sequence mixer). Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., Anandkumar, A. (2021). "Fourier Neural Operator for Parametric Partial Differential Equations." ICLR. arXiv:2010.08895 (https://arxiv.org/abs/2010.08895).
• Gated Linear Attention (evaluated in a development SSSL recall variant - NOT part of the shipped streaming checkpoint, which is pure FNO). Yang, S., Wang, B., Shen, Y., Panda, R., Kim, Y. (2023). "Gated Linear Attention Transformers with Hardware-Efficient Training." arXiv:2312.06635 (https://arxiv.org/abs/2312.06635).
• Connectionist Temporal Classification (CTC) (the sequence-to-label training loss and the greedy decode). Graves, A., Fernandez, S., Gomez, F., Schmidhuber, J. (2006). "Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks." ICML 2006, pp. 369-376. DOI: https://doi.org/10.1145/1143844.1143891.
• torchaudio - the log-mel front end and 16 kHz resampling (see modeling.py preprocess_audio). https://github.com/pytorch/audio (BSD-2-Clause).
• SentencePiece - the byte-pair tokenizer (bpe256.model, 256 units + 1 CTC blank). Kudo, T., Richardson, J. (2018). "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing." EMNLP 2018 (demo). arXiv:1808.06226 (https://arxiv.org/abs/1808.06226); repository https://github.com/google/sentencepiece (Apache-2.0).
• ONNX Runtime - the CPU/edge runtime for the exported graph (scripts/export_onnx.py, opset 17). https://github.com/microsoft/onnxruntime (MIT).

Model family

This is part of the FELA family from Lowdown Labs: one FNO architecture across many modalities, all CPU native and subquadratic. This repo is pushed as lowdown-labs/fela-streaming-asr.

These are grouped under the FELA Collection on Hugging Face. The models are independently trained per modality and do not share weights, so none carries a base_model link.

License

Released under the Lowdown Labs Lovely License 1.0 (CC BY-NC 4.0 + Hippocratic License 3.0). See LICENSE. For most LL models, a commercial license may be available; contact Lowdown Labs.