PoreBERT-DNA-VQI-007M-526

PoreBERT-DNA-VQI-256-526

A compact BERT-style foundation model for nanopore DNA sequencing signal representation learning.

PoreBERT-DNA-VQI-256-526 learns contextual representations from discretized nanopore electrical signal tokens generated by a Vector Quantization (VQ) tokenizer.

The model is pretrained with a Masked Language Modeling (MLM) objective on large-scale nanopore DNA sequencing token sequences.

The model is designed for downstream nanopore sequencing applications including:

• Basecalling
• Modified base detection
• Signal representation learning
• Read-level embedding

---

Model Summary

---

Model Name Explanation

PoreBERT-DNA-VQI-256-526

PoreBERT

Nanopore sequencing foundation model based on the BERT encoder architecture.

DNA

The model is trained on nanopore DNA sequencing electrical signal representations.

VQI

Vector Quantization based minimal vocabulary tokenizer.

The raw nanopore electrical signal is converted into discrete token IDs using a neural codec based on vector quantization.

256

The hidden representation dimension of the Transformer encoder.

hidden_size = 256

526

Internal development release identifier.

---

Architecture Overview

The complete representation pipeline:

Nanopore Electrical Signal

  |
  v

PoreCodec
(CNN Encoder + Vector Quantization)

  |
  v

Discrete Signal Tokens

  |
  v

PoreBERT-DNA-VQI-256-526

  |
  v

Contextual Token Embeddings

  |
  v

Downstream Applications

The PoreBERT model itself does not directly process raw electrical signals.

The input of PoreBERT is the discrete token sequence generated by the PoreCodec tokenizer.

---

Tokenizer

This model uses the following tokenizer:

ShuaiAnwo/pore-codec-rsq742c12a-511

The tokenizer converts nanopore electrical signals into discrete token IDs through a VQ-based neural codec.

Workflow:

Raw signal

| v

PoreCodec VQ tokenizer

| v

Discrete Token IDs

| v

PoreBERT Encoder

| v

Contextual Embeddings

---

Model Architecture Details

Configuration:

Architecture:

Token Embedding
|
Position Embedding
|
LayerNorm + Dropout
|
8 Transformer Encoder Layers
|
LayerNorm
|
Contextual Token Representation

---

Pretraining Objective

The model is pretrained using Masked Language Modeling (MLM).

During training:

1. Nanopore electrical signals are converted into discrete tokens.
2. Random tokens are masked.
3. The Transformer predicts the original tokens using bidirectional context.

Example:

Input:

A B [MASK] D E

Prediction:

C

Training configuration:

---

Learning Rate Schedule

The model uses:

cosine_with_restarts

Configuration:

The learning rate schedule consists of a warmup phase followed by cosine decay with restart cycles.

---

Usage

PoreBERT-DNA-VQI-256-526 operates on discrete signal tokens generated by the corresponding PoreCodec VQ tokenizer.

Pipeline:

Nanopore Raw Signal

|
v

PoreCodec VQ Tokenizer

|
v

Discrete Token IDs

|
v

PoreBERT Encoder

|
v

Contextual Signal Embeddings

Quick Start

import numpy as np
import torch

from transformers import AutoFeatureExtractor, AutoModel


codec_name = "ShuaiAnwo/pore-codec-rsq742c12a-511"
bert_name = "ShuaiAnwo/PoreBERT-DNA-VQI-256-526"


codec = AutoModel.from_pretrained(
    codec_name,
    trust_remote_code=True,
).eval()


feature_extractor = AutoFeatureExtractor.from_pretrained(
    codec_name,
    trust_remote_code=True,
)


bert = AutoModel.from_pretrained(
    bert_name,
).eval()



raw_signal = np.random.normal(
    70,
    8,
    1855,
).astype(np.float32)



with torch.no_grad():

    # Raw signal -> VQ token IDs

    signal = feature_extractor(
        raw_signal,
        return_tensors="pt",
    )["signal"]


    token_ids = codec.encode_signal(
        signal,
        layer=2,
    )


    # Token IDs -> contextual embeddings

    outputs = bert(
        input_ids=token_ids,
    )


    embeddings = outputs.last_hidden_state



print(
    "Embedding shape:",
    embeddings.shape
)

Output:

Embedding shape:

(batch_size, sequence_length, 256)

The generated embeddings can be used for downstream nanopore sequencing tasks:

• Basecalling
• Modified base detection
• Signal representation learning
• Read-level embedding
• Sequence classification

---

Training Data

The model was pretrained on nanopore DNA sequencing token sequences.

Dataset:

ShuaiAnwo/PoreDNA_S1_HG002_MOD_250F701901011_A50

Training pipeline:

Nanopore Electrical Signal

        |
        v

PoreCodec VQ Tokenizer

        |
        v

Discrete Token Sequence

        |
        v

Masked Language Modeling

        |
        v

PoreBERT Encoder

---

Training Configuration

Training configuration from the original experiment:

model:
  bert_model_type: electra
  vocab_size: 2560
  hidden_size: 256
  num_hidden_layers: 8
  num_attention_heads: 8
  intermediate_size: 1024
  max_position_embeddings: 1536


mlm_config:
  mlm_probability: 0.15


optimizer:
  name: adamw
  learning_rate: 8.0e-4
  weight_decay: 0.01
  beta1: 0.9
  beta2: 0.98


scheduler:
  name: cosine_with_restarts
  num_cycles: 2
  t_warmup: 2000


precision:
  amp_bf16

---

Gradient Checkpointing

The training framework supports gradient checkpointing.

Gradient checkpointing reduces GPU memory usage by recomputing intermediate activations during backward propagation.

Benefits:

• Lower GPU memory consumption
• Longer sequence training
• Larger models on limited hardware

Trade-off:

• Increased training computation time

---

Limitations

• The model does not directly accept raw nanopore electrical signals.
• Raw signals must first be converted into VQ token IDs.
• Performance depends on tokenizer quality and training data distribution.
• The model is optimized for nanopore DNA sequencing representation learning.

---

Citation

Coming soon.

---

License

Please refer to the LICENSE file for usage conditions.