FunDLM-Medium-1024bp-Centromere

FunDLM-Medium-1024bp-Centromere

Model summary

Installation

Environment setup follows the same steps as PlantCAD2.

Install core dependencies

pip install torch==2.12.0 transformers==5.15.0 accelerate==1.14.0 tokenizers==0.22.2
safetensors==0.8.0 huggingface_hub==1.27.0 datasets==3.1.0 peft==0.13.2

Install Mamba/Caduceus-specific dependencies (required — this model uses a

Mamba-based state-space backbone, same as PlantCAD/PlantCAD2)

pip install mamba_ssm==2.3.2.post1 causal_conv1d==1.6.0 --no-build-isolation

FunDLM-Medium-1024bp-Centromere/ ├── code/ │ └── lora_fine_tune.py # CLI entry point: tokenize / train / evaluate / predict ├── dataset/ # TSV data (sequence, label columns), split into train/valid/test └── model/ # Base FunDLM-Medium-1024bp checkpoint (or HF model id)

## Running classification

Classification is run through `lora_fine_tune.py`, which supports four subcommands:
`tokenize`, `train`, `evaluate`, and `predict`. This fine-tunes/evaluates a LoRA adapter
on top of the FunDLM-Medium-1024bp base model for the centromere classification task.

Set up your paths first:

```bash
PROCESSED_DIR="/path/to/FunDLM/centromere/FunDLM-Medium-1024bp"   # output dir for tokenized data + checkpoints
TRAIN_DATA_SRC="/path/to/FunDLM/centromere/dataset/data1024bp"     # train/valid TSVs
TEST_DATA_SRC="/path/to/FunDLM/centromere/dataset/data1024/test"      # test TSVs
EVAL_SCRIPT="/path/to/FunDLM/code/lora_fine_tune.py"
MODEL_PATH="/path/to/FunDLM/models/FunDLM-Medium-1024bp"                # base model checkpoint or HF model id

Step 1: Tokenize the data

Each of train / valid / test needs to be tokenized separately. Input TSVs must have a sequence column and a label column (class labels for classification).

python "$EVAL_SCRIPT" tokenize \
  --data_dir "$PROCESSED_DIR/train.tsv" \
  --output_path "$PROCESSED_DIR/train.parquet" \
  --model_name "$MODEL_PATH" \
  --sequence_length 1024 \
  --task_type classification \
  --seq_column sequence \
  --label_column label

python "$EVAL_SCRIPT" tokenize \
  --data_dir "$PROCESSED_DIR/valid.tsv" \
  --output_path "$PROCESSED_DIR/valid.parquet" \
  --model_name "$MODEL_PATH" \
  --sequence_length 1024 \
  --task_type classification \
  --seq_column sequence \
  --label_column label

python "$EVAL_SCRIPT" tokenize \
  --data_dir "$PROCESSED_DIR/test.tsv" \
  --output_path "$PROCESSED_DIR/test.parquet" \
  --model_name "$MODEL_PATH" \
  --sequence_length 1024 \
  --task_type classification \
  --seq_column sequence \
  --label_column label

Step 2: LoRA fine-tuning

python "$EVAL_SCRIPT" train \
  --train_dir "$PROCESSED_DIR/train.parquet" \
  --valid_dir "$PROCESSED_DIR/valid.parquet" \
  --output_dir "$PROCESSED_DIR" \
  --model_name "$MODEL_PATH" \
  --task_type classification \
  --train_batch_size #train batch size \
  --eval_batch_size #eval batch size \
  --eval_num_samples #eval_num_samples \
  --max_steps #max_steps \
  --gradient_accumulation_steps #gradient_accumulation_steps \
  --learning_rate 1e-5 \
  --warmup_steps #warmup_steps \
  --lr_scheduler_type cosine \
  --weight_decay 0.01 \
  --bf16 True \
  --fp16 False \
  --eval_strategy steps \
  --eval_steps 2000 \
  --save_strategy steps \
  --save_steps 2000 \
  --logging_steps 100 \
  --seed 42 \
  --use_wandb True \
  --wandb_project lora-fine-tune \
  --max_grad_norm 0.5 \
  --remove_unused_columns False

This saves periodic checkpoints under $PROCESSED_DIR/checkpoint-*. --use_wandb True logs training to Weights & Biases — set --use_wandb False to disable, or make sure you're logged in (wandb login) first.

Step 3: Evaluation

Find the most recent checkpoint and evaluate it on the held-out validation set:

export OUTPUT_DIR="$PROCESSED_DIR"

# Find the latest checkpoint
CHECKPOINT_PATH=$(ls -td ${OUTPUT_DIR}/checkpoint-* | head -1)
echo "Using checkpoint: $CHECKPOINT_PATH"

export CUDA_VISIBLE_DEVICES=0

python "$EVAL_SCRIPT" evaluate \
    --checkpoint_dir "$CHECKPOINT_PATH" \
    --data_dir "$PROCESSED_DIR/valid.parquet" \
    --output_dir "$OUTPUT_DIR/eval-valid" \
    --model_name "$MODEL_PATH" \
    --task_type "classification" \
    --batch_size 4 \
    --bf16 True \
    --seed 42

Step 4: Prediction (inference on new/test sequences)

export CUDA_VISIBLE_DEVICES=0

python "$EVAL_SCRIPT" predict \
     --checkpoint_dir "$CHECKPOINT_PATH" \
     --data_dir "$PROCESSED_DIR/test.parquet" \
     --output_file "$OUTPUT_DIR/eval-test/predictions_centromere_medium_1024bp.csv" \
     --model_name "$MODEL_PATH" \
     --task_type "classification" \
     --batch_size 32 \
     --bf16 True \
     --seed 42 \
     --seq_column "sequence"