glap_ft

SPEAR-GLAP fine-tuned checkpoint

This is the HF-native best checkpoint produced by audio-side fine-tuning of GLAP on 12,718 SPEAR audio-query pairs. The text encoder and text projection were frozen; the audio encoder and audio projection were optimized in FP32.

The implementation is released from commit cbf5599c5adc95c4d9f4128a0e9c6773aa2bc0d4 of SUAT-AIRI/SPEAR-GLAP. This repository contains custom model code. Review it and pin an immutable Hugging Face revision before setting trust_remote_code=True in a production or reproducible workflow.

Important evaluation status

The supplied 5,451-row test manifest was evaluated after every epoch and used to choose this checkpoint. Therefore all numbers below are test-tuned and must not be presented as unbiased holdout results. Retrieval uses one paired target audio per query row, including nine rows whose query text is repeated.

The selected checkpoint is epoch 4 at optimizer step 1,592. Its model.safetensors SHA-256 is cb97916b295ded7b78fe0b74561e2db09ee79948dd5bb98b822771e4148273f9.

Loading

The validated runtime uses Python 3.10, PyTorch/Torchaudio 2.6.0 with CUDA 12.4, Transformers 4.40.2, SentencePiece, Einops, and Safetensors. Pinned revisions support fully offline text and audio loading from a custom Hugging Face cache.

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "YirongSun/glap_ft",
    trust_remote_code=True,
)
model.eval()

with torch.inference_mode():
    text_embeddings = model.get_text_features(
        ["A calm adult speaker in a quiet room."],
        source_lang="eng_Latn",
    )

# Audio input is a float waveform batch shaped [batch, samples] at 16 kHz.
audio = torch.zeros(1, 16000)
with torch.inference_mode():
    audio_embeddings = model.get_audio_features(audio)

similarity = text_embeddings @ audio_embeddings.T

Both feature methods return L2-normalized 1,024-dimensional embeddings. Audio must be decoded, converted to mono, and resampled to 16 kHz by the caller.

Training recipe and provenance

The release includes training_recipe.yaml, training_metadata.json, evaluation_summary.json, and provenance.json. They record the frozen input hashes, exact executable source fingerprint, checkpoint-selection rule, and machine-readable metrics without publishing host-specific data paths.

Limitations

• Test-set checkpoint selection makes the reported results test-tuned.
• Repeated text is evaluated per manifest row against that row's paired audio, not as a multi-positive target.
• The supplied split does not guarantee speaker, dialogue, transcript, or identity isolation.
• The maintained fine-tuning recipe is single-GPU FP32 and performs no hyperparameter search.

Attribution

SPEAR-GLAP derives from Xiaomi Research's Dasheng-GLAP. The retained model implementation and this checkpoint are distributed under Apache-2.0; see LICENSE and the source repository for details.