Octopus

🐙 Octopus: Towards Building the Arabic Speech LLM Suite

📢 Overview

Octopus is a bilingual Audio-Language Model (Audio-LLM) family developed to understand, transcribe, translate, and reason over Arabic and English speech.
It unifies audio, text, and reasoning within one multimodal framework, supporting:

• Automatic Speech Recognition (ASR) for Arabic & English 🗣️
• Speech Translation (Arabic → English and vice versa) 🌍
• Arabic Dialect Identification (DID) 🏷️

The lightweight variant, TinyOctopus, maintains the same modular design but is optimized for efficiency on smaller GPUs.

🧩 Architecture

Core Components

The Octopus family scales across several encoder–decoder configurations, combining complementary strengths in acoustic understanding and text generation.

1. Audio Encoders

   • Distil-Whisper (distil-large-v3) → lightweight frozen encoder producing compact speech embeddings.
   • Whisper-large-v3 → high-capacity encoder for robust transcription and multilingual coverage.
   • BEATs (Microsoft) → self-supervised audio encoder capturing fine-grained acoustic cues such as timbre and speaker traits.

2. Alignment & Fusion

   • Cross-Attention Projection Layer → a trainable bridge that aligns audio representations with the text-language space through cross-modal attention.

3. Language / Decoder Models

   • DeepSeek 1.5B → efficient generative decoder for reasoning, dialogue, and translation.
   • LLaMA 3.2 1B → compact Arabic–English language model variant optimized for code-switching and reasoning on limited hardware.
   • ALLaM 13B → large bilingual decoder offering high-fidelity generation and deeper contextual grounding for Arabic tasks.

Together these components enable the Octopus line—from TinyOctopus (Distil-Whisper + LLaMA 3.2 1B or DeepSeek 1.5B) up to full ALLaM-Octopus (Whisper large v3 + BEATs + ALLaM 13 B) to handle diverse audio understanding and speech-to-text reasoning tasks across Arabic and English.

📚 Training Datasets

The Octopus models were trained and evaluated on a diverse collection of Arabic, English, and code-switching speech corpora, totaling ≈25,000 hours of high-quality data for ASR, translation, and dialect identification.

Total Coverage: ≈25,000 hours of speech across Arabic, English, and mixed-language domains — enabling broad generalization for ASR, translation, and dialect identification.

These datasets jointly provide:

• Balanced representation across dialects.
• Both natural and synthetic speech sources for enhanced robustness.
• Parallel Arabic–English pairs enabling bilingual text generation and translation.

🧮 Model Weights & Resources

The full set of model weights (including large checkpoints) is publicly available here:
➡️ Octopus Model Weights

⚙️ Installation & Usage

💻 Install Dependencies

pip install -r requirements.txt

Inference

from inference import transcribe

audio_path = "path/to/audio.wav"  # Replace with your actual audio file
output = transcribe(audio_path, task="asr")  # Options: "dialect", "asr", "translation"

print("Generated Text:", output)

🧪 Evaluation Results

🎙️ ASR Performance (WER ↓)

+86 % English improvement observed with the addition of language-tokens for bilingual and translation variants.

---

🪶 Tiny-Octopus & Fine-Tuning (WER ↓)

Code-Switch TTS augmentation yielded ≈ 20 % WER reduction across multilingual evaluation sets.

---

🌍 Translation Performance (BLEU ↑ / BERT-F1 ↑)

Trans-Octopus achieves the best BLEU and BERT-F1 on CoVoST2 and competitive results on FLEURS, surpassing SeamlessM4T in low-resource conditions.

---

🏷️ Dialect Identification

For dialect identification, the Tiny-Octopus models achieved 87 – 89 % accuracy across all 17 dialects in ADI-17.
The confusion matrices reveal clear separation among Gulf, Levantine, North-African, and Egyptian clusters — showing that even compact models can internalize subtle dialectal cues when trained in a multitask setting.

Examples

Example 1: Arabic Speech Recognition

🎵 Audio Input (Arabic):
<audio controls> <source src="https://huggingface.co/ArabicSpeech/Octopus/resolve/main/examples/03BD00C0_2C0B_4C81_BA8C_018175D0B4E3_utt_1_align.wav" type="audio/wav"> </audio>

📝 User Prompt:

Transcribe the audio or قم بتفريغ المقطع الصوتي

💡 System Response:

أهلا بكم مشاهدينا الكرام في حلقة جديدة من برنامج الاقتصاد والناس

🎵 Audio Input (English):
<audio controls> <source src="https://huggingface.co/ArabicSpeech/Octopus/resolve/main/examples/4970-29093-0016.wav" type="audio/wav"> </audio>

📝 User Prompt:

Transcribe the audio or قم بتفريغ المقطع الصوتي

💡 System Response:

NO IT'S NOT TOO SOON

---

Example 2: Arabic to English Translation

🎵 Audio Input:
<audio controls> <source src="https://huggingface.co/ArabicSpeech/Octopus/resolve/main/examples/03BD00C0_2C0B_4C81_BA8C_018175D0B4E3_utt_21_align.wav" type="audio/wav"> </audio>

📝 User Prompt:

Translate the following Arabic speech into English or قم بترجمة المقطع للإنجليزية

💡 System Response:

I took a loan a certain amount of money to pay off the debt

---

Example 3: Dialect Identification

🎵 Audio Input:
<audio controls> <source src="https://huggingface.co/ArabicSpeech/Octopus/resolve/main/examples/tYBpZAOFpvk_071631-073831.wav" type="audio/wav"> </audio>

📝 User Prompt:

Identify the dialect of the given speech or ماهي لهجة المتحدث؟

💡 System Response:

KSA

---