fal.ai Audio Models

Quick Reference

When to Use This Skill

Use for audio processing:

• Transcribing audio/​video to text
• Generating subtitles with timestamps
• Translating speech to English
• Cloning voices from reference audio
• Generating natural speech from text

Related skills:

• For video with audio: see fal-text-to-video
• For API integration: see fal-api-reference
• For model comparison: see fal-model-guide

---

fal.ai Audio Models

Complete reference for speech-to-text (STT) and text-to-speech (TTS) models on fal.ai.

Speech-to-Text Models

Whisper (OpenAI)

Endpoint: fal-ai/​whisper Best For: Accurate transcription and translation

The industry-standard speech recognition model with support for 99+ languages.

import { fal } from "@fal-ai/​client";

const result = await fal.subscribe("fal-ai/​whisper", {
  input: {
    audio_url: "https://example.com/​speech.mp3",
    task: "transcribe",
    language: "en",
    chunk_level: "segment"
  }
});

console.log(result.text);
console.log(result.chunks);  // With timestamps

import fal_client

result = fal_client.subscribe(
    "fal-ai/​whisper",
    arguments={
        "audio_url": "https://example.com/​speech.mp3",
        "task": "transcribe",
        "language": "en",
        "chunk_level": "segment"
    }
)
print(result["text"])
for chunk in result["chunks"]:
    print(f"[{chunk['timestamp'][0]:.2f}-{chunk['timestamp'][1]:.2f}] {chunk['text']}")

Whisper Parameters:

Response Structure:

interface WhisperOutput {
  text: string;  // Full transcription
  chunks?: Array<{
    text: string;
    timestamp: [number, number];  // [start, end] in seconds
  }>;
}

Whisper Turbo

Endpoint: fal-ai/​whisper-turbo Best For: Fast transcription

const result = await fal.subscribe("fal-ai/​whisper-turbo", {
  input: {
    audio_url: "https://example.com/​podcast.mp3",
    task: "transcribe"
  }
});

Whisper Large v3

Endpoint: fal-ai/​whisper-large-v3 Best For: Maximum accuracy

const result = await fal.subscribe("fal-ai/​whisper-large-v3", {
  input: {
    audio_url: "https://example.com/​meeting.mp3",
    task: "transcribe",
    language: "en"
  }
});

Whisper Usage Examples

Transcription with Timestamps:

const result = await fal.subscribe("fal-ai/​whisper", {
  input: {
    audio_url: audioUrl,
    task: "transcribe",
    chunk_level: "segment"
  }
});

// Format as SRT subtitles
result.chunks.forEach((chunk, i) => {
  const start = formatTime(chunk.timestamp[0]);
  const end = formatTime(chunk.timestamp[1]);
  console.log(`${i + 1}\n${start} --> ${end}\n${chunk.text}\n`);
});

function formatTime(seconds: number): string {
  const h = Math.floor(seconds / 3600);
  const m = Math.floor((seconds % 3600) / 60);
  const s = Math.floor(seconds % 60);
  const ms = Math.floor((seconds % 1) * 1000);
  return `${h.toString().padStart(2, '0')}:${m.toString().padStart(2, '0')}:${s.toString().padStart(2, '0')},${ms.toString().padStart(3, '0')}`;
}

Translation (Non-English to English):

const result = await fal.subscribe("fal-ai/​whisper", {
  input: {
    audio_url: "https://example.com/​french-speech.mp3",
    task: "translate",  // Translates to English
    language: "fr"
  }
});

console.log(result.text);  // English translation

Multi-Language Detection:

// Whisper auto-detects language if not specified
const result = await fal.subscribe("fal-ai/​whisper", {
  input: {
    audio_url: "https://example.com/​unknown-language.mp3",
    task: "transcribe"
    // language omitted - auto-detect
  }
});

Text-to-Speech Models

F5-TTS

Endpoint: fal-ai/​f5-tts Best For: Voice cloning from reference audio

const result = await fal.subscribe("fal-ai/​f5-tts", {
  input: {
    gen_text: "Hello! Welcome to our product demonstration. We're excited to show you what we've built.",
    ref_audio_url: "https://example.com/​voice-sample.wav",
    ref_text: "This is a sample of my voice for cloning purposes.",
    model_type: "F5-TTS"
  }
});

console.log(result.audio_url);

result = fal_client.subscribe(
    "fal-ai/​f5-tts",
    arguments={
        "gen_text": "Hello! Welcome to our product.",
        "ref_audio_url": "https://example.com/​voice-sample.wav",
        "ref_text": "This is a sample of my voice."
    }
)
print(result["audio_url"])

F5-TTS Parameters:

ElevenLabs TTS

Endpoint: fal-ai/​elevenlabs/​tts Best For: Premium voice quality

const result = await fal.subscribe("fal-ai/​elevenlabs/​tts", {
  input: {
    text: "Welcome to fal.ai! Let me tell you about our amazing AI models.",
    voice_id: "21m00Tcm4TlvDq8ikWAM",  // ElevenLabs voice ID
    model_id: "eleven_multilingual_v2"
  }
});

console.log(result.audio.url);

ElevenLabs Parameters:

ElevenLabs Voice IDs (examples):

• 21m00Tcm4TlvDq8ikWAM - Rachel (female)
• AZnzlk1XvdvUeBnXmlld - Domi (female)
• EXAVITQu4vr4xnSDxMaL - Bella (female)
• ErXwobaYiN019PkySvjV - Antoni (male)
• VR6AewLTigWG4xSOukaG - Arnold (male)

Kokoro TTS

Endpoint: fal-ai/​kokoro/​american-english Best For: Multi-language, natural sounding

const result = await fal.subscribe("fal-ai/​kokoro/​american-english", {
  input: {
    text: "This is a test of the Kokoro text-to-speech system.",
    voice: "af_bella"  // Voice style
  }
});

console.log(result.audio.url);

Kokoro Variants:

• fal-ai/​kokoro/​american-english - American English
• fal-ai/​kokoro/​british-english - British English
• fal-ai/​kokoro/​japanese - Japanese
• fal-ai/​kokoro/​mandarin - Mandarin Chinese

Kokoro Parameters:

XTTS (Coqui)

Endpoint: fal-ai/​xtts Best For: Open-source voice cloning

const result = await fal.subscribe("fal-ai/​xtts", {
  input: {
    text: "Hello, this is a cloned voice speaking.",
    audio_url: "https://example.com/​voice-reference.wav",
    language: "en"
  }
});

XTTS Parameters:

Model Comparison

Speech-to-Text

Text-to-Speech

Workflow Examples

Transcribe and Translate Pipeline

async function processAudio(audioUrl: string, targetLanguage: string = 'en') {
  // 1. Transcribe
  const transcription = await fal.subscribe("fal-ai/​whisper", {
    input: {
      audio_url: audioUrl,
      task: "transcribe",
      chunk_level: "segment"
    }
  });

  // 2. If not English, translate
  let translation = null;
  if (targetLanguage === 'en') {
    translation = await fal.subscribe("fal-ai/​whisper", {
      input: {
        audio_url: audioUrl,
        task: "translate"
      }
    });
  }

  return {
    original: transcription.text,
    translated: translation?.text,
    chunks: transcription.chunks
  };
}

Voice Cloning Pipeline

async function cloneVoiceAndSpeak(
  referenceAudioUrl: string,
  referenceText: string,
  textToSpeak: string
) {
  // Use F5-TTS for voice cloning
  const result = await fal.subscribe("fal-ai/​f5-tts", {
    input: {
      gen_text: textToSpeak,
      ref_audio_url: referenceAudioUrl,
      ref_text: referenceText,
      remove_silence: true
    }
  });

  return result.audio_url;
}

Subtitle Generation

async function generateSubtitles(videoUrl: string): Promise<string> {
  // Extract audio and transcribe
  const result = await fal.subscribe("fal-ai/​whisper", {
    input: {
      audio_url: videoUrl,  // Works with video URLs too
      task: "transcribe",
      chunk_level: "segment"
    }
  });

  // Generate SRT format
  let srt = '';
  result.chunks.forEach((chunk, i) => {
    srt += `${i + 1}\n`;
    srt += `${formatSrtTime(chunk.timestamp[0])} --> ${formatSrtTime(chunk.timestamp[1])}\n`;
    srt += `${chunk.text}\n\n`;
  });

  return srt;
}

function formatSrtTime(seconds: number): string {
  const date = new Date(seconds * 1000);
  return date.toISOString().substr(11, 12).replace('.', ',');
}

Audio Book Generation

async function generateAudioBook(chapters: string[], voiceId: string) {
  const audioUrls = [];

  for (const chapter of chapters) {
    // Split into manageable chunks
    const chunks = splitText(chapter, 5000);

    for (const chunk of chunks) {
      const result = await fal.subscribe("fal-ai/​elevenlabs/​tts", {
        input: {
          text: chunk,
          voice_id: voiceId,
          model_id: "eleven_multilingual_v2"
        }
      });
      audioUrls.push(result.audio.url);
    }
  }

  return audioUrls;
}

function splitText(text: string, maxLength: number): string[] {
  const chunks = [];
  let current = '';

  text.split('. ').forEach(sentence => {
    if ((current + sentence).length < maxLength) {
      current += sentence + '. ';
    } else {
      chunks.push(current.trim());
      current = sentence + '. ';
    }
  });

  if (current) chunks.push(current.trim());
  return chunks;
}

Parameter Reference

Speech-to-Text Input

interface STTInput {
  audio_url: string;
  task?: "transcribe" | "translate";
  language?: string;  // ISO 639-1 code
  chunk_level?: "segment";
  version?: string;
}

Text-to-Speech Input

interface TTSInput {
  // Common
  text?: string;
  gen_text?: string;

  // Voice cloning
  ref_audio_url?: string;
  ref_text?: string;
  audio_url?: string;  // XTTS

  // Voice selection
  voice_id?: string;  // ElevenLabs
  voice?: string;     // Kokoro
  model_type?: string; // F5-TTS

  // Control
  speed?: number;
  stability?: number;
  similarity_boost?: number;
  language?: string;
  remove_silence?: boolean;
}

Best Practices

Speech-to-Text

1. Audio Quality: Clean audio = better transcription
2. Specify Language: Provide language hint when known
3. Use Timestamps: Request chunk_level: "segment" for subtitles
4. Handle Long Audio: Whisper handles long files automatically
5. Translation: Use task: "translate" for non-English to English

Text-to-Speech

1. Reference Quality: For voice cloning, use 10-30 second clear samples
2. Reference Transcript: Accurate transcript improves cloning quality
3. Text Length: Split very long text into chunks
4. Punctuation: Proper punctuation improves prosody
5. Emotion Hints: Use punctuation (!, ?) to convey emotion

Common Supported Languages

File Format Support

Input Formats (STT)

Output Formats (TTS)

Error Handling

try {
  const result = await fal.subscribe("fal-ai/​whisper", {
    input: { audio_url: audioUrl, task: "transcribe" }
  });
} catch (error) {
  if (error.status === 400) {
    console.error("Invalid audio file or URL");
  } else if (error.status === 413) {
    console.error("Audio file too large");
  } else {
    console.error("Transcription failed:", error.message);
  }
}