Voice AI Engine Development

voice-ai-engine-development

Architecting real-time Voice AI agents.

Dokhacgiakhoa/antigravity-ide·0installs507stars

SKILL.md

Full skill instructions

Voice AI Engine Development

Goal: Build low-latency, conversational Voice AI agents capable of full-duplex communication.

1. The Voice Pipeline (Latency is King)

The total loop Latency (Voice-to-Ear) should be < 1000ms (Ideal < 500ms).

Transport: WebRTC (preferred for browser) or WebSocket (server-server).
VAD (Voice Activity Detection): Detect when user starts/stops speaking.
- Tools: Silero VAD, WebRTC VAD.
STT (Speech-to-Text): Transcribe audio to text.
- Tools: Deepgram (fastest), Whisper (high accuracy but slower), AssemblyAI.
LLM (Brain): Process text and generate response.
- Tools: Groq (Llama 3), GPT-4o, Claude 3.5 Sonnet.
TTS (Text-to-Speech): Convert response to audio.
- Tools: ElevenLabs (Quality), Cartesia (Speed), OpenAI TTS.

2. Architecture Patterns

Streaming Pipeline: DO NOT wait for full transcription or full generation. Stream everything.
- User Audio Stream -> VAD -> STT Stream -> LLM Stream -> TTS Stream -> Audio Output.
Interruption Handling (Barge-in):
- If VAD detects user speech while AI is talking -> Immediately CUT text generation and audio playback. Clear buffers.

3. Implementation Stack

Backend: Python (FastAPI) or Node.js. Python ecosystem is stronger for audio processing (numpy/scipy).
Frameworks:
- Pipecat: Open source framework for building voice agents.
- LiveKit: WebRTC infrastructure for real-time audio/video.
- Twilio: For telephony integration.

4. Optimization Techniques

Optimistic VAD: Tune VAD to be sensitive to start, but careful with "silence" timeout (usually 500ms-800ms) to detect end of turn.
Prompt Engineering: Instruct LLM to be concise and conversational.
- System Prompt: "You are a helpful voice assistant. Keep responses short (1-2 sentences). Do not use markdown or emojis."
Audio Formats: Use OPUS or PCM (16khz/24khz/48khz) for transmission. Avoid MP3 transcoding latency.

5. Debugging & Metrics

WER (Word Error Rate): For STT accuracy.
TTFT (Time to First Token): LLM speed.
TTA (Time to Audio): The critical metric. Time from user silence to first AI sound.

Common Pitfalls:

Echo cancellation issues (User hears themselves). Use WebRTC's built-in AEC.
Hallucination in STT (Whisper transcribing silence).
Race conditions during interruptions.

Install skill

Copy and run in your terminal

npx -y skills add https://github.com/Dokhacgiakhoa/antigravity-ide --skill voice-ai-engine-development

View on GitHub

.agent/skills/voice-ai-engine-development/SKILL.md

shopify-development
Build Shopify apps, extensions, themes using GraphQL Admin API, Shopify CLI, Polaris UI, and Liquid. TRIGGER: "shopify", "shopify app", "checkout extension", "admin extension", "POS extension", "shopify theme", "liquid template", "polaris", "shopify graphql", "shopify webhook", "shopify billing",...
1 installs
social-content
When the user wants help creating, scheduling, or optimizing social media content for LinkedIn, Twitter/X, Instagram, TikTok, Facebook, or other platforms. Also use when the user mentions 'LinkedIn post,' 'Twitter thread,' 'social media,' 'content calendar,' 'social scheduling,' 'engagement,' or ...
0 installs
javascript-testing-patterns
Implement comprehensive testing strategies using Jest, Vitest, and Testing Library for unit tests, integration tests, and end-to-end testing with mocking, fixtures, and test-driven development. Use when writing JavaScript/TypeScript tests, setting up test infrastructure, or implementing TDD/BDD w...
0 installs
api-documenter
Master API documentation with OpenAPI 3.1.
0 installs

More skills from Dokhacgiakhoa

Dokhacgiakhoa/antigravity-ide

langchain-architecture

Design LLM applications using the LangChain framework with agents, memory, and tool integration patterns. Use when building LangChain applications, implementing AI agents, or creating complex LLM workflows.

507 0

View

Dokhacgiakhoa/antigravity-ide

nextjs-app-router-patterns

Master Next.js 14+ App Router with Server Components, streaming, parallel routes, and advanced data fetching. Use when building Next.js applications, implementing SSR/SSG, or optimizing React Server Components.

507 0

View

Dokhacgiakhoa/antigravity-ide

startup-metrics-framework

This skill should be used when the user asks about "key startup

507 0

View

Dokhacgiakhoa/antigravity-ide

frontend-mobile-security-xss-scan

You are a frontend security specialist focusing on Cross-Site Scripting (XSS) vulnerability detection and prevention. Analyze React, Vue, Angular, and vanilla JavaScript code to identify injection poi

507 0

View

Dokhacgiakhoa/antigravity-ide

deployment-validation-config-validate

You are a configuration management expert specializing in validating, testing, and ensuring the correctness of application configurations. Create comprehensive validation schemas, implement configurat

507 0

View

Dokhacgiakhoa/antigravity-ide

Cross-Site Scripting and HTML Injection Testing

This skill should be used when the user asks to "test for XSS vulnerabilities", "perform cross-site scripting attacks", "identify HTML injection flaws", "exploit client-side injection vulnerabilities", "steal cookies via XSS", or "bypass content security policies". It provides comprehensive techn...

507 0

View

Dokhacgiakhoa/antigravity-ide

nodejs-backend-patterns

Build production-ready Node.js backend services with Express/Fastify, implementing middleware patterns, error handling, authentication, database integration, and API design best practices. Use when creating Node.js servers, REST APIs, GraphQL backends, or microservices architectures.

507 0

View

Dokhacgiakhoa/antigravity-ide

projection-patterns

Build read models and projections from event streams. Use when implementing CQRS read sides, building materialized views, or optimizing query performance in event-sourced systems.

507 0

View

Dokhacgiakhoa/antigravity-ide

backend-architect

Expert backend architect specializing in scalable API design,

507 0

View

Dokhacgiakhoa/antigravity-ide

seo-content-planner

Creates comprehensive content outlines and topic clusters for SEO.

507 0

View

Dokhacgiakhoa/antigravity-ide

llm-application-dev-langchain-agent

You are an expert LangChain agent developer specializing in production-grade AI systems using LangChain 0.1+ and LangGraph.

507 0

View

Dokhacgiakhoa/antigravity-ide

personal-tool-builder

Expert in building custom tools that solve your own problems first. The best products often start as personal tools - scratch your own itch, build for yourself, then discover others have the same itch. Covers rapid prototyping, local-first apps, CLI tools, scripts that grow into products, and the...

507 0

View