VibeVoice
ActiveDescription
Open-source frontier voice AI from Microsoft, providing high-quality speech synthesis and recognition for building real-time conversational voice agent applications.
Key Features
- 60-minute single-pass ASR — Processes up to 60 minutes of continuous audio in one pass with consistent speaker tracking throughout
- Structured transcription output — Jointly produces speaker identity, timestamps, and content (Who/When/What) without post-processing
- Custom hotword guidance — Accepts domain-specific terms and proper nouns to significantly improve recognition accuracy on specialized content
- 90-minute multi-speaker TTS — Synthesizes up to 90 minutes of conversational speech with up to 4 distinct speakers
- Continuous speech tokenizers — Uses Acoustic and Semantic tokenizers at 7.5Hz ultra-low frame rate for efficient long-sequence processing
- Real-time streaming TTS — Supports streaming text input and real-time speech generation for conversational agent applications
Use Cases
💡 Meeting transcription: Automatically convert hour-long meeting recordings into labeled transcripts with speaker identification and timestamps
💡 Podcast and audiobook generation: Use multi-speaker TTS to synthesize long-form conversational podcasts or audiobook content
💡 Real-time voice agent building: Construct AI agents with real-time voice interaction using streaming TTS and ASR capabilities
💡 Multilingual speech recognition: Process long audio in 50+ languages for multinational meetings and multilingual content
Strengths & Limitations
✅ Strengths
- • Actively maintained, recent updates
- • High community interest (53.6k stars)
- • Permissive open-source license (MIT)
- • Responsive to issues, low backlog
Categories
Quick Start
# Install dependencies
pip install torch torchaudio
pip install git+https://github.com/microsoft/VibeVoice.git
# ASR transcription example
from vibevoice.asr import VibeVoiceASR
model = VibeVoiceASR.from_pretrained("microsoft/VibeVoice-ASR")
result = model.transcribe("meeting_recording.wav")
print(result.text)