About OpenAudio S1

OpenAudio S1 represents the latest generation voice generation model officially launched by Fish Audio, setting a new benchmark in AI voice technology. It is a brand-new upgrade based on the Fish Speech series, designed to achieve unprecedented levels of speech naturalness and expressiveness. The mission is to reach the expressiveness and naturalness of professional voice actors, transforming what's possible in Text-to-Speech (TTS).

Innovations & Core Capabilities

• Highly Natural Sound: Generated voices are smooth and realistic, often described as almost indistinguishable from human voiceovers, making them suitable for professional scenarios like video dubbing, podcasts, and game character voices.
• Advanced Emotional & Tone Control: OpenAudio S1 supports over 50 emotions and tone markers, including (angry), (happy), (sad), (whisper), (sympathy), (excited), (nervous), and (joyful). It introduces layered emotional control, capable of understanding complex prompts like "excited but nervous" and adding subtle sound cues such as laughter, whispers, or even crying. This allows for highly personalized and emotionally nuanced voice outputs.
• Strong Instruction-Following Capability: Users can control speech details like rate, volume, pauses, and even laughter with simple text commands. Developers can further customize tone, emphasis, and pacing in real-time via API, providing full creative control.
• Ultra-Realistic Voice Cloning: Zero-shot and few-shot voice cloning is supported, requiring only 10 to 30 seconds of audio samples to generate high-fidelity cloned voices, with the entire process taking less than a minute. This feature captures the unique speaking patterns, rhythm, and style of any voice.
• Multilingual and Cross-lingual Support: OpenAudio S1 supports 13 languages, including English, Chinese, Japanese, Korean, French, German, Arabic, and Spanish. A significant technical advantage is its no phoneme dependency, allowing it to handle text in any language script by simply copying and pasting. It supports 11 languages with real-time performance and under 100 milliseconds of latency, making it ideal for global experiences.

Technical Excellence

• Extensive Training Data: OpenAudio S1 was trained on an enormous dataset of 2 million hours of audio training data, leading to significant breakthroughs in voice generation quality and diversity.
• Innovative Dual-AR Architecture: Utilizes a unique dual autoregressive (Dual-AR) architecture that combines fast and slow Transformer modules to optimize the stability and efficiency of voice generation. This architecture also enhances codebook processing with grouped finite scalar vector quantization (GFSQ) technology.
• RLHF-Driven Emotional Expression: Voice emotional expression is significantly enhanced through online Reinforcement Learning from Human Feedback (RLHF) technology, enabling the model to precisely capture voice timbre and intonation for more natural emotional expressions. Both S1 and S1-mini incorporate this technology.

Performance and Recognition

• Accuracy: In Seed TTS assessment, OpenAudio S1 achieved an English word error rate (WER) as low as 0.008 and a character error rate (CER) of only 0.004, significantly surpassing traditional models.
• Ranking: Ranked first on the TTS-Arena leaderboard (TTS-Arena2) under the name "Anonymous Sparkle," outperforming numerous other models and receiving widespread recognition for its realistic voice quality and delicate emotional expression.
• Speed: Cloud processing can generate high-quality voice in an average of 20 seconds, and supports batch processing for large-scale commercial applications. With "fish-tech acceleration," it achieves a real-time factor of approximately 1:5 on an Nvidia RTX 4060 laptop and 1:15 on an Nvidia RTX 4090.

Flexible Deployment Options

• OpenAudio S1 (4B parameters): The full-featured flagship proprietary model, available through Fish Audio's cloud services, providing the highest quality speech synthesis and advanced features.
• OpenAudio S1-mini (0.5B parameters): A distilled version with core capabilities, made fully open-source and available via GitHub and Hugging Face Space. It is optimized for faster inference while maintaining excellent quality, suitable for research and educational scenarios.

The platform features an easy-to-use, Gradio-based web UI compatible with common browsers, and also offers a PyQt6 graphical interface. It is designed to be deploy-friendly with native support for Linux and Windows. A free plan allows users to test the S1 model with a certain credit limit per month.

Practical Applications

• Content Creation: Generating professional-grade voiceovers for videos, podcasts, and audiobooks.
• Virtual Assistants: Creating personalized voice navigation or customer service systems with multilingual interactions.
• Games and Entertainment: Generating realistic dialogues and narrations for game characters to enhance immersive experiences.
• Education and Accessibility: Providing high-quality text-to-speech services for visually impaired users or generating multilingual learning content for educational platforms.

OpenAudio S1 serves as a creative partner for creators, developers, educators, and voice professionals.

Future Vision

The release of OpenAudio S1 is just the beginning. Fish Audio plans to introduce real-time voice interaction features to enable seamless conversations with voice library characters. Through continuous expansion of training data and optimization of RLHF, S1 is expected to support more languages and even more complex emotional expressions, further solidifying a leading position in the TTS field and reshaping the landscape of voice applications.