Speech Recognition on Producthunt daily

VibeVoice

Thu, 02 Apr 2026 16:14:14 +0800

microsoft/VibeVoice

🎙️ VibeVoice: Open-Source Frontier Voice AI

📰 News

🎉 VibeVoice-ASR is being adopted by the open-source community! Vibing, a voice-powered input method, is now built on top of VibeVoice-ASR. Download: macOS | Windows

https://github.com/user-attachments/assets/db0bb23f-ae06-4135-a66a-1ff1669f4f84

2026-03-06: 🚀 VibeVoice ASR is now part of a Transformers release! You can now use our speech recognition model directly through the Hugging Face Transformers library for seamless integration into your projects.

2026-01-21: 📣 We open-sourced VibeVoice-ASR, a unified speech-to-text model designed to handle 60-minute long-form audio in a single pass, generating structured transcriptions containing Who (Speaker), When (Timestamps), and What (Content), with support for User-Customized Context. Try it in Playground.

⭐️ VibeVoice-ASR is natively multilingual, supporting over 50 languages — check the supported languages for details.
🔥 The VibeVoice-ASR finetuning code is now available!
⚡️ vLLM inference is now supported for faster inference; see vllm-asr for more details.
📑 VibeVoice-ASR Technique Report is available.

2025-12-16: 📣 We added experimental speakers to VibeVoice‑Realtime‑0.5B for exploration, including multilingual voices in nine languages (DE, FR, IT, JP, KR, NL, PL, PT, ES) and 11 distinct English style voices. Try it. More speaker types will be added over time.

2025-12-03: 📣 We open-sourced VibeVoice‑Realtime‑0.5B, a real‑time text‑to‑speech model that supports streaming text input and robust long-form speech generation. Try it on Colab.

2025-09-05: VibeVoice is an open-source research framework intended to advance collaboration in the speech synthesis community. After release, we discovered instances where the tool was used in ways inconsistent with the stated intent. Since responsible use of AI is one of Microsoft’s guiding principles, we have removed the VibeVoice-TTS code from this repository.

2025-08-25: 📣 We open-sourced VibeVoice-TTS, a long-form multi-speaker text-to-speech model that can synthesize speech up to 90 minutes long with up to 4 distinct speakers. — accepted as an Oral at ICLR 2026! 🔥

Overview

VibeVoice is a family of open-source frontier voice AI models that includes both Text-to-Speech (TTS) and Automatic Speech Recognition (ASR) models.

A core innovation of VibeVoice is its use of continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz. These tokenizers efficiently preserve audio fidelity while significantly boosting computational efficiency for processing long sequences. VibeVoice employs a next-token diffusion framework, leveraging a Large Language Model (LLM) to understand textual context and dialogue flow, and a diffusion head to generate high-fidelity acoustic details.

For more information, demos, and examples, please visit our Project Page.

Model	Weight	Quick Try
VibeVoice-ASR-7B	HF Link	Playground
VibeVoice-TTS-1.5B	HF Link	Disabled
VibeVoice-Realtime-0.5B	HF Link	Colab

Models

1. 📖 VibeVoice-ASR - Long-form Speech Recognition

VibeVoice-ASR is a unified speech-to-text model designed to handle 60-minute long-form audio in a single pass, generating structured transcriptions containing Who (Speaker), When (Timestamps), and What (Content), with support for Customized Hotwords.

🕒 60-minute Single-Pass Processing: Unlike conventional ASR models that slice audio into short chunks (often losing global context), VibeVoice ASR accepts up to 60 minutes of continuous audio input within 64K token length. This ensures consistent speaker tracking and semantic coherence across the entire hour.
👤 Customized Hotwords: Users can provide customized hotwords (e.g., specific names, technical terms, or background info) to guide the recognition process, significantly improving accuracy on domain-specific content.
📝 Rich Transcription (Who, When, What): The model jointly performs ASR, diarization, and timestamping, producing a structured output that indicates who said what and when.

📖 Documentation | 🤗 Hugging Face | 🎮 Playground | 🛠️ Finetuning | 📊 Paper

https://github.com/user-attachments/assets/acde5602-dc17-4314-9e3b-c630bc84aefa

2. 🎙️ VibeVoice-TTS - Long-form Multi-speaker TTS

Best for: Long-form conversational audio, podcasts, multi-speaker dialogues

⏱️ 90-minute Long-form Generation: Synthesizes conversational/single-speaker speech up to 90 minutes in a single pass, maintaining speaker consistency and semantic coherence throughout.
👥 Multi-speaker Support: Supports up to 4 distinct speakers in a single conversation, with natural turn-taking and speaker consistency across long dialogues.
🎭 Expressive Speech: Generates expressive, natural-sounding speech that captures conversational dynamics and emotional nuances.
🌐 Multi-lingual Support: Supports English, Chinese and other languages.

📖 Documentation | 🤗 Hugging Face | 📊 Paper

English

https://github.com/user-attachments/assets/0967027c-141e-4909-bec8-091558b1b784

Chinese

https://github.com/user-attachments/assets/322280b7-3093-4c67-86e3-10be4746c88f

Cross-Lingual

https://github.com/user-attachments/assets/838d8ad9-a201-4dde-bb45-8cd3f59ce722

Spontaneous Singing

https://github.com/user-attachments/assets/6f27a8a5-0c60-4f57-87f3-7dea2e11c730

Long Conversation with 4 people

https://github.com/user-attachments/assets/a357c4b6-9768-495c-a576-1618f6275727

3. ⚡ VibeVoice-Streaming - Real-time Streaming TTS

VibeVoice-Realtime is a lightweight real‑time text-to-speech model supporting streaming text input and robust long-form speech generation.

Parameter size: 0.5B (deployment-friendly)
Real-time TTS (~300 milliseconds first audible latency)
Streaming text input
Robust long-form speech generation (~10 minutes)

📖 Documentation | 🤗 Hugging Face | 🚀 Colab

https://github.com/user-attachments/assets/0901d274-f6ae-46ef-a0fd-3c4fba4f76dc

Contributing

Please see CONTRIBUTING.md for detailed contribution guidelines.

⚠️ Risks and Limitations

While efforts have been made to optimize it through various techniques, it may still produce outputs that are unexpected, biased, or inaccurate. VibeVoice inherits any biases, errors, or omissions produced by its base model (specifically, Qwen2.5 1.5b in this release). Potential for Deepfakes and Disinformation: High-quality synthetic speech can be misused to create convincing fake audio content for impersonation, fraud, or spreading disinformation. Users must ensure transcripts are reliable, check content accuracy, and avoid using generated content in misleading ways. Users are expected to use the generated content and to deploy the models in a lawful manner, in full compliance with all applicable laws and regulations in the relevant jurisdictions. It is best practice to disclose the use of AI when sharing AI-generated content.

We do not recommend using VibeVoice in commercial or real-world applications without further testing and development. This model is intended for research and development purposes only. Please use responsibly.

Star History

vosk-api

Thu, 12 Jun 2025 15:29:28 +0800

alphacep/vosk-api

Vosk Speech Recognition Toolkit

Vosk is an offline open source speech recognition toolkit. It enables speech recognition for 20+ languages and dialects - English, Indian English, German, French, Spanish, Portuguese, Chinese, Russian, Turkish, Vietnamese, Italian, Dutch, Catalan, Arabic, Greek, Farsi, Filipino, Ukrainian, Kazakh, Swedish, Japanese, Esperanto, Hindi, Czech, Polish. More to come.

Vosk models are small (50 Mb) but provide continuous large vocabulary transcription, zero-latency response with streaming API, reconfigurable vocabulary and speaker identification.

Speech recognition bindings implemented for various programming languages like Python, Java, Node.JS, C#, C++, Rust, Go and others.

Vosk supplies speech recognition for chatbots, smart home appliances, virtual assistants. It can also create subtitles for movies, transcription for lectures and interviews.

Vosk scales from small devices like Raspberry Pi or Android smartphone to big clusters.

Documentation

For installation instructions, examples and documentation visit Vosk Website.

KrillinAI

Wed, 16 Apr 2025 15:29:17 +0800

krillinai/KrillinAI

AI Audio&Video Translation and Dubbing Tool

English｜简体中文｜日本語｜한국어｜Français｜Deutsch｜Español｜Português｜Русский｜اللغة العربية

📢 New Release for Win & Mac Desktop Version – Welcome to Test and Provide Feedback

Overview

Krillin AI is an all-in-one solution for effortless video localization and enhancement. This minimalist yet powerful tool handles everything from translation, dubbing to voice cloning，formatting—seamlessly converting videos between landscape and portrait modes for optimal display across all content platforms(YouTube, TikTok, Bilibili, Douyin, WeChat Channel, RedNote, Kuaishou). With its end-to-end workflow, Krillin AI transforms raw footage into polished, platform-ready content in just a few clicks.

Key Features:

🎯 One-Click Start - Launch your workflow instantly,New desktop version available—easier to use!

📥 Video download - yt-dlp and local file uploading supported

📜 Precise Subtitles - Whisper-powered high-accuracy recognition

🧠 Smart Segmentation - LLM-based subtitle chunking & alignment

🌍 Professional Translation - Paragraph-level translation for consistency

🔄 Term Replacement - One-click domain-specific vocabulary swap

🎙️ Dubbing and Voice Cloning - CosyVoice selected or cloning voices

🎬 Video Composition - Auto-formatting for horizontal/vertical layouts

Showcase

The following picture demonstrates the effect after the subtitle file, which was generated through a one-click operation after importing a 46-minute local video, was inserted into the track. There was no manual adjustment involved at all. There are no missing or overlapping subtitles, the sentence segmentation is natural, and the translation quality is also quite high.

🔍 Speech Recognition Support

All local models in the table below support automatic installation of executable files + model files. Just make your selection, and KrillinAI will handle everything else for you.

Service	Supported Platforms	Model Options	Local/Cloud	Notes
OpenAI Whisper	Cross-platform	-	Cloud	Fast with excellent results
FasterWhisper	Windows/Linux	`tiny`/`medium`/`large-v2` (recommend medium+)	Local	Faster speed, no cloud service overhead
WhisperKit	macOS (Apple Silicon only)	`large-v2`	Local	Native optimization for Apple chips
Alibaba Cloud ASR	Cross-platform	-	Cloud	Bypasses China mainland network issues

🚀 Large Language Model Support

✅ Compatible with all OpenAI API-compatible cloud/local LLM services including but not limited to:

OpenAI
DeepSeek
Qwen (Tongyi Qianwen)
Self-hosted open-source models
Other OpenAI-format compatible API services

🌍 Language Support

Input languages: Chinese, English, Japanese, German, Turkish supported (more languages being added)
Translation languages: 56 languages supported, including English, Chinese, Russian, Spanish, French, etc.

Interface Preview

🚀 Quick Start

Basic Steps

First, download the Release executable file that matches your device’s system. Follow the instructions below to choose between the desktop or non-desktop version, then place the software in an empty folder. Running the program will generate some directories, so keeping it in an empty folder makes management easier.

[For the desktop version (release files with “desktop” in the name), refer here]
The desktop version is newly released to address the difficulty beginners face in editing configuration files correctly. It still has some bugs and is being continuously updated.

Double-click the file to start using it.

[For the non-desktop version (release files without “desktop” in the name), refer here]
The non-desktop version is the original release, with more complex configuration but stable functionality. It is also suitable for server deployment, as it provides a web-based UI.

Create a config folder in the directory, then create a config.toml file inside it. Copy the contents of the config-example.toml file from the source code’s config directory into your config.toml and fill in your configuration details. (If you want to use OpenAI models but don’t know how to get a key, you can join the group for free trial access.)

Double-click the executable or run it in the terminal to start the service.

Open your browser and enter http://127.0.0.1:8888 to begin using it. (Replace 8888 with the port number you specified in the config file.)

To: macOS Users

[For the desktop version, i.e., release files with “desktop” in the name, refer here]
The current packaging method for the desktop version cannot support direct double-click execution or DMG installation due to signing issues. Manual trust configuration is required as follows:

Open the directory containing the executable file (assuming the filename is KrillinAI_1.0.0_desktop_macOS_arm64) in Terminal
Execute the following commands sequentially:

1
2
3

sudo xattr -cr ./KrillinAI_1.0.0_desktop_macOS_arm64  
sudo chmod +x ./KrillinAI_1.0.0_desktop_macOS_arm64  
./KrillinAI_1.0.0_desktop_macOS_arm64

[For the non-desktop version, i.e., release files without “desktop” in the name, refer here]
This software is not signed, so after completing the file configuration in the “Basic Steps,” you will need to manually trust the application on macOS. Follow these steps:

Open the terminal and navigate to the directory where the executable file (assuming the file name is KrillinAI_1.0.0_macOS_arm64) is located.
Execute the following commands in sequence:

1
2
3

sudo xattr -rd com.apple.quarantine ./KrillinAI_1.0.0_macOS_arm64
sudo chmod +x ./KrillinAI_1.0.0_macOS_arm64
./KrillinAI_1.0.0_macOS_arm64

This will start the service.

Docker Deployment

This project supports Docker deployment. Please refer to the Docker Deployment Instructions.

If you encounter video download failures, please refer to the Cookie Configuration Instructions to configure your cookie information.

Configuration Help

The quickest and most convenient configuration method:

Select openai for both transcription_provider and llm_provider. In this way, you only need to fill in openai.apikey in the following three major configuration item categories, namely openai, local_model, and aliyun, and then you can conduct subtitle translation. (Fill in app.proxy, model and openai.base_url as per your own situation.)

The configuration method for using the local speech recognition model (macOS is not supported for the time being) (a choice that takes into account cost, speed, and quality):

Fill in fasterwhisper for transcription_provider and openai for llm_provider. In this way, you only need to fill in openai.apikey and local_model.faster_whisper in the following three major configuration item categories, namely openai and local_model, and then you can conduct subtitle translation. The local model will be downloaded automatically. (The same applies to app.proxy and openai.base_url as mentioned above.)

The following usage situations require the configuration of Alibaba Cloud:

If llm_provider is filled with aliyun, it indicates that the large model service of Alibaba Cloud will be used. Consequently, the configuration of the aliyun.bailian item needs to be set up.
If transcription_provider is filled with aliyun, or if the “voice dubbing” function is enabled when starting a task, the voice service of Alibaba Cloud will be utilized. Therefore, the configuration of the aliyun.speech item needs to be filled in.
If the “voice dubbing” function is enabled and local audio files are uploaded for voice timbre cloning at the same time, the OSS cloud storage service of Alibaba Cloud will also be used. Hence, the configuration of the aliyun.oss item needs to be filled in. Configuration Guide: Alibaba Cloud Configuration Instructions

Frequently Asked Questions

Please refer to Frequently Asked Questions

Contribution Guidelines

Do not submit unnecessary files like .vscode, .idea, etc. Please make good use of .gitignore to filter them.
Do not submit config.toml; instead, submit config-example.toml.

Star History

FunASR

Wed, 09 Apr 2025 15:29:01 +0800

modelscope/FunASR

(简体中文|English)

FunASR hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun！

Highlights

FunASR is a fundamental speech recognition toolkit that offers a variety of features, including speech recognition (ASR), Voice Activity Detection (VAD), Punctuation Restoration, Language Models, Speaker Verification, Speaker Diarization and multi-talker ASR. FunASR provides convenient scripts and tutorials, supporting inference and fine-tuning of pre-trained models.
We have released a vast collection of academic and industrial pretrained models on the ModelScope and huggingface, which can be accessed through our Model Zoo. The representative Paraformer-large, a non-autoregressive end-to-end speech recognition model, has the advantages of high accuracy, high efficiency, and convenient deployment, supporting the rapid construction of speech recognition services. For more details on service deployment, please refer to the service deployment document.

What’s new:

2024/10/29: Real-time Transcription Service 1.12 released, The 2pass-offline mode supports the SensevoiceSmal model；(docs);
2024/10/10：Added support for the Whisper-large-v3-turbo model, a multitasking model that can perform multilingual speech recognition, speech translation, and language identification. It can be downloaded from the modelscope, and openai.
2024/09/26: Offline File Transcription Service 4.6, Offline File Transcription Service of English 1.7, Real-time Transcription Service 1.11 released, fix memory leak & Support the SensevoiceSmall onnx model；File Transcription Service 2.0 GPU released, Fix GPU memory leak; (docs);
2024/09/25：keyword spotting models are new supported. Supports fine-tuning and inference for four models: fsmn_kws, fsmn_kws_mt, sanm_kws, sanm_kws_streaming.
2024/07/04：SenseVoice is a speech foundation model with multiple speech understanding capabilities, including ASR, LID, SER, and AED.
2024/07/01: Offline File Transcription Service GPU 1.1 released, optimize BladeDISC model compatibility issues; ref to (docs)
2024/06/27: Offline File Transcription Service GPU 1.0 released, supporting dynamic batch processing and multi-threading concurrency. In the long audio test set, the single-thread RTF is 0.0076, and multi-threads’ speedup is 1200+ (compared to 330+ on CPU); ref to (docs)
2024/05/15：emotion recognition models are new supported. emotion2vec+large，emotion2vec+base，emotion2vec+seed. currently supports the following categories: 0: angry 1: happy 2: neutral 3: sad 4: unknown.
2024/05/15: Offline File Transcription Service 4.5, Offline File Transcription Service of English 1.6, Real-time Transcription Service 1.10 released, adapting to FunASR 1.0 model structure；(docs)

Full Changelog

2024/03/05：Added the Qwen-Audio and Qwen-Audio-Chat large-scale audio-text multimodal models, which have topped multiple audio domain leaderboards. These models support speech dialogue, usage.
2024/03/05：Added support for the Whisper-large-v3 model, a multitasking model that can perform multilingual speech recognition, speech translation, and language identification. It can be downloaded from themodelscope, and openai.
2024/03/05: Offline File Transcription Service 4.4, Offline File Transcription Service of English 1.5，Real-time Transcription Service 1.9 released，docker image supports ARM64 platform, update modelscope；(docs)
2024/01/30：funasr-1.0 has been released (docs)
2024/01/30：emotion recognition models are new supported. model link, modified from repo.
2024/01/25: Offline File Transcription Service 4.2, Offline File Transcription Service of English 1.3 released，optimized the VAD (Voice Activity Detection) data processing method, significantly reducing peak memory usage, memory leak optimization; Real-time Transcription Service 1.7 released，optimizatized the client-side；(docs)
2024/01/09: The Funasr SDK for Windows version 2.0 has been released, featuring support for The offline file transcription service (CPU) of Mandarin 4.1, The offline file transcription service (CPU) of English 1.2, The real-time transcription service (CPU) of Mandarin 1.6. For more details, please refer to the official documentation or release notes(FunASR-Runtime-Windows)
2024/01/03: File Transcription Service 4.0 released, Added support for 8k models, optimized timestamp mismatch issues and added sentence-level timestamps, improved the effectiveness of English word FST hotwords, supported automated configuration of thread parameters, and fixed known crash issues as well as memory leak problems, refer to (docs).
2024/01/03: Real-time Transcription Service 1.6 released，The 2pass-offline mode supports Ngram language model decoding and WFST hotwords, while also addressing known crash issues and memory leak problems, (docs)
2024/01/03: Fixed known crash issues as well as memory leak problems, (docs).
2023/12/04: The Funasr SDK for Windows version 1.0 has been released, featuring support for The offline file transcription service (CPU) of Mandarin, The offline file transcription service (CPU) of English, The real-time transcription service (CPU) of Mandarin. For more details, please refer to the official documentation or release notes(FunASR-Runtime-Windows)
2023/11/08: The offline file transcription service 3.0 (CPU) of Mandarin has been released, adding punctuation large model, Ngram language model, and wfst hot words. For detailed information, please refer to docs.
2023/10/17: The offline file transcription service (CPU) of English has been released. For more details, please refer to (docs).
2023/10/13: SlideSpeech: A large scale multi-modal audio-visual corpus with a significant amount of real-time synchronized slides.
2023/10/10: The ASR-SpeakersDiarization combined pipeline Paraformer-VAD-SPK is now released. Experience the model to get recognition results with speaker information.
2023/10/07: FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec.
2023/09/01: The offline file transcription service 2.0 (CPU) of Mandarin has been released, with added support for ffmpeg, timestamp, and hotword models. For more details, please refer to (docs).
2023/08/07: The real-time transcription service (CPU) of Mandarin has been released. For more details, please refer to (docs).
2023/07/17: BAT is released, which is a low-latency and low-memory-consumption RNN-T model. For more details, please refer to (BAT).
2023/06/26: ASRU2023 Multi-Channel Multi-Party Meeting Transcription Challenge 2.0 completed the competition and announced the results. For more details, please refer to (M2MeT2.0).

Installation

Requirements

1
2
3

python>=3.8
torch>=1.13
torchaudio

Install for pypi

`1`	`pip3 install -U funasr`

Or install from source code

1
2

git clone https://github.com/alibaba/FunASR.git && cd FunASR
pip3 install -e ./

Install modelscope or huggingface_hub for the pretrained models (Optional)

`1`	`pip3 install -U modelscope huggingface_hub`

Model Zoo

FunASR has open-sourced a large number of pre-trained models on industrial data. You are free to use, copy, modify, and share FunASR models under the Model License Agreement. Below are some representative models, for more models please refer to the Model Zoo.

(Note: ⭐ represents the ModelScope model zoo, 🤗 represents the Huggingface model zoo, 🍀 represents the OpenAI model zoo)

Model Name	Task Details	Training Data	Parameters
SenseVoiceSmall (⭐ 🤗 )	multiple speech understanding capabilities, including ASR, ITN, LID, SER, and AED, support languages such as zh, yue, en, ja, ko	300000 hours	234M
paraformer-zh (⭐ 🤗 )	speech recognition, with timestamps, non-streaming	60000 hours, Mandarin	220M
paraformer-zh-streaming ( ⭐ 🤗 )	speech recognition, streaming	60000 hours, Mandarin	220M
paraformer-en ( ⭐ 🤗 )	speech recognition, without timestamps, non-streaming	50000 hours, English	220M
conformer-en ( ⭐ 🤗 )	speech recognition, non-streaming	50000 hours, English	220M
ct-punc ( ⭐ 🤗 )	punctuation restoration	100M, Mandarin and English	290M
fsmn-vad ( ⭐ 🤗 )	voice activity detection	5000 hours, Mandarin and English	0.4M
fsmn-kws ( ⭐ )	keyword spotting，streaming	5000 hours, Mandarin	0.7M
fa-zh ( ⭐ 🤗 )	timestamp prediction	5000 hours, Mandarin	38M
cam++ ( ⭐ 🤗 )	speaker verification/diarization	5000 hours	7.2M
Whisper-large-v3 (⭐ 🍀 )	speech recognition, with timestamps, non-streaming	multilingual	1550 M
Whisper-large-v3-turbo (⭐ 🍀 )	speech recognition, with timestamps, non-streaming	multilingual	809 M
Qwen-Audio (⭐ 🤗 )	audio-text multimodal models (pretraining)	multilingual	8B
Qwen-Audio-Chat (⭐ 🤗 )	audio-text multimodal models (chat)	multilingual	8B
emotion2vec+large (⭐ 🤗 )	speech emotion recongintion	40000 hours	300M

Quick Start

Below is a quick start tutorial. Test audio files (Mandarin, English).

Command-line usage

`1`	`funasr ++model=paraformer-zh ++vad_model="fsmn-vad" ++punc_model="ct-punc" ++input=asr_example_zh.wav`

Notes: Support recognition of single audio file, as well as file list in Kaldi-style wav.scp format: wav_id wav_pat

Speech Recognition (Non-streaming)

SenseVoice

from funasr import AutoModel
from funasr.utils.postprocess_utils import rich_transcription_postprocess

model_dir = "iic/SenseVoiceSmall"

model = AutoModel(
    model=model_dir,
    vad_model="fsmn-vad",
    vad_kwargs={"max_single_segment_time": 30000},
    device="cuda:0",
)

# en
res = model.generate(
    input=f"{model.model_path}/example/en.mp3",
    cache={},
    language="auto",  # "zn", "en", "yue", "ja", "ko", "nospeech"
    use_itn=True,
    batch_size_s=60,
    merge_vad=True,  #
    merge_length_s=15,
)
text = rich_transcription_postprocess(res[0]["text"])
print(text)

Parameter Description:

model_dir: The name of the model, or the path to the model on the local disk.
vad_model: This indicates the activation of VAD (Voice Activity Detection). The purpose of VAD is to split long audio into shorter clips. In this case, the inference time includes both VAD and SenseVoice total consumption, and represents the end-to-end latency. If you wish to test the SenseVoice model’s inference time separately, the VAD model can be disabled.
vad_kwargs: Specifies the configurations for the VAD model. max_single_segment_time: denotes the maximum duration for audio segmentation by the vad_model, with the unit being milliseconds (ms).
use_itn: Whether the output result includes punctuation and inverse text normalization.
batch_size_s: Indicates the use of dynamic batching, where the total duration of audio in the batch is measured in seconds (s).
merge_vad: Whether to merge short audio fragments segmented by the VAD model, with the merged length being merge_length_s, in seconds (s).
ban_emo_unk: Whether to ban the output of the emo_unk token.

Paraformer

from funasr import AutoModel
# paraformer-zh is a multi-functional asr model
# use vad, punc, spk or not as you need
model = AutoModel(model="paraformer-zh",  vad_model="fsmn-vad",  punc_model="ct-punc", 
                  # spk_model="cam++", 
                  )
res = model.generate(input=f"{model.model_path}/example/asr_example.wav", 
                     batch_size_s=300, 
                     hotword='魔搭')
print(res)

Note: hub: represents the model repository, ms stands for selecting ModelScope download, hf stands for selecting Huggingface download.

Speech Recognition (Streaming)

from funasr import AutoModel

chunk_size = [0, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms
encoder_chunk_look_back = 4 #number of chunks to lookback for encoder self-attention
decoder_chunk_look_back = 1 #number of encoder chunks to lookback for decoder cross-attention

model = AutoModel(model="paraformer-zh-streaming")

import soundfile
import os

wav_file = os.path.join(model.model_path, "example/asr_example.wav")
speech, sample_rate = soundfile.read(wav_file)
chunk_stride = chunk_size[1] * 960 # 600ms

cache = {}
total_chunk_num = int(len((speech)-1)/chunk_stride+1)
for i in range(total_chunk_num):
    speech_chunk = speech[i*chunk_stride:(i+1)*chunk_stride]
    is_final = i == total_chunk_num - 1
    res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size, encoder_chunk_look_back=encoder_chunk_look_back, decoder_chunk_look_back=decoder_chunk_look_back)
    print(res)

Note: chunk_size is the configuration for streaming latency. [0,10,5] indicates that the real-time display granularity is 10*60=600ms, and the lookahead information is 5*60=300ms. Each inference input is 600ms (sample points are 16000*0.6=960), and the output is the corresponding text. For the last speech segment input, is_final=True needs to be set to force the output of the last word.

More Examples

Voice Activity Detection (Non-Streaming)

from funasr import AutoModel

model = AutoModel(model="fsmn-vad")
wav_file = f"{model.model_path}/example/vad_example.wav"
res = model.generate(input=wav_file)
print(res)

Note: The output format of the VAD model is: [[beg1, end1], [beg2, end2], ..., [begN, endN]], where begN/endN indicates the starting/ending point of the N-th valid audio segment, measured in milliseconds.

Voice Activity Detection (Streaming)

from funasr import AutoModel

chunk_size = 200 # ms
model = AutoModel(model="fsmn-vad")

import soundfile

wav_file = f"{model.model_path}/example/vad_example.wav"
speech, sample_rate = soundfile.read(wav_file)
chunk_stride = int(chunk_size * sample_rate / 1000)

cache = {}
total_chunk_num = int(len((speech)-1)/chunk_stride+1)
for i in range(total_chunk_num):
    speech_chunk = speech[i*chunk_stride:(i+1)*chunk_stride]
    is_final = i == total_chunk_num - 1
    res = model.generate(input=speech_chunk, cache=cache, is_final=is_final, chunk_size=chunk_size)
    if len(res[0]["value"]):
        print(res)

Note: The output format for the streaming VAD model can be one of four scenarios:

[[beg1, end1], [beg2, end2], .., [begN, endN]]：The same as the offline VAD output result mentioned above.
[[beg, -1]]：Indicates that only a starting point has been detected.
[[-1, end]]：Indicates that only an ending point has been detected.
[]：Indicates that neither a starting point nor an ending point has been detected.

The output is measured in milliseconds and represents the absolute time from the starting point.

Punctuation Restoration

from funasr import AutoModel

model = AutoModel(model="ct-punc")
res = model.generate(input="那今天的会就到这里吧 happy new year 明年见")
print(res)

Timestamp Prediction

from funasr import AutoModel

model = AutoModel(model="fa-zh")
wav_file = f"{model.model_path}/example/asr_example.wav"
text_file = f"{model.model_path}/example/text.txt"
res = model.generate(input=(wav_file, text_file), data_type=("sound", "text"))
print(res)

Speech Emotion Recognition

from funasr import AutoModel

model = AutoModel(model="emotion2vec_plus_large")

wav_file = f"{model.model_path}/example/test.wav"

res = model.generate(wav_file, output_dir="./outputs", granularity="utterance", extract_embedding=False)
print(res)

More usages ref to docs, more examples ref to demo

Export ONNX

Command-line usage

`1`	`funasr-export ++model=paraformer ++quantize=false ++device=cpu`

Python

from funasr import AutoModel

model = AutoModel(model="paraformer", device="cpu")

res = model.export(quantize=False)

Test ONNX

# pip3 install -U funasr-onnx
from funasr_onnx import Paraformer
model_dir = "damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch"
model = Paraformer(model_dir, batch_size=1, quantize=True)

wav_path = ['~/.cache/modelscope/hub/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/example/asr_example.wav']

result = model(wav_path)
print(result)

More examples ref to demo

Deployment Service

FunASR supports deploying pre-trained or further fine-tuned models for service. Currently, it supports the following types of service deployment:

File transcription service, Mandarin, CPU version, done
The real-time transcription service, Mandarin (CPU), done
File transcription service, English, CPU version, done
File transcription service, Mandarin, GPU version, in progress
and more.

For more detailed information, please refer to the service deployment documentation.

Community Communication

If you encounter problems in use, you can directly raise Issues on the github page.

You can also scan the following DingTalk group to join the community group for communication and discussion.

DingTalk group

Contributors

The contributors can be found in contributors list

License

This project is licensed under The MIT License. FunASR also contains various third-party components and some code modified from other repos under other open source licenses. The use of pretraining model is subject to model license

Citations

@inproceedings{gao2023funasr,
  author={Zhifu Gao and Zerui Li and Jiaming Wang and Haoneng Luo and Xian Shi and Mengzhe Chen and Yabin Li and Lingyun Zuo and Zhihao Du and Zhangyu Xiao and Shiliang Zhang},
  title={FunASR: A Fundamental End-to-End Speech Recognition Toolkit},
  year={2023},
  booktitle={INTERSPEECH},
}
@inproceedings{An2023bat,
  author={Keyu An and Xian Shi and Shiliang Zhang},
  title={BAT: Boundary aware transducer for memory-efficient and low-latency ASR},
  year={2023},
  booktitle={INTERSPEECH},
}
@inproceedings{gao22b_interspeech,
  author={Zhifu Gao and ShiLiang Zhang and Ian McLoughlin and Zhijie Yan},
  title={Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition},
  year=2022,
  booktitle={Proc. Interspeech 2022},
  pages={2063--2067},
  doi={10.21437/Interspeech.2022-9996}
}
@inproceedings{shi2023seaco,
  author={Xian Shi and Yexin Yang and Zerui Li and Yanni Chen and Zhifu Gao and Shiliang Zhang},
  title={SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Ability},
  year={2023},
  booktitle={ICASSP2024}
}

Speech Recognition on Producthunt daily

VibeVoice

microsoft/VibeVoice

🎙️ VibeVoice: Open-Source Frontier Voice AI

📰 News

Overview

Models

1. 📖 VibeVoice-ASR - Long-form Speech Recognition

2. 🎙️ VibeVoice-TTS - Long-form Multi-speaker TTS

3. ⚡ VibeVoice-Streaming - Real-time Streaming TTS

Contributing

⚠️ Risks and Limitations

Star History

vosk-api

alphacep/vosk-api

Vosk Speech Recognition Toolkit

Documentation

KrillinAI

krillinai/KrillinAI

AI Audio&Video Translation and Dubbing Tool

📢 New Release for Win & Mac Desktop Version – Welcome to Test and Provide Feedback

Overview

Key Features:

Showcase

Subtitle Translation

Dubbing

Portrait

🔍 Speech Recognition Support

🚀 Large Language Model Support

🌍 Language Support

Interface Preview

🚀 Quick Start

Basic Steps

To: macOS Users

Docker Deployment

Cookie Configuration Instructions

Configuration Help

Frequently Asked Questions

Contribution Guidelines

Star History

FunASR

modelscope/FunASR

Highlights

What’s new:

Installation

Model Zoo

Quick Start

Command-line usage

Speech Recognition (Non-streaming)

SenseVoice

Paraformer

Speech Recognition (Streaming)

Voice Activity Detection (Non-Streaming)

Voice Activity Detection (Streaming)

Punctuation Restoration

Timestamp Prediction

Speech Emotion Recognition

Export ONNX

Command-line usage

Python

Test ONNX

Deployment Service

Community Communication

Contributors

License

Citations