Microsoft targets ultra-realistic voice agents with its first streaming transcription model
The first design issue is turn-taking. A stream that emits provisional transcript hypotheses in a few hundred milliseconds is only valuable if the orchestration layer knows when to act on partial text and when to wait for a final transcript. Trigger too early and the agent may interrupt, misroute intent or generate unnecessary downstream inference cost; wait too long and the experience stops feeling conversational. Teams will need confidence thresholds, cancellation handling, barge-in support and policies for revising an in-flight response when the transcript changes.
The second issue is end-to-end latency budgeting. The quoted transcription speed is only one component in the path. WebSocket handling, language detection, prompt assembly, reasoning inference, text-to-speech generation and client playback all add delay. In practice, ultra-realistic voice agents will depend less on any single model benchmark than on whether developers can pipeline these stages and keep network hops minimal.
There is also a cost-control angle. Microsoft’s separate models make it easier to mix premium and lower-latency tiers by use case, but that increases routing complexity. Enterprise teams should evaluate whether they need streaming transcription for every interaction, or only for scenarios such as live captioning, interruption-heavy support flows or hands-free assistants where partial-input processing has measurable user value.
Microsoft Corp. has expanded its MAI artificial intelligence model family with its first streaming transcription model, debuting alongside two others focused on text-to-speech. They’re designed for developers who want to build voice agents that can listen to people’s voices and reply instantly, similar to how humans talk to one another.
The most consequential of the three is MAI-Transcribe-2-Streaming, which the company said accepts human speech via a WebSocket, transforming it into a transcript that’s continuously updated as the person keeps on talking. Once the person has finished speaking, it will confirm that the transcript is final. In this way, the model can power applications that display live captions or start processing a user’s request before they’ve finished saying it, Microsoft said.
MAI-Transcribe-2-Streaming is listed on Microsoft’s Vercel AI Gateway and priced at $0.54 per audio hour, the company said. It supports more than 60 languages, and can detect which one someone is speaking automatically. It delivers its first transcript hypotheses within 320 milliseconds on average, although Microsoft said it cannot guarantee this kind of performance in every scenario, because speed of response is also determined by the network connection and the AI system that generates the reply.
Microsoft isn’t new to transcription models. Last month, it debuted the MAI-Transcribe-2 model with a price of $0.10 per audio hour, which is more than five-times cheaper than the Streaming variant. That’s mainly because the Streaming model does a lot more processing, continuously returning provisional results while the audio is still arriving. The regular MAI-Transcribe-2 model simply waits for the speaker to finish what they’re saying before processing begins.
Moving away from transcription, there are two models focused on generating speech from text, which are equally important for conversational AI agents. They include MAI-Voice-2.1, which is said to be the best choice for more expressive and higher fidelity outputs, and MAI-Voice-2.1-Flash, which tones down those attributes in favor of a speedier response and lower cost. According to the Vercel AI Gateway listings, MAI-Voice-2.1 is priced at $22 per million characters, while the Flash version costs $15 per million characters. Both of the text-to-speech models support 23 languages, Microsoft said.
The new releases demonstrate that Microsoft is accelerating its transition away from model providers such as OpenAI Group PBC and Anthropic PBC, despite being a major investor in both of those companies. In July, it was reported that Microsoft AI Chief Executive Mustafa Suleyman was becoming increasingly concerned about the costs associated with OpenAI’s and Anthropic’s powerful frontier models. In response, he instructed Microsoft’s AI researchers to double down on the MAI model family, with the goal of eventually using those models to power its Copilot agents in platforms such as Excel and Outlook. “We pay a lot of money to Anthropic, so our goal is to reduce and ultimately eliminate that cost,” Suleyman told Bloomberg in an interview.
With the launch of the new models, Microsoft has everything it needs to build powerful voice AI agents that can converse with their human users in a natural, humanlike way. Voice agents need to be able to do three things to function – they must be able to recognize and understand what someone is saying to them, then decide what to do based on what was said, and finally generate an audible response.
MAI-Transcribe-2-Streaming takes care of the first problem, and the MAI-Voice models solve the third. What sits in the middle is Microsoft’s powerful reasoning model, Mai-Thinking-1, which is a standard large language model that reads through the transcripts of what was said to decide what the agent should do. By using three separate models, developers have much more control over the quality, latency and costs of their voice agents.
Microsoft targets ultra-realistic voice agents with its first streaming transcription model
Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.