Mini Max teases upcoming M3 model with new sparse attention mechanism and 15.6X long-context response speed boost | Venture Beat

Overview

Mini Max teases upcoming M3 model with new sparse attention mechanism and 15.6X long-context response speed boost

Credit: Venture Beat made with Google Gemini 3 Pro Image

Details

Credit: Venture Beat made with Google Gemini 3 Pro Image

Among the many Chinese AI companies and laboratories vying for market share and attention (no pun intended) on the global marketplace, Mini Max stands out for its commitment to providing frontier-level intelligence across a range of modalities, including text, coding, and video (through its Hailuo model series) — often under permissive, enterprise-friendly, standard open source licenses.

Now, Mini Max is again raising the eyebrows of AI power users and developers around the world by releasing a new, in-depth technical report on the making of its popular M2 series of language models (M2, M2.5, and M2.7) shedding light on its numerous engineering innovations and clever approaches — while the company and its leaders also teased a whole new sparse attention approach for its upcoming Mini Max M3 series of models, which it says yields up to 15.6 times faster decoding (or LLM response) speed by adopting a custom sub-quadratic framework. In so doing, Mini Max has designed M3 to make ultra-long-context AI agent deployment economically viable.

The M2 report is noteworthy for any enterprise working with AI models, and especially those looking to fine-tune and train their own in-house. After all, Mini Max's M2 series models often achieved top benchmarks in the world for open source AI performance when they were released.

While the title has since been eclipsed by several other Chinese labs including Deep Seek and Xiaomi, Mini Max's new report offers a blueprint that can be used to improve AI model and agent performance by enterprises around the world.

As Adina Yakup of Hugging Face observed online, "Beyond the benchmarks, they’ve done some really solid work on Mo E efficiency and agent oriented design. Excited to see where M3 goes next!"

The core technical architecture of the M2 series relies on a sparse Mixture-of-Experts (Mo E) decoder-only Transformer layout used by numerous other state-of-the-art LLMs.

The foundational backbone houses 229.9 billion total parameters, yet maintains a remarkably lean operational footprint by activating just 9.8 billion parameters per token across 256 fine-grained experts.

To optimize routing and avoid standard load-balancing issues, however, Mini Max implemented sigmoid gating paired with learnable, expert-specific bias terms, heavily reducing reliance on restrictive auxiliary losses.

The most definitive engineering decision documented in the M2 paper was the strict adherence to full multi-head attention with Grouped Query Attention (GQA) across all 62 layers.

In large language models, "quadratic scaling" refers to the computationally expensive reality of standard full attention mechanisms, where every token in a sequence must mathematically connect to every other token. To use a real-world analogy, it is akin to attending a networking event and being forced to have a deep conversation with every single person in the room while simultaneously monitoring all other ongoing conversations.

While this approach yields incredibly thorough context, the processing power and memory required explode at the square of the input length, creating a severe hardware bottleneck as models attempt to ingest hundreds of thousands of words.

"Sub-quadratic" scaling introduces architectural shortcuts designed to bypass this exponential computational load. Instead of mapping every possible connection, sub-quadratic methods—such as Sliding Window Attention or compressed linear attention—might only analyze a localized window of nearby words or generate a compressed summary of the broader text.

These efficient methods drastically reduce hardware costs and allow models to process massive documents at high speeds, but they historically introduce severe trade-offs in accuracy, often causing the AI to miss the "big picture" or lose track of distant context.

This mathematical dilemma defines the architectural evolution from Mini Max's M2 to its upcoming M3 series. During M2's development, researchers rigorously tested sub-quadratic shortcuts but found they crippled the model's "multi-hop reasoning"—its ability to connect disparate clues across a long document—forcing the team to absorb the massive computational cost of full quadratic attention to maintain frontier-level intelligence.

Indeed, they aggressively benchmarked efficient attention alternatives during pre-training but intentionally threw them out. They experimented extensively with hybrid setups, interleaving full attention with sub-quadratic architectures like Lightning Attention or hybrid Sliding Window Attention (SWA) configurations.

The empirical results were definitive: at a larger scale, linear and windowed attention variants exhibited severe reasoning deficits.

On evaluations exceeding 32K context windows, SWA variants performed significantly worse than full attention, dropping from a baseline score of 90.0 to 72.0 on the RULER 128K complex word extraction task.

Sub-quadratic configurations proved prone to memory-bound constraints during training, lacked native prefix caching support, and failed to smoothly align with Multi-Token Prediction (MTP) modules used for speculative decoding. Full attention was deemed necessary to preserve multi-hop reasoning capability.

However, recognizing that physical hardware limits cannot sustain quadratic scaling indefinitely, Mini Max is designing the M3 series around a novel sub-quadratic framework to finally deliver both high-speed processing and uncompromised reasoning.

Mini Max Sparse Attention (MSA) and sub-quadratic scaling incoming

The upcoming Mini Max-M3 breaks away from the compute-heavy constraints of its predecessor. As disclosed by Mini Max’s engineering team under the banner "Something BIG is coming," M3 introduces "Mini Max Sparse Attention" (MSA).

Unlike Deep Seek’s Multi-head Latent Attention (MLA), which compresses keys and values into a low-dimensional latent space, MSA operates on a standard GQA backbone but utilizes block-level selection on real, uncompressed Key-Values.

Elie Bakouch at AI training infrastructure and platform lab Prime Intellect posted on X noting that the main changes feature "block level selection like in CSA but attention is done on the real KV, not in [compressed space]."

This solves the precision loss and prefix-caching obstacles noted in the M2 paper. By filtering and selecting block-level sequences dynamically, MSA delivers an architectural leap: early hardware profiling indicates a 9.7x speedup in prefilling latency and a massive 15.6x speedup during decoding phases at a 1-million token sequence length compared to the full-attention M2 architecture.

To understand why a speedup in the "decoding phase" is so significant, it helps to break down how an AI actually reads and writes information. When you interact with an AI, the processing happens in two distinct steps: prefilling and decoding.

When you hand an AI a prompt—whether it’s a short sentence or a massive 1,000-page document—it processes that entire chunk of text all at once in parallel, known as "prefilling." It essentially "reads" the input in one big gulp to build its initial understanding and establish context.

In order to generate a response, the AI must enter a "decoding phase." To predict the first word of its response, it looks at the prompt. To predict the second word, it has to look at the prompt plus the first word. To predict the hundredth word, it must recalculate the context of the prompt and the previous 99 words it just wrote. So the response actually becomes harder to generate as it goes on, with the end requiring a full review of all prior parts.

For a layperson, imagine reading a dense legal brief (prefilling) and then being forced to write a summary report where, before writing every single new word, you must rapidly reread the entire brief plus everything you've written so far to ensure your next word makes sense (decoding).

Because the AI must constantly and repetitively look backward to generate each new step forward, the decoding phase is the most severe computational bottleneck in generating text. It is why AI models often type out their answers word-by-word, and why they slow down significantly as conversations get longer.

Therefore, when the passage states the new architecture achieves a massive 15.6x speedup during the decoding phase at a 1-million token sequence length, it means the model has found a structural shortcut to generate its answer—token by token—nearly 16 times faster. It directly solves the exact bottleneck that normally makes AI chatbots freeze or stutter when handling massive amounts of information.

The evolution of the Mini Max M series and the creation of 'Forge'

On a product level, Mini Max has consistently evolved its models from simple text generation interfaces into autonomous workers.

The M2 series pioneered an "interleaved thinking" protocol where the model alternates between natural-language planning traces and explicit tool invocations inside a single trajectory. Rather than dropping the intermediate chain-of-thought blocks between execution turns, M2 appends the full thinking history directly into the conversation context. This planning persistence prevents state drift, allowing the model to recover gracefully from runtime errors and revise its strategies based on environment feedback.

To train these long-horizon workflows, Mini Max built "Forge," a scalable agent-native reinforcement learning system. Forge decouples execution into three independent modules—the Agent Side, the middleware abstraction layer (Gateway Server and Data Pool), and the Training/Inference engines.

As Mini Max engineer Olive Song explained on the Thursd AI podcast, "What we realized is that there's a lot of potential with a small model like this if we train reinforcement learning on it with a large amount of environments and agents... But it's not a very easy thing to do," adding that this environmental training was where the team spent a significant portion of their development timeline. To absorb the extreme trajectory-length variance common in multi-step agent environments, Forge implements two vital engineering solutions:

Windowed FIFO Scheduling: A training scheduler that maps a sliding window over the generation queue. It permits greedy, high-throughput fetching of completed tasks within the window to prevent cluster idle time, while strictly enforcing FIFO boundaries to maintain distributional stability and avoid gradient oscillation.

Windowed FIFO Scheduling: A training scheduler that maps a sliding window over the generation queue. It permits greedy, high-throughput fetching of completed tasks within the window to prevent cluster idle time, while strictly enforcing FIFO boundaries to maintain distributional stability and avoid gradient oscillation.

Prefix Tree Merging: An optimization that restructures batch training into tree computation. Completions sharing identical conversation prefixes are calculated exactly once in the forward pass before branching. This eliminates redundant calculations, generating up to a 40x training speedup with zero approximation error.

Prefix Tree Merging: An optimization that restructures batch training into tree computation. Completions sharing identical conversation prefixes are calculated exactly once in the forward pass before branching. This eliminates redundant calculations, generating up to a 40x training speedup with zero approximation error.

This reinforcement infrastructure directly spawned the M2.7 checkpoint, moving the series toward "self-evolution". Operating inside an automated agent harness, M2.7 functions as an independent machine learning engineer. The model profiles its own active training runs, diagnoses anomalies, reads logs, and automatically modifies its own codebase and configurations.

According to Mini Max, M2.7 successfully handled between 30% and 50% of its own development workflow.

On Open AI’s rigorous MLE Bench Lite suite, which tests autonomous ML research capability, M2.7 achieved a 66.6% medal rate across independent 24-hour trials, effectively tying Google’s closed-weight Gemini 3.1 Pro.

The continuous cadence from M2 to M2.5, which famously completed 30% of internal tasks and 80% of newly committed code at Mini Max HQ, underlines a broader vision.

As the Mini Max team noted during that phase of deployment, "we believe that M2.5 provides virtually limitless possibilities for the development and operation of agents in the economy."

With the technical report codifying the M2 generation's successes and the MSA tech blog on the horizon, Mini Max is signaling that the next frontier of AI is explicitly about translating a mini-activation footprint into maximum real-world intelligence.

Deep insights for enterprise AI, data, and security leaders

By submitting your email, you agree to our Terms and Privacy Notice.

Key Takeaways

• Mini Max teases upcoming M3 model with new sparse attention mechanism and 15

• Credit: Venture Beat made with Google Gemini 3 Pro Image

• Credit: Venture Beat made with Google Gemini 3 Pro Image

• Among the many Chinese AI companies and laboratories vying for market share and attention (no pun intended) on the global marketplace, Mini Max stands out for its commitment to providing frontier-level intelligence across a range of modalities, including text, coding, and video (through its Hailuo model series) — often under permissive, enterprise-friendly, standard open source licenses

• Now, Mini Max is again raising the eyebrows of AI power users and developers around the world by releasing a new, in-depth technical report on the making of its popular M2 series of language models (M2, M2