Qwen3-Next 80B-A3B (Thinking)

Qwen3-Next 80B-A3B Thinking is built on a specialized architecture designed to maximize efficiency in both training and inference. It utilizes a high-sparsity Mixture-of-Experts framework that activates only 3 billion parameters out of its 80-billion total, significantly reducing the computational cost per token while maintaining broad model capacity. To handle long-context tasks, the model replaces standard attention mechanisms with a hybrid system combining Gated DeltaNet and Gated Attention. This design, paired with multi-token prediction, allows the model to achieve high throughput and effective context modeling, making it a powerful tool for complex, long-form reasoning tasks.

The development of this model involved rigorous stability optimizations, including the use of zero-centered and weight-decayed layer normalization to ensure robust performance during pre-training and post-training. By leveraging specialized reinforcement learning techniques, the team successfully addressed the stability challenges typically associated with combining hybrid attention and high-sparsity MoE structures. These advancements result in a model that delivers significant performance gains over previous dense iterations, offering a highly efficient solution for users who require deep reasoning capabilities and rapid inference speeds across extensive document contexts.