DeepSeek V3 is built as a Mixture-of-Experts model with 671 billion total parameters but only 37 billion activated per token, a design that lets the model handle massive capacity while keeping inference efficient. The architecture relies on Multi-head Latent Attention (MLA) and DeepSeekMoE layouts that were refined through DeepSeek-V2, and it introduces an auxiliary-loss-free strategy for load balancing that helps distribute work across experts more naturally. The multi-token prediction training objective also allows the model to predict several tokens at once rather than one after another, which can improve both generation quality and throughput. The goal was to build a model that matches or exceeds the performance of leading closed-source systems while remaining practical to train and deploy.
The model was pre-trained on 14.8 trillion diverse and high-quality tokens, then passed through Supervised Fine-Tuning and Reinforcement Learning stages to fully develop its capabilities. Training proved remarkably efficient, requiring only 2.788 million H800 GPU hours, and the process stayed stable throughout with no irrecoverable loss spikes or rollbacks needed. Evaluation results show DeepSeek V3 outperforming other open-source models and reaching performance levels comparable to leading closed-source systems. Public model checkpoints are available on GitHub, making it a practical choice for developers who want to run or study the model directly rather than relying solely on API access.
