导航|地址|刷新
GitHub - deepseek-ai/DeepSeek-V3
















!DOCTYPE html>



Skip to content


Navigation Menu
Toggle navigation










Sign in
Appearance settings









Platform


AI CODE CREATION

GitHub CopilotWrite better code with AI



GitHub Copilot appDirect agents from issue to merge



MCP RegistryIntegrate external tools





DEVELOPER WORKFLOWS

ActionsAutomate any workflow



CodespacesInstant dev environments



IssuesPlan and track work



Code ReviewManage code changes



Code QualityEnforce quality at merge





APPLICATION SECURITY

GitHub Advanced SecurityFind and fix vulnerabilities



Code securitySecure your code as you build



Secret protectionStop leaks before they start





EXPLORE
Why GitHub

Documentation

Blog

Changelog

Marketplace



View all features





Solutions


BY COMPANY SIZE
Enterprises

Small and medium teams

Startups

Nonprofits




BY USE CASE
App Modernization

DevSecOps

DevOps

CI/CD

View all use cases




BY INDUSTRY
Healthcare

Financial services

Manufacturing

Government

View all industries



View all solutions





Resources


EXPLORE BY TOPIC
AI

Software Development

DevOps

Security

View all topics




EXPLORE BY TYPE
Customer stories

Events & webinars

Ebooks & reports

Business insights

GitHub Skills




SUPPORT & SERVICES
Documentation

Customer support

Community forum

Trust center

Partners



View all resources





Open Source


COMMUNITY

GitHub SponsorsFund open source developers





PROGRAMS
Security Lab

Maintainer Community

Accelerator

GitHub Stars

Archive Program




REPOSITORIES
Topics

Trending

Collections







Enterprise


ENTERPRISE SOLUTIONS

Enterprise platformAI-powered developer platform





AVAILABLE ADD-ONS

GitHub Advanced SecurityEnterprise-grade security features



Copilot for BusinessEnterprise-grade AI features



Premium SupportEnterprise-grade 24/7 support







Pricing





Search or jump to...



Search code, repositories, users, issues, pull requests...




Search








Clear








Search syntax tips














Provide feedback







We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted

Cancel Submit feedback




Saved searches

Use saved searches to filter your results more quickly












Name

Query

To see all available qualifiers, see our documentation.







Cancel Create saved search


Sign in
Sign up
Appearance settings


Resetting focus




You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert








{{ message }}











Uh oh!


There was an error while loading. Please reload this page.









deepseek-ai /DeepSeek-V3Public



Notifications You must be signed in to change notification settings

Fork 16.7k


Star 104k






Code

Issues160

Pull requests62

Actions

Projects

Security and quality0

Insights

Additional navigation options



Code

Issues

Pull requests

Actions

Projects

Security and quality

Insights

































main





BranchesTags










Go to file

Code
Open more actions menu







Folders and files

Name	Name	Last commit message	Last commit date
Latest commit History 73 Commits 73 Commits
.github	.github
figures	figures
inference	inference
.gitignore	.gitignore
LICENSE-CODE	LICENSE-CODE
LICENSE-MODEL	LICENSE-MODEL
README.md	README.md
README_WEIGHTS.md	README_WEIGHTS.md
View all files

Repository files navigation



README

MIT license

License



More items








------












Paper Link👁️


Table of Contents


Introduction

Model Summary

Model Downloads

Evaluation Results

Chat Website & API Platform

How to Run Locally

License

Citation

Contact


1. Introduction


We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks.




2. Model Summary


------

Architecture: Innovative Load Balancing Strategy and Training Objective

On top of the efficient architecture of DeepSeek-V2, we pioneer an auxiliary-loss-free strategy for load balancing, which minimizes the performance degradation that arises from encouraging load balancing.

We investigate a Multi-Token Prediction (MTP) objective and prove it beneficial to model performance. It can also be used for speculative decoding for inference acceleration.

------

Pre-Training: Towards Ultimate Training Efficiency

We design an FP8 mixed precision training framework and, for the first time, validate the feasibility and effectiveness of FP8 training on an extremely large-scale model.

Through co-design of algorithms, frameworks, and hardware, we overcome the communication bottleneck in cross-node MoE training, nearly achieving full computation-communication overlap.
This significantly enhances our training efficiency and reduces the training costs, enabling us to further scale up the model size without additional overhead.

At an economical cost of only 2.664M H800 GPU hours, we complete the pre-training of DeepSeek-V3 on 14.8T tokens, producing the currently strongest open-source base model. The subsequent training stages after pre-training require only 0.1M GPU hours.

------

Post-Training: Knowledge Distillation from DeepSeek-R1

We introduce an innovative methodology to distill reasoning capabilities from the long-Chain-of-Thought (CoT) model, specifically from one of the DeepSeek R1 series models, into standard LLMs, particularly DeepSeek-V3. Our pipeline elegantly incorporates the verification and reflection patterns of R1 into DeepSeek-V3 and notably improves its reasoning performance. Meanwhile, we also maintain a control over the output style and length of DeepSeek-V3.

------


3. Model Downloads

Model	#Total Params	#Activated Params	Context Length	Download
DeepSeek-V3-Base	671B	37B	128K	🤗 Hugging Face
DeepSeek-V3	671B	37B	128K	🤗 Hugging Face

Note

The total size of DeepSeek-V3 models on Hugging Face is 685B, which includes 671B of the Main Model weights and 14B of the Multi-Token Prediction (MTP) Module weights.


To ensure optimal performance and flexibility, we have partnered with open-source communities and hardware vendors to provide multiple ways to run the model locally. For step-by-step guidance, check out Section 6: How_to Run_Locally.

For developers looking to dive deeper, we recommend exploring README_WEIGHTS.md for details on the Main Model weights and the Multi-Token Prediction (MTP) Modules. Please note that MTP support is currently under active development within the community, and we welcome your contributions and feedback.


4. Evaluation Results



Base Model



Standard Benchmarks

	Benchmark (Metric)	# Shots	DeepSeek-V2	Qwen2.5 72B	LLaMA3.1 405B	DeepSeek-V3
	Architecture	-	MoE	Dense	Dense	MoE
	# Activated Params	-	21B	72B	405B	37B
	# Total Params	-	236B	72B	405B	671B
English	Pile-test (BPB)	-	0.606	0.638	0.542	0.548
	BBH (EM)	3-shot	78.8	79.8	82.9	87.5
	MMLU (Acc.)	5-shot	78.4	85.0	84.4	87.1
	MMLU-Redux (Acc.)	5-shot	75.6	83.2	81.3	86.2
	MMLU-Pro (Acc.)	5-shot	51.4	58.3	52.8	64.4
	DROP (F1)	3-shot	80.4	80.6	86.0	89.0
	ARC-Easy (Acc.)	25-shot	97.6	98.4	98.4	98.9
	ARC-Challenge (Acc.)	25-shot	92.2	94.5	95.3	95.3
	HellaSwag (Acc.)	10-shot	87.1	84.8	89.2	88.9
	PIQA (Acc.)	0-shot	83.9	82.6	85.9	84.7
	WinoGrande (Acc.)	5-shot	86.3	82.3	85.2	84.9
	RACE-Middle (Acc.)	5-shot	73.1	68.1	74.2	67.1
	RACE-High (Acc.)	5-shot	52.6	50.3	56.8	51.3
	TriviaQA (EM)	5-shot	80.0	71.9	82.7	82.9
	NaturalQuestions (EM)	5-shot	38.6	33.2	41.5	40.0
	AGIEval (Acc.)	0-shot	57.5	75.8	60.6	79.6
Code	HumanEval (Pass@1)	0-shot	43.3	53.0	54.9	65.2
	MBPP (Pass@1)	3-shot	65.0	72.6	68.4	75.4
	LiveCodeBench-Base (Pass@1)	3-shot	11.6	12.9	15.5	19.4
	CRUXEval-I (Acc.)	2-shot	52.5	59.1	58.5	67.3
	CRUXEval-O (Acc.)	2-shot	49.8	59.9	59.9	69.8
Math	GSM8K (EM)	8-shot	81.6	88.3	83.5	89.3
	MATH (EM)	4-shot	43.4	54.4	49.0	61.6
	MGSM (EM)	8-shot	63.6	76.2	69.9	79.8
	CMath (EM)	3-shot	78.7	84.5	77.3	90.7
Chinese	CLUEWSC (EM)	5-shot	82.0	82.5	83.0	82.7
	C-Eval (Acc.)	5-shot	81.4	89.2	72.5	90.1
	CMMLU (Acc.)	5-shot	84.0	89.5	73.7	88.8
	CMRC (EM)	1-shot	77.4	75.8	76.0	76.3
	C3 (Acc.)	0-shot	77.4	76.7	79.7	78.6
	CCPM (Acc.)	0-shot	93.0	88.5	78.6	92.0
Multilingual	MMMLU-non-English (Acc.)	5-shot	64.0	74.8	73.8	79.4

Note

Best results are shown in bold. Scores with a gap not exceeding 0.3 are considered to be at the same level. DeepSeek-V3 achieves the best performance on most benchmarks, especially on math and code tasks. For more evaluation details, please check our paper.



Context Window




Evaluation results on the Needle In A Haystack (NIAH) tests. DeepSeek-V3 performs well across all context window lengths up to 128K.


Chat Model



Standard Benchmarks (Models larger than 67B)




…(内容过长已截断)

------
导航页 打开网址