DeepSeek is a Chinese AI business "devoted to making AGI a truth" and open-sourcing all its models. They began in 2023, however have been making waves over the past month or two, and specifically this previous week with the release of their 2 latest reasoning models: DeepSeek-R1-Zero and the more innovative DeepSeek-R1, also referred to as DeepSeek Reasoner.


They've launched not only the designs however also the code and evaluation prompts for public use, in addition to an in-depth paper describing their method.


Aside from developing 2 highly performant models that are on par with OpenAI's o1 model, the paper has a lot of valuable details around reinforcement knowing, chain of idea reasoning, timely engineering with reasoning designs, and more.


We'll start by focusing on the training procedure of DeepSeek-R1-Zero, which distinctively relied solely on support knowing, rather of conventional monitored knowing. We'll then carry on to DeepSeek-R1, how it's thinking works, and some prompt engineering finest practices for thinking designs.


Hey everybody, Dan here, co-founder of PromptHub. Today, we're diving into DeepSeek's newest design release and comparing it with OpenAI's thinking models, particularly the A1 and A1 Mini designs. We'll explore their training procedure, thinking capabilities, and some essential insights into timely engineering for thinking models.


DeepSeek is a Chinese-based AI company dedicated to open-source advancement. Their current release, the R1 reasoning design, is groundbreaking due to its open-source nature and innovative training methods. This consists of open access to the models, triggers, and research documents.


Released on January 20th, DeepSeek's R1 achieved impressive performance on numerous criteria, equaling OpenAI's A1 models. Notably, they also launched a precursor design, R10, which works as the foundation for R1.


Training Process: R10 to R1


R10: This design was trained solely utilizing support learning without monitored fine-tuning, making it the very first open-source model to achieve high efficiency through this method. Training involved:


- Rewarding right responses in deterministic jobs (e.g., math issues).
- Encouraging structured reasoning outputs using templates with "" and "" tags


Through thousands of versions, R10 developed longer thinking chains, self-verification, and even reflective behaviors. For example, during training, the model showed "aha" moments and self-correction behaviors, which are unusual in traditional LLMs.


R1: Building on R10, R1 added numerous enhancements:


- Curated datasets with long Chain of Thought examples.
- Incorporation of R10-generated reasoning chains.
- Human preference alignment for refined reactions.
- Distillation into smaller sized designs (LLaMA 3.1 and 3.3 at numerous sizes).


Performance Benchmarks


DeepSeek's R1 design performs on par with OpenAI's A1 models throughout lots of thinking criteria:


Reasoning and Math Tasks: R1 rivals or outshines A1 models in accuracy and depth of reasoning.
Coding Tasks: A1 designs normally perform much better in LiveCode Bench and CodeForces tasks.
Simple QA: R1 frequently outmatches A1 in structured QA jobs (e.g., 47% precision vs. 30%).


One noteworthy finding is that longer reasoning chains normally enhance efficiency. This lines up with insights from Microsoft's Med-Prompt structure and OpenAI's observations on test-time compute and thinking depth.


Challenges and Observations


Despite its strengths, R1 has some constraints:


- Mixing English and Chinese actions due to a lack of monitored fine-tuning.
- Less polished reactions compared to chat designs like OpenAI's GPT.


These issues were attended to throughout R1's improvement procedure, consisting of monitored fine-tuning and human feedback.


Prompt Engineering Insights


An interesting takeaway from DeepSeek's research is how few-shot triggering abject R1's performance compared to zero-shot or succinct tailored prompts. This aligns with findings from the Med-Prompt paper and OpenAI's suggestions to limit context in reasoning designs. Overcomplicating the input can overwhelm the model and lower accuracy.


DeepSeek's R1 is a substantial action forward for open-source thinking designs, showing abilities that equal OpenAI's A1. It's an interesting time to explore these designs and their chat user interface, which is free to utilize.


If you have questions or desire to discover more, examine out the resources linked below. See you next time!


Training DeepSeek-R1-Zero: A reinforcement learning-only method


DeepSeek-R1-Zero stands apart from most other state-of-the-art models because it was trained utilizing just reinforcement knowing (RL), no monitored fine-tuning (SFT). This challenges the existing standard technique and opens brand-new chances to train thinking models with less human intervention and effort.


DeepSeek-R1-Zero is the first open-source model to verify that sophisticated reasoning capabilities can be developed simply through RL.


Without pre-labeled datasets, the model discovers through trial and mistake, refining its habits, criteria, and weights based entirely on feedback from the services it produces.


DeepSeek-R1-Zero is the base design for DeepSeek-R1.


The RL procedure for DeepSeek-R1-Zero


The training process for DeepSeek-R1-Zero involved providing the model with various reasoning tasks, varying from math issues to abstract logic challenges. The model created outputs and was evaluated based upon its performance.


DeepSeek-R1-Zero received feedback through a benefit system that helped direct its knowing procedure:


Accuracy rewards: Evaluates whether the output is proper. Used for when there are deterministic outcomes (math problems).

Format benefits: Encouraged the model to structure its thinking within and tags.


Training prompt template


To train DeepSeek-R1-Zero to produce structured chain of idea series, the researchers utilized the following timely training template, changing timely with the thinking concern. You can access it in PromptHub here.


This design template triggered the design to explicitly outline its thought process within tags before delivering the last answer in tags.


The power of RL in reasoning


With this training procedure DeepSeek-R1-Zero started to produce sophisticated thinking chains.


Through thousands of training actions, DeepSeek-R1-Zero progressed to solve progressively intricate problems. It found out to:


- Generate long thinking chains that enabled much deeper and more structured analytical

- Perform self-verification to cross-check its own answers (more on this later).

- Correct its own errors, showcasing emerging self-reflective behaviors.




DeepSeek R1-Zero efficiency


While DeepSeek-R1-Zero is mainly a precursor to DeepSeek-R1, it still attained high efficiency on a number of criteria. Let's dive into some of the experiments ran.


Accuracy enhancements during training


- Pass@1 accuracy started at 15.6% and by the end of the training it improved to 71.0%, comparable to OpenAI's o1-0912 design.

- The red strong line represents performance with bulk voting (comparable to ensembling and self-consistency techniques), which increased precision further to 86.7%, exceeding o1-0912.


Next we'll take a look at a table comparing DeepSeek-R1-Zero's efficiency across multiple reasoning datasets versus OpenAI's reasoning designs.


AIME 2024: 71.0% Pass@1, a little below o1-0912 however above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a score of 73.3%.

- Performed much even worse on coding jobs (CodeForces and LiveCode Bench).




Next we'll look at how the response length increased throughout the RL training procedure.


This chart reveals the length of actions from the model as the training procedure advances. Each "action" represents one cycle of the model's learning process, where feedback is offered based upon the output's performance, evaluated utilizing the prompt template gone over earlier.


For each concern (corresponding to one step), 16 responses were sampled, and the typical precision was computed to guarantee stable assessment.


As training advances, the design produces longer reasoning chains, enabling it to fix increasingly complicated thinking jobs by leveraging more test-time calculate.


While longer chains do not always guarantee better outcomes, they usually correlate with enhanced performance-a trend likewise observed in the MEDPROMPT paper (find out more about it here) and in the initial o1 paper from OpenAI.


Aha minute and self-verification


Among the coolest aspects of DeepSeek-R1-Zero's development (which likewise uses to the flagship R-1 model) is simply how great the model became at thinking. There were advanced reasoning behaviors that were not clearly configured but developed through its support learning process.


Over thousands of training steps, the model started to self-correct, review problematic reasoning, and verify its own solutions-all within its chain of idea


An example of this noted in the paper, referred to as a the "Aha moment" is below in red text.


In this circumstances, the model actually said, "That's an aha moment." Through DeepSeek's chat feature (their version of ChatGPT) this type of reasoning generally emerges with expressions like "Wait a minute" or "Wait, however ... ,"


Limitations and obstacles in DeepSeek-R1-Zero


While DeepSeek-R1-Zero was able to perform at a high level, there were some downsides with the model.


Language mixing and coherence issues: The model sometimes produced responses that blended languages (Chinese and English).

Reinforcement knowing trade-offs: The absence of supervised fine-tuning (SFT) indicated that the design lacked the refinement required for totally polished, human-aligned outputs.

DeepSeek-R1 was established to resolve these problems!


What is DeepSeek R1


DeepSeek-R1 is an open-source thinking design from the Chinese AI laboratory DeepSeek. It builds on DeepSeek-R1-Zero, which was trained totally with support knowing. Unlike its predecessor, DeepSeek-R1 includes supervised fine-tuning, making it more improved. Notably, it surpasses OpenAI's o1 model on a number of benchmarks-more on that later.


What are the primary differences in between DeepSeek-R1 and DeepSeek-R1-Zero?


DeepSeek-R1 builds on the foundation of DeepSeek-R1-Zero, which functions as the base design. The 2 vary in their training techniques and overall efficiency.


1. Training approach


DeepSeek-R1-Zero: Trained completely with reinforcement knowing (RL) and no monitored fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that consists of monitored fine-tuning (SFT) first, followed by the same support learning process that DeepSeek-R1-Zero wet through. SFT helps improve coherence and readability.


2. Readability & Coherence


DeepSeek-R1-Zero: Fought with language mixing (English and Chinese) and readability problems. Its thinking was strong, but its outputs were less polished.

DeepSeek-R1: Addressed these problems with cold-start fine-tuning, making responses clearer and more structured.


3. Performance


DeepSeek-R1-Zero: Still an extremely strong reasoning model, sometimes beating OpenAI's o1, but fell the language mixing problems lowered functionality considerably.

DeepSeek-R1: Outperforms R1-Zero and OpenAI's o1 on many reasoning criteria, and the actions are much more polished.


In other words, DeepSeek-R1-Zero was an evidence of principle, while DeepSeek-R1 is the completely enhanced variation.


How DeepSeek-R1 was trained


To deal with the readability and coherence concerns of R1-Zero, the scientists incorporated a cold-start fine-tuning stage and a multi-stage training pipeline when developing DeepSeek-R1:


Cold-Start Fine-Tuning:


- Researchers prepared a premium dataset of long chains of idea examples for initial monitored fine-tuning (SFT). This information was collected utilizing:- Few-shot triggering with detailed CoT examples.

- Post-processed outputs from DeepSeek-R1-Zero, fine-tuned by human annotators.




Reinforcement Learning:


DeepSeek-R1 underwent the very same RL process as DeepSeek-R1-Zero to improve its thinking abilities further.


Human Preference Alignment:


- A secondary RL phase enhanced the design's helpfulness and harmlessness, making sure much better alignment with user requirements.


Distillation to Smaller Models:


- DeepSeek-R1's thinking capabilities were distilled into smaller, efficient models like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.




DeepSeek R-1 standard efficiency


The scientists tested DeepSeek R-1 throughout a range of benchmarks and against top models: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.


The standards were broken down into a number of classifications, revealed below in the table: English, Code, Math, and Chinese.


Setup


The following specifications were used throughout all models:


Maximum generation length: 32,768 tokens.

Sampling configuration:- Temperature: 0.6.

- Top-p worth: 0.95.




- DeepSeek R1 exceeded o1, Claude 3.5 Sonnet and other models in the bulk of thinking standards.

o1 was the best-performing design in four out of the 5 coding-related standards.

- DeepSeek performed well on creative and long-context task task, like AlpacaEval 2.0 and ArenaHard, outshining all other models.




Prompt Engineering with thinking designs


My preferred part of the post was the researchers' observation about DeepSeek-R1's level of sensitivity to triggers:


This is another datapoint that lines up with insights from our Prompt Engineering with Reasoning Models Guide, which referrals Microsoft's research study on their MedPrompt structure. In their research study with OpenAI's o1-preview design, they discovered that frustrating reasoning models with few-shot context degraded performance-a sharp contrast to non-reasoning models.


The crucial takeaway? Zero-shot triggering with clear and succinct directions appear to be best when using thinking designs.