PPO & GRPO for language models

The same two algorithms that fly the lander on the other pages of this section are, with small changes, how instruction-following and reasoning are trained into large language models. This page is conceptual and mathematical: how text generation becomes an MDP, how a reward model is fit from human preferences, why a KL penalty to a reference model is essential, what four model copies cost in memory, how GRPO removes the critic, what the length normalisation does and why Dr. GRPO objects to it, and how DPO skips RL entirely. It closes by explaining why the same ingredients behave so differently on LunarLander.

No new experiments on this page. The RL machinery (policy gradient, GAE, the clip) is derived on the RL overview and the PPO and GRPO pages. For an applied run, Llama-2-7B trained with QLoRA SFT and then GRPO against an SQL-execution reward, see LLMs & SQL-GRPO, which this page is the theory companion to.

1 · The RLHF pipeline

InstructGPT (Ouyang et al., 2022) set the three-stage recipe that most later work modifies:

1. Supervised fine-tuning (SFT). Next-token cross-entropy on human-written demonstrations gives $\pi^\text{SFT}$. It becomes both the initial policy and the reference $\pi_\text{ref}$.
2. Reward model (RM). Humans compare two completions of the same prompt. A copy of the LM with a scalar head, $r_\phi(x, y)$, is trained to rank them (§2).
3. RL. The policy is optimised with PPO to maximise $r_\phi$ while staying close to $\pi_\text{ref}$ (§3–4).

2 · The reward model: Bradley–Terry, derived

For prompt $x$ and two completions $y_w$ (preferred) and $y_l$ (rejected), the Bradley–Terry model (1952) posits a latent scalar "strength" $r(x,y)$ for each completion, with preference probability

$$ P(y_w \succ y_l \mid x) = \frac{\exp r(x,y_w)}{\exp r(x,y_w) + \exp r(x,y_l)} = \sigma\big(r(x,y_w) - r(x,y_l)\big) \tag{1} $$

where $\sigma(z) = 1/(1+e^{-z})$. The second form follows by dividing numerator and denominator by $\exp r(x,y_w)$. Only reward differences are identified: adding any function of $x$ alone leaves (1) unchanged. Given a dataset $\mathcal D = \{(x, y_w, y_l)\}$ of human comparisons, maximum likelihood over (1) gives the reward-model loss:

$$ \mathcal L_\text{RM}(\phi) = -\,\mathbb{E}_{(x,y_w,y_l)\sim\mathcal D}\Big[\log\sigma\big(r_\phi(x,y_w) - r_\phi(x,y_l)\big)\Big] \tag{2} $$

Its gradient makes the mechanism plain. With $\Delta = r_\phi(x,y_w) - r_\phi(x,y_l)$, $\nabla_\phi\mathcal L = -(1-\sigma(\Delta))\,\big(\nabla_\phi r_\phi(x,y_w) - \nabla_\phi r_\phi(x,y_l)\big)$. Each pair pushes the two scores apart, weighted by $1-\sigma(\Delta)$, the model's current probability of getting the pair wrong. Pairs it already ranks confidently contribute almost nothing. Because of the shift invariance, implementations usually normalise $r_\phi$ (for example to zero mean over a reference set) before RL.

2.1 Training the reward model, step by step

3 · Text generation as an MDP

Written over whole completions, the policy factorises over tokens. This is what makes the token-level MDP exact, and it is how every sequence log-probability below is computed:

$$ \pi_\theta(y\mid x) = \prod_{t=1}^{T}\pi_\theta(y_t\mid x, y_{\lt t}), \qquad \log\pi_\theta(y\mid x) = \sum_{t=1}^{T}\log\pi_\theta(y_t\mid x, y_{\lt t}) \tag{3} $$

Because the transition is deterministic and known, "the environment" is just the tokenizer and the reward model. All the randomness is in the policy's sampling. This makes the problem unusually friendly for on-policy methods. Generating more episodes is only more inference, and many completions of the same prompt can be sampled in one batch.

3.1 The per-token KL penalty in the reward

InstructGPT's PPO does not maximise $r_\phi$ alone. The per-token reward is

$$ r_t = -\beta\,\underbrace{\log\frac{\pi_\theta(y_t\mid x,y_{\lt t})}{\pi_\text{ref}(y_t\mid x,y_{\lt t})}}_{\mathrm{KL}_t} \;+\; \mathbb 1[t = T]\; r_\phi(x, y) \tag{4} $$

i.e. a KL cost on every token, with the RM score added at the final token $T$. Summed over the episode, the expected return is $\mathbb{E}[r_\phi(x,y)] - \beta\, D_\mathrm{KL}(\pi_\theta(\cdot\mid x)\,\Vert\,\pi_\text{ref}(\cdot\mid x))$, because the sum of per-token log-ratios of a sampled sequence is an unbiased (k1) estimate of the sequence-level KL. So the RL problem is a KL-regularised objective:

$$ \max_{\pi_\theta}\; \mathbb{E}_{x\sim\mathcal D,\,y\sim\pi_\theta(\cdot\mid x)}\big[r_\phi(x,y)\big] - \beta\, D_\mathrm{KL}\big(\pi_\theta(\cdot\mid x)\,\Vert\,\pi_\text{ref}(\cdot\mid x)\big) \tag{5} $$

Why the KL term is indispensable:

• Reward hacking. $r_\phi$ is a learned proxy, accurate only near the data it was trained on. Unconstrained, the policy finds completions where the RM is wrong but scores high, such as repetitive flattery, length inflation, or odd token sequences. Goodhart's law applies to the model directly: measured true quality first rises and then falls as the RM score keeps climbing (Gao et al., 2023).
• Mode collapse. Without the KL, the entropy of the policy collapses onto a few high-reward phrasings and the diversity of the SFT model is lost. The KL to a broad reference acts as an entropy regulariser.
• Staying on the RM's distribution. The reward model only knows how to score text that resembles its training comparisons, which were sampled from SFT-like policies. The KL keeps the policy in that region, where the scores mean something.

3.2 The value head, GAE per token, and four models in memory

PPO needs per-token advantages. A value model $V_\psi(x, y_{\le t})$, typically initialised from the RM and with the same architecture, predicts the expected remaining return from each prefix. GAE (overview 12) is then run over the token sequence with the rewards (4). With $\gamma = \lambda = 1$ the terminal RM score is spread backward through the value baseline, and the per-token KL terms give the only dense signal.

For a $P$-parameter model with a same-size critic, that is roughly $16P + 16P + 2P + 2P = 36P$ bytes before activations and the KV-cache. A 7B policy needs about 250 GB, so multi-GPU sharding is unavoidable. The two trained models dominate, and the value model is half of that. This byte-per-parameter accounting is the standard mixed-precision Adam estimate (Rajbhandari et al., 2020), not a measurement on this machine. LoRA, as used in the SQL-GRPO track, shrinks the optimiser state dramatically, but the forward passes through four networks remain.

3.3 PPO-RLHF, step by step

3.4 Inference: what actually gets deployed

As on LunarLander, every training-time model except the actor disappears at deployment. The difference is the decoding rule. An LLM is usually sampled from, whereas the lander's controller is the argmax. A policy trained with a KL to a broad reference keeps enough entropy for sampling to be meaningful, but not so much that the samples stop being good. The §7 table returns to this.

4 · GRPO's pitch: drop the value model

GRPO (Shao et al., 2024) replaces $V_\psi$ with a sampled baseline. For each prompt, sample $G$ completions, score each, and use the within-group z-score (1 on the GRPO page) as the advantage of every token in that completion. The memory accounting above loses the $16P$ value term. Against PPO's two trained models, that is about half the trained-model memory. It is also one fewer forward and backward pass per token, and one fewer model to keep stable, since a value model trained on sparse terminal rewards is itself a source of instability.

• KL goes in the loss, not the reward. GRPO subtracts $\beta\,\mathbb{D}_\mathrm{KL}[\pi_\theta\Vert\pi_\text{ref}]$ directly in the objective, using the unbiased non-negative k3 estimator (derived here), instead of folding $-\beta\,\mathrm{KL}_t$ into $r_t$. The advantage then reflects the task reward alone, and the KL acts as a separate regulariser rather than being normalised away within the group.
• Verifiable rewards. Because the group statistic only needs a scalar per completion, the reward model can be replaced by a rule: does the final answer match, do the unit tests pass, does the SQL return the right rows. That removes the learned RM as well, and reward hacking of a learned proxy with it. Rule-based rewards can still be gamed through format exploits, which is why practical rewards add format checks, like the formatting bonus in the SQL-GRPO reward. DeepSeek-R1 (2025) used exactly this: accuracy and format rewards, no neural RM, for its reasoning RL stage.
• Degenerate groups. If all $G$ completions get the same reward (all right on an easy prompt, all wrong on a hard one), every advantage in that group is zero. With binary verifiable rewards this happens often, and practical recipes filter or resample such prompts. On LunarLander, with continuous returns, it measured 0.00 (GRPO §5.1).

4.1 Length normalisation and its biases

GRPO's objective averages each completion's per-token terms with weight $1/|o_i|$ (eq. 2 on the GRPO page). Every completion therefore contributes the same total weight, and the per-token gradient for completion $i$ is scaled by $\hat A_i/|o_i|$. That has a side effect:

• For a positive advantage, short completions get a larger per-token push, so correct answers are nudged toward brevity.
• For a negative advantage, long completions get a smaller per-token penalty, so wrong answers are penalised less the longer they are.

The net effect is a systematic pressure toward longer incorrect responses. Combined with the per-prompt std division in the advantage, which up-weights prompts whose rewards have low spread (very easy or very hard), it changes the objective away from expected reward in ways that have nothing to do with the task.

4.2 The Dr. GRPO critique

Liu et al. (2025), Understanding R1-Zero-Like Training, identify both terms as optimisation biases and propose Dr. GRPO ("GRPO Done Right"), which removes them:

$$ \hat A_{i,t} = r_i - \operatorname{mean}(r_1,\dots,r_G), \qquad \mathcal J = \frac{1}{G}\sum_{i}\sum_{t=1}^{|o_i|}\min\big(\rho_{i,t}\hat A_{i,t},\ \operatorname{clip}(\rho_{i,t},1\pm\epsilon)\,\hat A_{i,t}\big) \tag{6} $$

That is, no division by the group std and no $1/|o_i|$. The per-token sum is normalised by a constant, such as the maximum generation length, rather than by each completion's own length. They report that this keeps token efficiency (wrong answers stop growing) at comparable accuracy, and argue that some of the "response length grows during RL" phenomenon reported for R1-Zero-style training is this bias rather than emergent reasoning. The same paper notes that the standard PPO loss implementation has a similar length bias, from averaging per-token losses over each sequence's own length.

4.3 GRPO for LLMs, step by step

5 · DPO: the RL-free alternative

Rafailov et al. (2023) observed that the KL-regularised objective (5) has a closed-form optimum. For each $x$, maximise $\sum_y \pi(y\mid x)\,r(x,y) - \beta\sum_y\pi(y\mid x)\log\frac{\pi(y\mid x)}{\pi_\text{ref}(y\mid x)}$ subject to $\sum_y\pi(y\mid x) = 1$. Setting the derivative of the Lagrangian to zero gives a Gibbs distribution:

$$ \pi^*(y\mid x) = \frac{1}{Z(x)}\,\pi_\text{ref}(y\mid x)\,\exp\!\Big(\frac{1}{\beta}\,r(x,y)\Big), \qquad Z(x) = \sum_y \pi_\text{ref}(y\mid x)\exp\!\Big(\frac{1}{\beta}r(x,y)\Big) \tag{7} $$

$Z(x)$ sums over all possible completions, so it is intractable, which is why (7) cannot be used directly. But (7) can be inverted to express the reward in terms of the optimal policy: $r(x,y) = \beta\log\frac{\pi^*(y\mid x)}{\pi_\text{ref}(y\mid x)} + \beta\log Z(x)$. Substituting into Bradley–Terry (1), the $\beta\log Z(x)$ term cancels in the difference $r(x,y_w) - r(x,y_l)$, because it depends only on $x$. Parameterising $\pi^*$ by $\pi_\theta$ and maximising the likelihood of the preferences gives the DPO loss:

$$ \mathcal L_\text{DPO}(\theta) = -\,\mathbb{E}_{(x,y_w,y_l)}\Big[\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_\text{ref}(y_w\mid x)} - \beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_\text{ref}(y_l\mid x)}\Big)\Big] \tag{8} $$

This is exactly the reward-model loss (2) with the "implicit reward" $\hat r_\theta = \beta\log(\pi_\theta/\pi_\text{ref})$. The policy is the reward model. No sampling, no critic, no RL loop: (8) is a supervised loss on a fixed preference dataset, with two models in memory (policy and frozen reference). The trade-offs are that DPO is offline, so it cannot improve beyond what the preference pairs cover, and its implicit reward can over-fit the pairs. Practitioners report that a reward model plus on-policy RL generalises better when the preference data is narrow (Xu et al., 2024).

Other critic-free baselines. RLOO (Ahmadian et al., 2024) is REINFORCE with a leave-one-out baseline: sample $k$ completions and use $r_i - \frac{1}{k-1}\sum_{j\ne i} r_j$ as the advantage. It is unbiased and has no std division and no clipping, so it is essentially Dr. GRPO without the PPO machinery. REINFORCE++ (Hu, 2025) keeps PPO's clip and token-level KL-in-reward but replaces both the critic and the group with a global, batch-level advantage normalisation, so it does not need several samples per prompt.

5.1 DPO, step by step

Put this listing next to the reward-model listing in §2.1. They are the same loop: two forward passes per pair, a log-sigmoid of a difference, one optimiser step. DPO's "reward" is a log-probability ratio against the frozen reference, which is why the fitted model is a policy rather than a scorer. Nothing in the loop samples, so it cannot discover completions better than the ones in the dataset.

6 · PPO-RLHF vs GRPO vs DPO

7 · Back to LunarLander: same ingredients, different outcome

The GRPO port on this site is algorithmically faithful: same prompt as same seed, $G$ samples, a group-normalised outcome reward, the clip, the k3 KL. It still lost to PPO by about 15× on the deterministic metric. The two settings differ in exactly the places this page identifies as GRPO's assumptions:

And PPO's ingredients carry over the other way. The per-token KL-in-reward of (4) is structurally the same as a shaped per-step reward. It gives PPO's critic a dense signal even when the task reward is terminal, just as LunarLander's shaping (overview 15) gives the critic something to fit at every step.

8 · References

• Schulman, Wolski, Dhariwal, Radford, Klimov, Proximal Policy Optimization Algorithms, arXiv:1707.06347, 2017.
• Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT), NeurIPS 2022.
• Stiennon et al., Learning to summarize from human feedback, NeurIPS 2020. The per-token KL-in-reward formulation.
• Bradley & Terry, Rank Analysis of Incomplete Block Designs, Biometrika 1952. Eq. (1).
• Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, arXiv:2402.03300, 2024. GRPO.
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv:2501.12948, 2025.
• Rafailov, Sharma, Mitchell, Ermon, Manning, Finn, Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS 2023. Eqs. (7)–(8).
• Liu, Chen, Li, Du, Pang, Lin, Lee, Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO), arXiv:2503.20783, 2025.
• Ahmadian et al., Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs (RLOO), ACL 2024.
• Hu, REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models, arXiv:2501.03262, 2025.
• Gao, Schulman, Hilton, Scaling Laws for Reward Model Overoptimization, ICML 2023.
• Xu et al., Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, ICML 2024.
• Rajbhandari, Rasley, Ruwase, He, ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, SC 2020. The 16-bytes-per-parameter accounting.

Related: applied LLMs & SQL-GRPO · GRPO on LunarLander · PPO · back to the RL overview