GRPO — group relative policy optimisation

GRPO (Shao et al., 2024, DeepSeekMath) is PPO with the critic removed. Instead of learning $V(s)$, it samples a group of $G$ completions for the same prompt, scores each one, and uses the z-score of each score within its group as the advantage. It was designed for language models and is the algorithm behind DeepSeek-R1. This page derives it, ports it as faithfully as possible to LunarLander, and reports honestly what happened: it did not solve the task. The reason it failed explains why it works so well for LLMs.

Background: PPO's clipped surrogate is derived on the PPO page and the baseline argument on the RL overview. The LLM side (RLHF, Dr. GRPO, DPO) is on PPO & GRPO for LLMs. An applied GRPO run with an SQL execution reward is in the LLMs & SQL-GRPO track. Numbers marked measured are from 5 seeds on this machine.

1 · The objective

1.1 Group-relative advantage

For a prompt $q$, sample $G$ outputs $o_1,\dots,o_G \sim \pi_{\theta_\text{old}}(\cdot\mid q)$ and score each with a scalar reward $r_i$. Under outcome supervision, the paper's default, every token $t$ of output $i$ gets the same advantage:

$$ \hat A_{i,t} = \tilde r_i = \frac{r_i - \operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)} \tag{1} $$

Why this is a sensible advantage follows from the baseline argument (overview 8–9). The group mean is a Monte-Carlo estimate of $V^{\pi_\text{old}}(q)$, the expected reward from this prompt, computed from samples instead of a learned network. Subtracting it is a baseline. Dividing by the group std is a per-prompt version of PPO's advantage normalisation: it makes easy prompts (where all outputs score similarly) and hard prompts contribute gradients of comparable size. Two technical notes. Including $r_i$ in its own mean couples the baseline to the sample: since $r_i - \operatorname{mean}_j r_j = \tfrac{G-1}{G}\big(r_i - \operatorname{mean}_{j\ne i} r_j\big)$, the expected gradient is the leave-one-out (RLOO) one scaled by $(G-1)/G$, a change of scale rather than of direction. And the std division is not unbiased either, which is one of the Dr. GRPO criticisms (LLM page).

1.2 PPO's clip, no critic, KL in the loss

The objective maximised (DeepSeekMath eq. 3), with $\rho_{i,t} = \pi_\theta(o_{i,t}\mid q, o_{i,\lt t}) / \pi_{\theta_\text{old}}(o_{i,t}\mid q, o_{i,\lt t})$:

$$ \mathcal J_\text{GRPO}(\theta) = \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \Big\{\min\Big(\rho_{i,t}\hat A_{i,t},\ \operatorname{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\,\hat A_{i,t}\Big) - \beta\, \mathbb{D}_\mathrm{KL}\big[\pi_\theta\Vert\pi_\text{ref}\big]_{i,t}\Big\} \tag{2} $$

Three differences from PPO. There is no value network: the advantage is (1). The KL to a reference policy $\pi_\text{ref}$ (for an LLM, the SFT model) is a term in the loss, whereas InstructGPT-style PPO puts it in the reward. And there is a per-sequence length normalisation $1/|o_i|$, with $|o_i|$ the number of tokens, so each output contributes equally to the average, however long it is.

2 · The KL estimator, derived

The per-token KL in (2) is estimated at the sampled token with (DeepSeekMath eq. 4):

$$ \mathbb{D}_\mathrm{KL}\big[\pi_\theta\Vert\pi_\text{ref}\big]_{i,t} \approx \frac{\pi_\text{ref}(o_{i,t}\mid\cdot)}{\pi_\theta(o_{i,t}\mid\cdot)} - \log\frac{\pi_\text{ref}(o_{i,t}\mid\cdot)}{\pi_\theta(o_{i,t}\mid\cdot)} - 1 \tag{3} $$

Write $x = \pi_\text{ref}(a)/\pi_\theta(a)$ for a single action (token) $a$ drawn from $\pi_\theta$.

Unbiased. The first term has expectation one: $\mathbb{E}_{a\sim\pi_\theta}[x] = \sum_a \pi_\theta(a)\,\pi_\text{ref}(a)/\pi_\theta(a) = \sum_a\pi_\text{ref}(a) = 1$ (assuming $\pi_\text{ref}$'s support is inside $\pi_\theta$'s). So

$$ \mathbb{E}_{a\sim\pi_\theta}\big[x - \log x - 1\big] = 1 - \mathbb{E}_{\pi_\theta}\Big[\log\frac{\pi_\text{ref}(a)}{\pi_\theta(a)}\Big] - 1 = \mathbb{E}_{\pi_\theta}\Big[\log\frac{\pi_\theta(a)}{\pi_\text{ref}(a)}\Big] = D_\mathrm{KL}(\pi_\theta\Vert\pi_\text{ref}) \tag{4} $$

Non-negative, sample by sample. Let $f(x) = x - \log x - 1$ on $x \gt 0$. Then $f'(x) = 1 - 1/x$ is zero only at $x = 1$, and $f''(x) = 1/x^2 \gt 0$, so $x = 1$ is the global minimum and $f(1) = 0$. Hence $f(x) \ge 0$, which is the familiar $\log x \le x - 1$. The naive estimator $-\log x$ (k1) is also unbiased, but it is negative whenever the sampled token is more likely under the reference, which makes a single-sample penalty noisy and sometimes a reward. Adding the zero-mean control variate $x - 1$ removes that without changing the expectation. It is the same k3 construction PPO uses for its diagnostic (PPO §4.1), applied to the opposite KL direction with the reference in the numerator.

Two caveats on "unbiased". The samples come from $\pi_{\theta_\text{old}}$, not $\pi_\theta$. After the first minibatch step they differ, so (4) holds only approximately during the inner epochs. And an unbiased estimate of a KL does not give an unbiased estimate of its gradient: differentiating (3) at a fixed sample is not the gradient of the true KL. Later work examines which KL estimator and placement gives the right gradient. For a penalty with $\beta = 0.04$ this matters less than the advantage design.

3 · Outcome versus process supervision

Outcome supervision: one reward per output, broadcast to every token, eq. (1). This is the default, and it is what was run here.
Process supervision: a process reward model scores each reasoning step. Rewards are group-normalised across all steps of all outputs, and the advantage of token $t$ is the sum of normalised rewards of the steps at or after it, $\hat A_{i,t} = \sum_{\text{index}(j)\ge t}\tilde r_i^{\,\text{index}(j)}$. This restores per-step credit assignment, at the cost of needing per-step rewards. Not run here; see §6.

4 · The port to LunarLander, exactly

LLM (DeepSeekMath)	LunarLander (`grpo.py`)
Prompt $q$	A reset seed. `env.reset(seed=q)` fixes the terrain and the random initial push. Two envs reset with the same seed and fed the same 300 actions were verified to differ by exactly 0.0, so "same prompt" is exact.
$G$ completions $o_1..o_G$	$G = 16$ complete episodes from that same initial state, actions sampled from $\pi_{\theta_\text{old}}$
Token $o_{i,t}$	Action $a_{i,t} \in \{0,1,2,3\}$
Reward-model score $r_i$	Undiscounted episode return $R_i = \sum_t r_{i,t}$
Advantage	$(R_i - \operatorname{mean}_g R)/(\operatorname{std}_g R + 10^{-4})$, broadcast to every step of episode $i$
$1/\|o_i\|$	Per-step weight $\propto 1/L_i$, with $L_i$ the episode length
$\pi_\text{ref}$ = SFT model	A copy of the policy, refreshed $\pi_\text{ref}\leftarrow\pi_\theta$ every 10 iterations (iterative GRPO)
No value model	No value model. Only the 4,996-parameter actor is trained.

Each iteration draws 4 prompts × 16 samples = 64 episodes. The core of the update:

prompts = rng.integers(0, C.TRAIN_SEED_MAX, size=B) # B = 4 prompts seeds = np.repeat(prompts, G) # same seed => same initial state, G = 16 res = C.run_episodes(envs, seeds, sample_act, keep_traj=True) R = res["returns"].reshape(B, G) A_ep = ((R - R.mean(1, keepdims=True)) / (R.std(1, keepdims=True) + 1e-4)).reshape(-1) # eq. (1) adv = np.repeat(A_ep, lens) # every step of episode i gets A_i w = np.repeat(n_steps / (len(lens) * lens), lens) # 1/|o_i|, rescaled so mean weight = 1 ... surr = torch.min(ratio * A, ratio.clamp(1 - eps, 1 + eps) * A) lr_ref = logp_ref[mb] - logp # log(pi_ref / pi_theta) kl = lr_ref.exp() - lr_ref - 1 # eq. (3) loss = -(w[mb] * (surr - beta * kl)).mean() # eq. (2), beta = 0.04

4.1 Choices that had to be made

Parameter	Value	Why
$G$	16	DeepSeekMath uses 64. 16 keeps one iteration at or below 64k steps
Prompts / iteration	4	64 episodes per iteration
Clip $\epsilon$, epochs, minibatch, lr, grad clip	0.2, 4, 64, Adam 3e-4, 0.5	Identical to PPO, so the comparison isolates "critic + GAE" vs "group-relative outcome reward"
$\beta$	0.04	Paper value
Reference refresh	every 10 iterations	A frozen $\pi_\text{ref}$ would pin a from-scratch policy to its random initialisation. An SFT'd LLM does not have that problem
Entropy bonus, discount	0, none	The paper has neither. Outcome reward is the plain episode return

Note what the length normalisation does here. A 70-step crash and a 1,000-step hover each contribute the same total weight to the loss. Per step, the crash's actions are weighted about 14× more heavily.

4.2 Training loop, step by step

One GRPO iteration on LunarLander. Four prompts (reset seeds) each spawn 16 complete episodes from an identical start; each episode collapses to one return; the return is z-scored within its group and copied onto every step of that episode; PPO's clipped loss and a k3 KL to the reference policy do the rest. Compare the PPO diagram: the GAE box, which is where per-step credit comes from, is gone.

# GRPO TRAINING (grpo.py) equations refer to this page # B = 4 prompts x G = 16 samples = 64 whole episodes / iteration; eps 0.2, beta 0.04 # K = 4 epochs, minibatch 64 steps, Adam(lr 3e-4, eps 1e-5), clip-norm 0.5, no entropy bonus, no discount actor pi_theta : 8 -> 64 -> 64 -> 4 logits (4,996 params) # NO critic pi_ref = copy(pi_theta) envs = 64 independent LunarLander-v3 instances while env_steps < 2,000,000: # 74-79 iterations fit (measured) # (a) GROUP ROLLOUT: same prompt = same reset seed (bit-identical starts, verified) q[0..3] ~ Uniform{0, ..., 899,999,999} # training seed range only seeds = repeat(q, 16) # [64] run all 64 episodes to termination, a ~ Categorical(pi_theta(s)) # gradients off -> pi_old store per step: s [8], a, logp_old; L_i = length of episode i; n = sum_i L_i (~6k-64k) R = undiscounted episode returns, reshaped [4, 16] # (b) GROUP-RELATIVE OUTCOME ADVANTAGE, eq. (1) A_ep = (R - mean_g R) / (std_g R + 1e-4) # [4, 16] population std Ahat = repeat(A_ep.flatten(), L_i) # [n] SAME value on every step of episode i w = repeat(n / (64 * L_i), L_i) # [n] 1/|o_i| weight, rescaled to mean 1 logp_ref = log pi_ref(a | s) # [n] no grad # (c) K = 4 epochs over the n steps, eq. (2) for epoch in 1 .. 4: for idx in randperm(n).split(64): logp = log pi_theta(a[idx] | s[idx]) # [64] rho = exp(logp - logp_old[idx]) surr = min(rho * Ahat[idx], clip(rho, 0.8, 1.2) * Ahat[idx]) u = logp_ref[idx] - logp # log(pi_ref / pi_theta) kl = exp(u) - u - 1 # eq. (3): unbiased, >= 0 loss = -mean(w[idx] * (surr - 0.04 * kl)) Adam step on theta, ||g|| clipped to 0.5 if iteration % 10 == 0: pi_ref = copy(pi_theta) # iterative GRPO log degenerate_frac = mean(std_g R < 1e-4) # measured 0.00 throughout log near_degenerate_frac = mean(std_g R < 1.0) # measured 0.00 throughout

Line (b) is the whole difference from PPO. Where PPO computes a different $\hat A_t$ for every one of its 16,384 steps from what the critic expected at that state, GRPO computes 64 numbers and copies each one hundreds of times. Everything after that, the ratio, the clip and the epochs, is PPO's code unchanged, by design, so that the comparison isolates the advantage.

4.3 Inference

# GRPO INFERENCE (common.evaluate = deterministic; eval_stochastic.py = sampled) actor = load("best.pt"); actor.eval(); torch.no_grad() # pi_ref is discarded s = env.reset(seed = q) repeat until term or trunc (at most 1,000 steps): deterministic (shipped): a = argmax(pi_theta(s)) # what the 50-seed table and videos use sampled (as trained): a ~ Categorical(pi_theta(s)) # what the group rollouts used s, r, term, trunc = env.step(a)

For PPO the two branches of this listing give the same controller to within a few points. For GRPO they do not, and §5.2 measures the gap.

5 · Results: it did not solve (measured)

	seed 0	seed 1	seed 2	seed 3	seed 4	Summary
Final policy, deterministic (argmax), 50 eps	22.5	−126.9	40.5	−3.0	−1,877.5	−388.9 ± 746.6 (−16.7 ± 63.3 excl. seed 4); 2.0 % solved
`best.pt`, deterministic, 50 eps	77.8	−18.4	86.3	136.2	−26.8	51.0 ± 61.0

No seed's periodic evaluation ever reached 200. Only 74–79 policy iterations fitted in 2M steps, because each iteration costs 64 complete episodes, up to 64k steps once the policy hovers. PPO gets 123 rollouts from the same budget. Wall-clock 743 s per seed. KL to the reference: median 0.013–0.019, max 0.119–0.167. Clip fraction median 0.028–0.042, max 0.071–0.101. So the optimisation itself was well-behaved.

GRPO training curves on discrete LunarLander with group reward, group std, KL to reference, entropy and degenerate-group fraction — **GRPO on discrete LunarLander-v3**, 2M steps, 5 seeds, measured on this machine. Training return is the rolling mean over sampled-action episodes. Evaluation is deterministic (argmax) on 10 held-out seeds every 25k steps. The diagnostic panels show the within-group return std, the KL to $\pi_\text{ref}$, the policy entropy and the degenerate-group fraction.

5.1 Degenerate groups were not the problem

The standard worry about GRPO is degenerate groups: if all $G$ outputs get the same reward, the std in (1) is zero, every advantage is zero, and the prompt contributes no learning signal. That is common for LLMs on prompts that are far too easy or far too hard. grpo.py logs the fraction of groups with std below $10^{-4}$ (exactly degenerate) and below 1.0 (returns within about one reward point): both were 0.00 in every iteration of every seed. Sixteen episodes from a stochastic policy always spread out in LunarLander, with a within-group return std of 41–82 at the end of training. The signal was there. It was the wrong kind of signal.

5.2 The real failure: the policy never sharpens

The final policies were evaluated on the same 50 held-out seeds both ways (eval_stochastic.py):

Config	Deterministic (argmax, shipped)	Sampled (the policy it trained as)	Policy entropy at start states
PPO	273–280	269–279	0.27–0.44
A2C	−120 … +50	−118 … +62	0.08–0.24
GRPO	−1,878 … +41	+31 … +234 (seed 1: 94 % solved)	0.97–1.21 (max ln 4 = 1.39)

For PPO and A2C, argmax and sampling agree, because their policies became confident (low entropy). GRPO's did not. Its action distribution at the start states stays at 0.97–1.21 nats, close to uniform. Its sampled policy really did learn: seed 1's solves 94 % of the final evaluation seeds. But the argmax of a near-flat distribution is a different controller. It picks whichever action is marginally ahead in each state, consistently, with none of the mixing that made the stochastic policy work. Seed 4 is the extreme case: its sampled policy scores +65, while its argmax fires one engine continuously and scores −1,877 (−2,411 at 1.9M steps, flying out of bounds).

Why the gap means something. A whole-episode advantage gives every action in an episode the same credit. In a 300-step landing, the few thrusts that mattered and the hundreds that did not all get the same $+\hat A_i$, and in a crash they all get the same $-\hat A_i$. Averaged over many episodes, the per-state signal that one action is better than another in this state is the small residual correlation between that state's action and the episode's outcome. That is enough to tilt the distribution, which is why sampling improves, and too weak to make one action dominate, which is why argmax does not. PPO's GAE advantage (overview 12) is per-step: it compares what happened over the next roughly 48 steps with what the critic expected from this state, so it can sharpen the policy state by state. The shaped reward of LunarLander (overview 15) carries that per-step information. Outcome-supervised GRPO sums it away before the learner sees it.

5.3 The best GRPO controller, flown

GRPO, seed 3 best.pt, deterministic (argmax) policy, recorded on this machine. Flown on a pool of 24 held-out video seeds (3,000,000,000 onward): only 3 / 24 score ≥ 200 (13 landed, 2 crashed, 9 timed out). The video shows all three successes (233, 212, 210, all landed) and then, because more than 15 % failed, the first pool episode below 200. That one is a slow 118-point landing, not a crash. 640×400 H.264, 56.0 s. The success rate is printed on the video's stats panel.

5.4 Learning progression from the same starts

Median seed by final eval (seed 3), deterministic, the same 20 held-out initial states. Mean return / fraction landed: −363 / 0 % → −183 / 0 % → −168 / 0 % → 31 / 50 % → 84 / 35 % → −25 / 0 % at 0 / 2.5 / 10 / 25 / 50 / 100 %. At the end, 100 % of those episodes time out: the argmax controller has learned to hover, not to land. Fuel use stays high (75–180 on the fuel proxy), while PPO's falls to 31.

GRPO learning progression from 20 held-out starts at six training stages — **GRPO progression**, seed 3 (median by final eval), deterministic (argmax), the same 20 held-out initial states at 0, 2.5, 10, 25, 50 and 100 % of 2M steps, measured on this machine.

GRPO progression video: seed 3, deterministic, one held-out start replayed with each stage's snapshot. 640×400 H.264, 21.0 s, recorded on this machine. 1920×1080 masters of this and the rollout video exist for YouTube but are not deployed with the site.

6 · Why GRPO suits LLMs and this setup does not suit GRPO

Ingredient	LLM (math, code, SQL)	LunarLander, from scratch
Natural reward granularity	One reward per completion is the natural signal: the answer is right or wrong, the SQL returns the right rows or not. No dense signal is being thrown away.	A dense, shaped per-step reward exists (overview 15). Outcome supervision discards it.
Prior	A strong pretrained, SFT'd prior. Sampled completions are already near-deterministic yet varied, so RL only needs to reweight good modes the model can already produce.	A random 4,996-parameter MLP with a near-uniform policy. RL must discover the behaviour, which needs per-step credit.
Cost of $G$ samples	Cheap: $G$ completions of one prompt batch on a GPU, sharing the prompt's KV-cache. No value model saves a full model copy in memory (LLM page).	Expensive: 64 complete episodes per iteration, so only 74–79 iterations fit in 2M steps.
Mode at deployment	Usually sampled, or greedy from an already peaked distribution.	Argmax of a near-flat distribution, which is a different controller (§5.2).
What a critic costs	A second multi-billion-parameter model, hard to train on sparse terminal rewards.	4,801 parameters. Trivially cheap, and it explains 94–99 % of return variance for PPO.

In short: GRPO removes the critic because, for an LLM, the critic is expensive and nearly useless. Rewards are terminal, and the pretrained policy already concentrates its probability mass. For a from-scratch controller with dense rewards, the critic is cheap and is exactly what supplies per-step credit. Removing it removes the thing that would have sharpened the policy.

7 · Honest limitations of this experiment

Reference configuration, not tuned. GRPO was deliberately not tuned until it beat PPO. Larger $G$, more prompts per iteration, an entropy schedule, or deploying the sampled policy would all likely look better.
Process supervision was not run. A per-step variant using the env's shaped reward, normalised across the group's steps as in §3, is the obvious next experiment. It would partly restore the per-step credit whose absence §5.2 identifies.
The reference schedule is a choice. Refreshing $\pi_\text{ref}$ every 10 iterations makes the KL term a slowly moving trust region rather than an anchor to a meaningful prior. That is necessary from scratch, but it is not how the term is used for LLMs.
The causal story is inferred. The entropy and stochastic-vs-deterministic numbers are consistent with the credit-assignment explanation, but a controlled ablation (GRPO with a per-step advantage) was not run.

8 · References

Shao, Wang, Zhu, Xu, Song, Bi, Zhang, Zhang, Li, Wu, Guo, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, arXiv:2402.03300, 2024. §4.1, eqs. (3)–(4).
DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv:2501.12948, 2025.
Schulman, Approximating KL Divergence, 2020. The k3 estimator.
Liu et al., Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO), arXiv:2503.20783, 2025.
Ahmadian et al., Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs (RLOO), ACL 2024.

Next: PPO & GRPO for LLMs · applied: LLMs & SQL-GRPO · PPO · back to the RL overview