Policy Gradients

Chapters 04–05 optimized behavior by learning \(Q\) and reading \(\pi(s)=\arg\max_a Q(s,a)\). That recipe breaks for continuous actions and struggles when the optimal policy is inherently stochastic. Policy gradients flip the approach: parameterize \(\pi_\theta\) and push \(\theta\) uphill on expected return.

Everything from here through PPO and (with a fork) SAC is “policy gradients, made stabler or more sample-efficient.” This chapter is the spine.

1. Continuous actions. \(\arg\max_a Q(s,a)\) over \(a\in\mathbb{R}^m\) is an optimization problem at every timestep — expensive and awkward. A Gaussian policy \(\pi_\theta(a\mid s)=\mathcal{N}(\mu_\theta(s),\Sigma_\theta(s))\) outputs actions by sampling, no inner \(\arg\max\).
2. Stochastic optima. Rock–paper–scissors, mixed strategies, or robots that must inject noise to avoid getting stuck: the optimal policy need not be deterministic. Softmax / Gaussian policies express that naturally; greedy Q does not (unless you add external \(\varepsilon\)).
3. Direct optimization. You optimize the thing you care about \(J(\theta)\) rather than solving a fixed-point for \(Q^*\) and hoping greedy readout is well-behaved under approximation.

We want \(\nabla_\theta J(\theta)\). The difficulty: \(R(\tau)\) depends on states visited under \(\pi_\theta\), and the transition kernel \(P(s'|s,a)\) is an opaque black box — you cannot backprop through physics in the general RL setting.

For any density \(p_\theta(x)\) with \(p_\theta(x)>0\),

Therefore \(\nabla_\theta \mathbb{E}_{x\sim p_\theta}[f(x)] = \mathbb{E}_{x\sim p_\theta}\big[f(x)\,\nabla_\theta\log p_\theta(x)\big]\) when \(f\) does not depend on \(\theta\) explicitly. Apply this with \(x=\tau\) and \(f=R\):

Factor the trajectory density (Chapter 01):

Take \(\log\), then \(\nabla_\theta\): every \(P(\cdot)\) term drops out — it does not depend on \(\theta\). What remains is

Combining (2) and (4), and refining the credit so that action \(a_t\) is weighted only by future return from \(t\) (rewards before \(t\) are independent of \(a_t\) given the past), one obtains the standard form:

Equivalently with advantages \(A^{\pi}=Q^{\pi}-V^{\pi}\) (same expectation — Section 7). Here \(d^{\pi}\) is the discounted occupancy from Chapter 01.

Replace \(Q^{\pi}(s_t,a_t)\) by the Monte Carlo return \(G_t\) and use a single sampled trajectory:

Intuition: if the episode went well (large \(G_t\)), increase the log-probability of the actions you took; if it went poorly, decrease them. This is Monte Carlo applied to policies — unbiased (for the true gradient, in expectation) but high variance. You must wait until the episode ends to form \(G_t\).

Subtract any action-independent baseline \(b(s_t)\) from \(G_t\):

This does not change the expected gradient: \(\mathbb{E}_{a\sim\pi}[\nabla\log\pi(a\mid s)\,b(s)]=b(s)\nabla\sum_a\pi(a\mid s)=0\). Choosing \(b(s)=V^{\pi}(s)\) yields the advantage \(A_t \approx G_t - V(s_t)\): not “was return good?” but “was it better than usual from this state?” — much lower variance.

REINFORCE still needs full \(G_t\). Replace the Monte Carlo advantage by a one-step TD advantage — the TD error from Chapter 04:

Two learners in parallel:

• Actor \(\pi_\theta\) — updated by (8).
• Critic \(V_w\) — updated by ordinary TD(0): \(w \leftarrow w + \beta\,\delta_t\,\nabla_w V_w(s_t)\) (tabular: \(V(s)\leftarrow V(s)+\beta\delta_t\)).

The same \(\delta_t\) trains both. This is the skeleton of A2C/A3C and, with GAE and trust regions, of TRPO/PPO. SAC is also an actor–critic, but off-policy with an entropy-augmented critic target.

Define \(\delta_t = r_{t+1}+\gamma V(s_{t+1})-V(s_t)\). The \(k\)-step advantage is \(\hat A_t^{(k)}=\sum_{l=0}^{k-1}\gamma^l\delta_{t+l}\). GAE exponentially averages all \(k\):

\(\lambda=0\): pure one-step TD advantage (low variance, more bias). \(\lambda=1\): Monte Carlo advantage (unbiased if \(V\) correct only at end, high variance). Typical deep RL: \(\lambda\approx 0.95\). GAE is the advantage estimator inside standard PPO implementations.

The update \(\theta\leftarrow\theta+\alpha\nabla_\theta J\) is steepest ascent in Euclidean parameter space. A small step in \(\theta\) can be a huge step in policy distribution space (KL divergence) — or almost no step — depending on which weights you touch. One bad update collapses an on-policy method: all new data comes from the broken policy.

The natural policy gradient preconditions with the Fisher information matrix \(F(\theta)=\mathbb{E}[\nabla\log\pi\,\nabla\log\pi^\top]\), which locally approximates KL:

TRPO enforces a hard KL trust region and solves the constrained step approximately with conjugate gradients. PPO approximates the same “don’t move \(\pi\) too far” idea with a clipped surrogate — cheap, no Fisher matrix. That is the next chapter.

Two actions, \(\pi=(\pi_1,\pi_2)=(0.2,0.8)\), took \(a=1\), advantage \(A=2\), learning rate \(\alpha=0.1\). Softmax score-function gradient for the taken action: \(\partial\log\pi_1/\partial h_1 = 1-\pi_1=0.8\), \(\partial\log\pi_1/\partial h_2 = -\pi_2=-0.8\).

After the update, renormalising the softmax increases \(\pi_1\) — we reinforced the action that had positive advantage.

Vanilla policy gradients work but are brittle in deep nets. Next: TRPO (trust-region / natural gradient done carefully), then PPO (clipped surrogate), then the off-policy continuous-control line toward SAC. Optionally, an imitation-learning detour (behavioral cloning, DAgger) can sit between Phase 1 and those algorithms — your original roadmap allowed either order.