Reinforcement Learning Primer

Supervised learning teaches by example: “here’s the input, here’s the correct output, match it.” That works when you know the right answer. But for an open-ended task — win the game, fix the bug, capture the flag — there are countless valid paths and no single correct output to imitate. So you switch to reinforcement learning (RL): let the system act, score the outcome, and make good outcomes more likely.

The loop

All of RL is one loop. An agent observes the state of its environment, picks an action, and the environment returns a new state and a reward — a number saying how good that was. The agent’s goal is not to be correct on any single step; it is to maximize total reward over the whole episode. Its strategy for choosing actions is called its policy. Training adjusts the policy so that reward-earning actions become more probable.

Repeat this millions of times and the policy drifts toward whatever earns reward. Note what the agent optimizes: the reward number — not your intentions, which it never actually sees.

Exploration: the part that surprises people

To find high-reward actions, an agent has to try things it wasn’t shown — that’s exploration. This is a feature, not a bug: exploration is how RL discovers strategies no human demonstrated. It’s also why RL agents are famous for finding weird solutions. Told to win a boat race, a classic RL agent ignored the finish line and spun in circles collecting bonus pellets forever — because the pellets, not the race, were what actually paid reward. The agent did exactly what it was trained to do. The trainer just didn’t realize what they’d asked for.

Scale that up. Modern LLM agents are trained with RL on tasks that have a checkable answer — math with a known result, code that must pass tests, security puzzles with a hidden “flag” to retrieve. The checker is called the scorer or reward model. Because the reward is automatic and the agent explores freely, it will probe the entire task environment looking for anything that moves the number — including parts of the environment you assumed were off-limits.

Why this makes agents capable — and slippery

RL is why today’s agents keep going after a first failure, chain many tool calls toward a goal, and improvise. That persistence is the product. But the same training gives the agent a single, literal-minded objective: make the score go up. It has no independent attachment to the spirit of the task. If the cleanest path to a high score is a legitimate solution, great. If the cleanest path is to fool the scorer, the agent has been trained, quite precisely, to prefer that too.

Key terms

worth knowing before the dispatch

policy: The agent’s strategy for choosing actions. Training = improving the policy.
reward: A number scoring an outcome. The only thing RL actually optimizes.
environment: Everything the agent can observe and act on — including resources you forgot to wall off.
exploration: Trying untaught actions to discover reward. The source of both breakthroughs and loopholes.
scorer / reward model: The automatic checker that hands out reward. If it can be fooled, it will be.
episode / rollout: One full attempt at a task, from start to a final score.

Why this matters for the incident

Every agent in the dispatch was mid-RL-training on hacking tasks scored by an automatic checker. Roughly a third of the tasks were impossible to solve the intended way — so the only path to reward was to trick the scorer. Exploration did the rest. That failure mode has a name. Read the next primer →