At Alquist, we build and deploy autonomous machines that have to adapt to our customers' operations, often in highly unstructured environments. A language model serves as the planner: it interprets a request and decides what the machine should do through tools. These decisions have physical consequences: tool calls can cause the machine to act or move through space. Getting the task right requires following the rules of the environment, the expectations of the supervisor, and the requirements of the task itself.

Many of these environments require our machines to operate without connectivity, using only the compute available onboard. This limits the size of the model we can deploy and makes scaling to larger models prohibitive. Instead, our models must continuously improve and learn from corrections.

As a simple illustrative example, consider a request to find an object. The planner might need to confirm the request before moving. It also needs to distinguish “not here” from “not found”: checking one area should not end the search while other permitted locations remain.

For a narrowly specified task, such as placing an object at a target location, success can often be verified directly. For our planner, finding the object is only part of success: it might achieve that outcome while skipping confirmation, violating a restriction, or making unnecessary tool calls. A reward must account for these requirements, but a score alone does not explain which decisions should change.

As we add tasks, the guidelines grow. So does the list of ways to violate them. Putting more instructions and exceptions into the system prompt becomes difficult to maintain. We want the planner to learn from corrections and carry that learning into future interactions.

This led us to study how to train a model from verbal feedback. A capable judge can review an attempt and explain what needs to change. The challenge is turning that explanation into an effective training signal for the model we deploy.

We turn verbal feedback into a distillation loss. We compare a teacher model’s predictions with and without feedback, then use that change to adjust the student’s own predictions. We also train on feedback-guided continuations, so the student can learn what comes after a corrected decision. With this approach, we substantially outperform the baselines in both knowledge acquisition and robot planning. The trained student acts without the feedback at inference time.

A numerical reward leaves the correction implicit

One way to train against a behavioral specification is to score each requirement, assign weights, and combine the results into a reward. The agent receives a higher reward when its actions satisfy more of the specification.

A student produces text, a verifier grades four requirements, and weighted grades are combined into a reward of 0.6 that drives the student’s update.
Figure 1. A rubric can capture several requirements, but combining them into one reward discards the explanation of what happened.

The hard part is assigning credit. A rollout is the model’s complete generated attempt, including its text and tool calls. Suppose the agent moves without permission, searches the first area correctly, then stops before checking the remaining permitted areas. Which choices should training reinforce, and which should it discourage?

Common RL recipes such as GRPO1 apply the same reward-derived advantage to every token in an attempt. Across many attempts, the model can learn which decisions lead to higher rewards. But the reward itself leaves much of that work implicit. We can build more detailed reward functions, at the cost of more engineering for each clause and failure case.

A judge can instead provide a correction like this:

You started moving without the supervisor’s permission, then reported “not found” after checking only one area. Ask for permission and wait for approval before moving. If the object is not in the first area, continue searching the remaining permitted areas before reporting “not found”.

This feedback identifies the missing permission and explains how to continue when the object is not in the first area. A scalar reward compresses these instructions into a measure of how well the attempt went. Verbal feedback preserves them for the learner.

Turning feedback into a training signal

Distillation trains a student, the model we want to improve, using the predictions of a teacher. In the robot setting, the student is our agent. We use a frozen copy of the student’s initial model as the teacher, and a separate judge supplies verbal feedback.

At each position in a response, the student and teacher assign probabilities to possible next tokens. These probabilities provide a more detailed signal than a score for the whole response.

In on-policy distillation2,3, the student generates the response. The teacher then scores possible next tokens at each prefix: the text generated up to that point. Training moves the student’s predictions toward the teacher’s at the situations the student actually encounters.

Standard on-policy distillation: the student generates text, the teacher scores it, and a loss compares student and teacher logits to update the student.
Figure 2. The teacher evaluates the student’s own continuation.

Feedback gives the teacher information the student did not have when it made the attempt. A judge reviews the rollout against the task criteria and writes a critique. We give that critique to the teacher, whose predictions now reflect the additional information. Figure 3 shows how this feedback enters the distillation loop.

A judge reads the specification and student’s attempt, writes verbal feedback, and passes it to the teacher. The teacher’s predictions supervise the student.
Figure 3. The judge supplies verbal feedback, and the teacher translates it into predictions over the student’s vocabulary. Our method implements this loop by specifying how the Loss uses teacher and student predictions to update the student.

The judge and teacher have different jobs. The judge can be a frontier model accessed through a text API. It can spend time reasoning about the attempt and the criteria before writing its feedback. We do not need its token probabilities. The teacher needs to expose probabilities over the student’s vocabulary and interpret the feedback at each prefix.

Designing that loss involves two choices: what target should the student learn from, and on which continuations should we evaluate it? We address each in turn.

1. Measure what the feedback changes

How should we measure that change? Prior contrastive methods use log contrast: the difference between two log-probabilities.5 Applied to our feedback setup, it scores tokens by the proportional change in the teacher’s probability when feedback is added.

We propose using relative distance for this comparison. It measures how much of the remaining gap to certainty or impossibility the feedback closes. We use this score to adjust the student’s probabilities and form the corrected target. Before explaining the math, let’s see what this choice changes.

Learning from an imperfect teacher

We built a small experiment to see how the choice of training target affects what the student learns. There are just three possible answers, one correct. We simulate a teacher that receives useful feedback but also makes noisy changes to the probabilities of wrong answers.

The interactive Figure 5 compares the OPD target (R), which matches the teacher directly, with two corrected targets: one built using log contrast (A) and one using relative distance (Z, ours). Three small neural networks train with the same starting weights, examples, and loss. Only the target changes.

The controls vary how much the teacher knows before feedback and how strongly it responds to feedback. The curves show the learning progress at your chosen setting; the maps show the final result across both controls. Darker green means a higher probability of the correct answer.

Relative distance (ours) stays dark green across teacher settings, while the direct OPD target and log contrast vary.

Static preview of a completed training run.

Figure 5. Each curve shows the student’s average probability of the correct answer on the toy training examples.

Relative distance stays green across the whole map. Its average final correct-answer probability remains near certainty at every tested setting. Log contrast works well when the teacher responds strongly to feedback, but can fall below chance when a capable teacher responds weakly. With the OPD target, the student largely inherits the teacher’s confidence. Relative distance can learn from the correction and become more confident in the right answer than the teacher supplying it.

Learning facts the teacher did not know

Trivia Fantasy recall for Qwen3.5-2B through training step 293. Ours approaches 100% and sustains it; SDPO and W2S-OPD remain substantially lower. Faint traces show the original evaluations.
Figure 6. Recall without feedback.7

We also tested this transfer with an LLM, on a real task. Trivia Fantasy contains 20 invented facts known only to the judge. The judge explains the correct answer in feedback, which the teacher turns into token-level supervision for the student.

For the student, Qwen3.5-2B, the task is simply to memorize these facts. We test recall on the same questions without feedback. Our full method reaches near-complete recall, while the baselines learn substantially less.6

What should count as stronger evidence?

Suppose feedback changes two of the teacher’s probabilities:

  • Correct token: 95% → 98%. A proportional increase of about 3.2%.
  • Wrong token: 1% → 1.2%. A proportional increase of 20%.

Log contrast measures proportional growth, so it gives the wrong token the higher score.

Relative distance measures how much of the remaining gap the feedback closes. The correct token moved 60% of the way from its starting probability to certainty. The wrong token closed only about 0.2% of its own remaining gap.

Our confirmation score applies this idea to both increases toward certainty and decreases toward impossibility. For a candidate next token vv, write the teacher’s probability before feedback as b(v)b(v) and its probability with feedback as f(v)f(v). For 0<b(v)<10<b(v)<1, the score is

S(v)={f(v)−b(v)1−b(v)if f(v)≥b(v),f(v)−b(v)b(v)otherwise.S(v)= \begin{cases} \dfrac{f(v)-b(v)}{1-b(v)} & \text{if } f(v)\ge b(v),\\[6pt] \dfrac{f(v)-b(v)}{b(v)} & \text{otherwise}. \end{cases}

An increase is divided by the probability left to gain; a decrease is divided by the starting probability. For the correct token above, this gives (0.98−0.95)/(1−0.95)=0.6(0.98-0.95)/(1-0.95)=0.6. The score stays between −1 and 1, with unchanged probabilities receiving zero.

Why this score? It turns out that’s the only ordering that fits the rules.

We treat a possible next token as a hypothesis, and the feedback as evidence for or against it. The teacher’s probability before feedback is the prior; its probability afterward is the posterior.

To compare those probabilities, we use four requirements from probabilistic confirmation:

  1. Use the before-and-after probabilities. The score should depend on those two values.
  2. More support should give a higher score. For a fixed starting probability, a higher probability after feedback should count as stronger confirmation.
  3. Measure decreases proportionally. When feedback lowers a probability, the score should depend on the fraction of its original probability that remains.
  4. Keep a hypothesis and its opposite consistent. If the feedback supports token A more strongly than token B, it should count against “anything except A” more strongly than “anything except B.”

Together, these requirements determine an ordering of the tokens, as established by Crupi and Tentori’s representation theorem8. They leave the numerical scale free. The relative-distance score used above follows this ordering.

The complement requirement explains the distinction. Feedback strongly lowers the probability of “anything except the correct token,” from 5% to 2%. It barely changes “anything except the wrong token,” from 99% to 98.8%. Yet the log-probability contrast ranks both the wrong token and its complement higher than their counterparts. This violates the requirement that those rankings reverse.

Build the target around the student

We use these scores to reweight the student’s current next-token probabilities. If the student assigns probability q(v)q(v) to token vv, we construct the corrected target as

pcorr(v)=q(v)eβS(v)∑uq(u)eβS(u).p^{\mathrm{corr}}(v)=\frac{q(v)e^{\beta S(v)}}{\sum_u q(u)e^{\beta S(u)}}.

The parameter β>0\beta>0 controls how strongly feedback changes the target.

In the numerical example above, both tokens have positive scores, and the correct token gets the larger boost. Log contrast reverses that ranking. Throughout training, we rebuild the target from the student’s current predictions. Learning can therefore continue even after the student becomes more confident than the teacher.

This construction has a useful property: if feedback leaves the teacher’s predictions unchanged, the target equals the student’s current distribution. There is no local distillation update at that prefix. Because the scores are bounded, the target’s change relative to the student is also bounded for a fixed reweighting strength. The resulting target reflects the feedback through the teacher’s interpretation.

2. Learn how to continue after the correction

The second choice is where to evaluate the loss. Even with a better target, training only on student rollouts can leave much of a critique unused.

Returning to the object-finding task, the judge asks the agent to get the supervisor’s permission before issuing a navigation call and to keep searching if the object is not in the first area. Two illustrative continuations show why the source of the rollout matters:

Student attempt Feedback-guided teacher attempt
Navigate without permission Ask for permission and wait for approval
Search the first area; object not found Navigate and search the first area; object not found
Stop and report “not found” Continue to the next permitted area

The histories diverge at the first decision. On the student’s attempt, the teacher can favor asking for permission before the navigation call. Later prefixes, however, follow a decision to move without approval. Once the student ends the search, there are also no further search steps to learn from.

Figure 3 shows the loop with a student-generated attempt. To evaluate our loss, we allocate part of the rollout pool to the teacher conditioned on feedback. The teacher’s attempts can provide prefixes after the supervisor’s approval and during a continued search. Both kinds of rollout train the student against the corrected target. This provides supervision on how to continue before the student can reliably generate the whole sequence itself.

An objective that covers both paths

We want the student to learn both from its own attempts and from continuations suggested by feedback. Here, q(y)q(y) is the probability that the current student generates a complete rollout yy. The corrected distribution, pcorr(y)p^{\mathrm{corr}}(y), gives the probability of that same rollout if we use our feedback-corrected next-token predictions at every step. Each rollout probability is the product of the corresponding token probabilities along that rollout.

KL divergence measures disagreement between two probability distributions. It is asymmetric: each direction averages over rollouts from the distribution on its left. Our ideal objective adds the two directions:

J(q,pcorr)=KL ⁣(q ∥ pcorr)⏟student prefixes+KL ⁣(pcorr ∥ q)⏟corrected-policy prefixes.\mathcal{J}(q,p^{\mathrm{corr}})= \underbrace{\mathrm{KL}\!\left(q\,\|\,p^{\mathrm{corr}}\right)}_{\text{student prefixes}} +\underbrace{\mathrm{KL}\!\left(p^{\mathrm{corr}}\,\|\,q\right)}_{\text{corrected-policy prefixes}}.

We aim to minimize this sum, also known as the Jeffreys divergence, which is zero when the student and corrected rollout distributions agree. The first term provides supervision at prefixes visited by the student. The second provides supervision at prefixes visited by the corrected policy, including continuations the student rarely generates.

We estimate both terms using the same pool of student and feedback-conditioned teacher rollouts. Pooling lets every rollout contribute to both parts of the loss, giving us more learning signal from the same rollout budget. Importance weights account for how each prefix was sampled. You can find the derivation and practical approximations in our paper.

Putting this together, here is how we train on a task:

  1. The student makes an attempt. It receives the task and generates a rollout without feedback.
  2. The judge reviews the attempt. It checks the rollout against the task’s requirements and writes feedback explaining what should change.
  3. The teacher generates additional attempts. It receives the task, the student’s attempt, and the judge’s feedback, then generates a rollout. We pool its feedback-guided rollouts with the original student rollout.
  4. We turn the feedback into training targets. At each prefix in the pool, we compare the teacher’s next-token predictions with and without feedback. The confirmation scores from this comparison reweight the student’s current predictions to form the corrected target.
  5. We update the student. We train its predictions toward these targets using both directions of the loss. Every rollout in the pool contributes to both terms.

Following the robot’s behavioral requirements

For our robots, learning from feedback needs to carry over to new tasks. We tested this in EmbodiedEval, our internal benchmark, where the agent carries out requests through tool calls in conversations with a simulated user.

We trained Qwen3.5-4B and evaluated it on 240 held-out tasks across 12 suites. A task passes when the evaluation judge detects no violation of its requirements. We report the mean pass rate across suites.

We reduced the mean failure rate by roughly a third compared with the strongest baseline.

EmbodiedEval mean pass rates for Qwen3.5-4B: Ours 82.1%, SDPO 73.7%, W2S-OPD 60.8%, and Base model 57.3%. Error bars show the reported 95% confidence intervals.
Figure 7. The gains carry over to held-out tasks.9

Conclusion

A useful correction can identify a mistake, explain a requirement, supply a missing fact, or describe the next attempt. Our method uses the teacher’s response to that information to construct a training target, and feedback-guided continuations to teach the student how to carry out the correction. The student can then use what it learned without receiving the critique at inference time.

For Alquist, the appeal is practical: the person supervising a robot can explain what should change in the language they already use. We want those explanations to become part of how the robot behaves on future tasks, so it can keep learning as its tasks and requirements change.

You can find the full derivation, proofs, benchmark details, and additional experiments in our paper.

Notes and references

  1. Zhihong Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Back to text
  2. Rishabh Agarwal et al. (2024). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. Back to text
  3. Kevin Lu and Thinking Machines Lab (2025). On-Policy Distillation. Back to text
  4. Work such as RLCSD shows how giving a teacher extra information can shift the training signal toward style rather than the task. Back to text
  5. See RLCSD and W2S-OPD. Back to text
  6. We compare with SDPO and W2S-OPD, adapted to learn from the judge’s feedback. Back to text
  7. Faint lines show the original evaluation results. Bold curves show an exponential moving average with a smoothing factor of 0.35. Back to text
  8. Vincenzo Crupi and Katya Tentori (2013). Confirmation as Partial Entailment: A Representation Theorem in Inductive Logic. Back to text
  9. Error bars show 95% confidence intervals for the mean pass rate. Back to text