At Alquist, we build and deploy autonomous machines that have to adapt to our customers' operations, often in highly unstructured environments. A language model serves as the planner: it interprets a request and decides what the machine should do through tools. These decisions have physical consequences: tool calls can cause the machine to act or move through space. Getting the task right requires following the rules of the environment, the expectations of the supervisor, and the requirements of the task itself.
Many of these environments require our machines to operate without connectivity, using only the compute available onboard. This limits the size of the model we can deploy and makes scaling to larger models prohibitive. Instead, our models must continuously improve and learn from corrections.
As a simple illustrative example, consider a request to find an object. The planner might need to confirm the request before moving. It also needs to distinguish “not here” from “not found”: checking one area should not end the search while other permitted locations remain.
For a narrowly specified task, such as placing an object at a target location, success can often be verified directly. For our planner, finding the object is only part of success: it might achieve that outcome while skipping confirmation, violating a restriction, or making unnecessary tool calls. A reward must account for these requirements, but a score alone does not explain which decisions should change.
As we add tasks, the guidelines grow. So does the list of ways to violate them. Putting more instructions and exceptions into the system prompt becomes difficult to maintain. We want the planner to learn from corrections and carry that learning into future interactions.
This led us to study how to train a model from verbal feedback. A capable judge can review an attempt and explain what needs to change. The challenge is turning that explanation into an effective training signal for the model we deploy.
We turn verbal feedback into a distillation loss. We compare a teacher model’s predictions with and without feedback, then use that change to adjust the student’s own predictions. We also train on feedback-guided continuations, so the student can learn what comes after a corrected decision. With this approach, we substantially outperform the baselines in both knowledge acquisition and robot planning. The trained student acts without the feedback at inference time.
A numerical reward leaves the correction implicit
One way to train against a behavioral specification is to score each requirement, assign weights, and combine the results into a reward. The agent receives a higher reward when its actions satisfy more of the specification.
The hard part is assigning credit. A rollout is the model’s complete generated attempt, including its text and tool calls. Suppose the agent moves without permission, searches the first area correctly, then stops before checking the remaining permitted areas. Which choices should training reinforce, and which should it discourage?
Common RL recipes such as GRPO1 apply the same reward-derived advantage to every token in an attempt. Across many attempts, the model can learn which decisions lead to higher rewards. But the reward itself leaves much of that work implicit. We can build more detailed reward functions, at the cost of more engineering for each clause and failure case.
A judge can instead provide a correction like this:
You started moving without the supervisor’s permission, then reported “not found” after checking only one area. Ask for permission and wait for approval before moving. If the object is not in the first area, continue searching the remaining permitted areas before reporting “not found”.
This feedback identifies the missing permission and explains how to continue when the object is not in the first area. A scalar reward compresses these instructions into a measure of how well the attempt went. Verbal feedback preserves them for the learner.
Turning feedback into a training signal
Distillation trains a student, the model we want to improve, using the predictions of a teacher. In the robot setting, the student is our agent. We use a frozen copy of the student’s initial model as the teacher, and a separate judge supplies verbal feedback.
At each position in a response, the student and teacher assign probabilities to possible next tokens. These probabilities provide a more detailed signal than a score for the whole response.
In on-policy distillation2,3, the student generates the response. The teacher then scores possible next tokens at each prefix: the text generated up to that point. Training moves the student’s predictions toward the teacher’s at the situations the student actually encounters.
Feedback gives the teacher information the student did not have when it made the attempt. A judge reviews the rollout against the task criteria and writes a critique. We give that critique to the teacher, whose predictions now reflect the additional information. Figure 3 shows how this feedback enters the distillation loop.
The judge and teacher have different jobs. The judge can be a frontier model accessed through a text API. It can spend time reasoning about the attempt and the criteria before writing its feedback. We do not need its token probabilities. The teacher needs to expose probabilities over the student’s vocabulary and interpret the feedback at each prefix.
Designing that loss involves two choices: what target should the student learn from, and on which continuations should we evaluate it? We address each in turn.
1. Measure what the feedback changes
We first need a target for the loss. Standard OPD uses the teacher’s predictions directly as its target. In our setting, the teacher has already received feedback, but its predictions still reflect many preferences at once. Some concern the correction. Others concern phrasing, formatting, or habits we did not ask the student to learn.4
For the object-finding task, the feedback should favor asking for permission over moving immediately. Differences in unrelated wording should not become a reason to retrain the student.
An alternative is to build a corrected target around the student’s own predictions. The teacher then supplies guidance about how to adjust those predictions.
To find that guidance, we score the same prefix twice with the same teacher. Both versions see the task and the student’s original attempt; one also sees the feedback. The difference tells us how the teacher responds to that feedback. Figure 4 illustrates this comparison for the permission requirement.
How should we measure that change? Prior contrastive methods use log contrast: the difference between two log-probabilities.5 Applied to our feedback setup, it scores tokens by the proportional change in the teacher’s probability when feedback is added.
We propose using relative distance for this comparison. It measures how much of the remaining gap to certainty or impossibility the feedback closes. We use this score to adjust the student’s probabilities and form the corrected target. Before explaining the math, let’s see what this choice changes.
Learning from an imperfect teacher
We built a small experiment to see how the choice of training target affects what the student learns. There are just three possible answers, one correct. We simulate a teacher that receives useful feedback but also makes noisy changes to the probabilities of wrong answers.
The interactive Figure 5 compares the OPD target (R), which matches the teacher directly, with two corrected targets: one built using log contrast (A) and one using relative distance (Z, ours). Three small neural networks train with the same starting weights, examples, and loss. Only the target changes.
The controls vary how much the teacher knows before feedback and how strongly it responds to feedback. The curves show the learning progress at your chosen setting; the maps show the final result across both controls. Darker green means a higher probability of the correct answer.

Static preview of a completed training run.
Relative distance stays green across the whole map. Its average final correct-answer probability remains near certainty at every tested setting. Log contrast works well when the teacher responds strongly to feedback, but can fall below chance when a capable teacher responds weakly. With the OPD target, the student largely inherits the teacher’s confidence. Relative distance can learn from the correction and become more confident in the right answer than the teacher supplying it.
Learning facts the teacher did not know

We also tested this transfer with an LLM, on a real task. Trivia Fantasy contains 20 invented facts known only to the judge. The judge explains the correct answer in feedback, which the teacher turns into token-level supervision for the student.
For the student, Qwen3.5-2B, the task is simply to memorize these facts. We test recall on the same questions without feedback. Our full method reaches near-complete recall, while the baselines learn substantially less.6
What should count as stronger evidence?
Suppose feedback changes two of the teacher’s probabilities:
- Correct token: 95% → 98%. A proportional increase of about 3.2%.
- Wrong token: 1% → 1.2%. A proportional increase of 20%.
Log contrast measures proportional growth, so it gives the wrong token the higher score.
Relative distance measures how much of the remaining gap the feedback closes. The correct token moved 60% of the way from its starting probability to certainty. The wrong token closed only about 0.2% of its own remaining gap.
Our confirmation score applies this idea to both increases toward certainty and decreases toward impossibility. For a candidate next token , write the teacher’s probability before feedback as and its probability with feedback as . For , the score is
An increase is divided by the probability left to gain; a decrease is divided by the starting probability. For the correct token above, this gives . The score stays between −1 and 1, with unchanged probabilities receiving zero.
Why this score? It turns out that’s the only ordering that fits the rules.
We treat a possible next token as a hypothesis, and the feedback as evidence for or against it. The teacher’s probability before feedback is the prior; its probability afterward is the posterior.
To compare those probabilities, we use four requirements from probabilistic confirmation:
- Use the before-and-after probabilities. The score should depend on those two values.
- More support should give a higher score. For a fixed starting probability, a higher probability after feedback should count as stronger confirmation.
- Measure decreases proportionally. When feedback lowers a probability, the score should depend on the fraction of its original probability that remains.
- Keep a hypothesis and its opposite consistent. If the feedback supports token A more strongly than token B, it should count against “anything except A” more strongly than “anything except B.”
Together, these requirements determine an ordering of the tokens, as established by Crupi and Tentori’s representation theorem8. They leave the numerical scale free. The relative-distance score used above follows this ordering.
The complement requirement explains the distinction. Feedback strongly lowers the probability of “anything except the correct token,” from 5% to 2%. It barely changes “anything except the wrong token,” from 99% to 98.8%. Yet the log-probability contrast ranks both the wrong token and its complement higher than their counterparts. This violates the requirement that those rankings reverse.
Build the target around the student
We use these scores to reweight the student’s current next-token probabilities. If the student assigns probability to token , we construct the corrected target as
The parameter controls how strongly feedback changes the target.
In the numerical example above, both tokens have positive scores, and the correct token gets the larger boost. Log contrast reverses that ranking. Throughout training, we rebuild the target from the student’s current predictions. Learning can therefore continue even after the student becomes more confident than the teacher.
This construction has a useful property: if feedback leaves the teacher’s predictions unchanged, the target equals the student’s current distribution. There is no local distillation update at that prefix. Because the scores are bounded, the target’s change relative to the student is also bounded for a fixed reweighting strength. The resulting target reflects the feedback through the teacher’s interpretation.
2. Learn how to continue after the correction
The second choice is where to evaluate the loss. Even with a better target, training only on student rollouts can leave much of a critique unused.
Returning to the object-finding task, the judge asks the agent to get the supervisor’s permission before issuing a navigation call and to keep searching if the object is not in the first area. Two illustrative continuations show why the source of the rollout matters:
| Student attempt | Feedback-guided teacher attempt |
|---|---|
| Navigate without permission | Ask for permission and wait for approval |
| Search the first area; object not found | Navigate and search the first area; object not found |
| Stop and report “not found” | Continue to the next permitted area |
The histories diverge at the first decision. On the student’s attempt, the teacher can favor asking for permission before the navigation call. Later prefixes, however, follow a decision to move without approval. Once the student ends the search, there are also no further search steps to learn from.
Figure 3 shows the loop with a student-generated attempt. To evaluate our loss, we allocate part of the rollout pool to the teacher conditioned on feedback. The teacher’s attempts can provide prefixes after the supervisor’s approval and during a continued search. Both kinds of rollout train the student against the corrected target. This provides supervision on how to continue before the student can reliably generate the whole sequence itself.
An objective that covers both paths
We want the student to learn both from its own attempts and from continuations suggested by feedback. Here, is the probability that the current student generates a complete rollout . The corrected distribution, , gives the probability of that same rollout if we use our feedback-corrected next-token predictions at every step. Each rollout probability is the product of the corresponding token probabilities along that rollout.
KL divergence measures disagreement between two probability distributions. It is asymmetric: each direction averages over rollouts from the distribution on its left. Our ideal objective adds the two directions:
We aim to minimize this sum, also known as the Jeffreys divergence, which is zero when the student and corrected rollout distributions agree. The first term provides supervision at prefixes visited by the student. The second provides supervision at prefixes visited by the corrected policy, including continuations the student rarely generates.
We estimate both terms using the same pool of student and feedback-conditioned teacher rollouts. Pooling lets every rollout contribute to both parts of the loss, giving us more learning signal from the same rollout budget. Importance weights account for how each prefix was sampled. You can find the derivation and practical approximations in our paper.
Putting this together, here is how we train on a task:
- The student makes an attempt. It receives the task and generates a rollout without feedback.
- The judge reviews the attempt. It checks the rollout against the task’s requirements and writes feedback explaining what should change.
- The teacher generates additional attempts. It receives the task, the student’s attempt, and the judge’s feedback, then generates a rollout. We pool its feedback-guided rollouts with the original student rollout.
- We turn the feedback into training targets. At each prefix in the pool, we compare the teacher’s next-token predictions with and without feedback. The confirmation scores from this comparison reweight the student’s current predictions to form the corrected target.
- We update the student. We train its predictions toward these targets using both directions of the loss. Every rollout in the pool contributes to both terms.
Following the robot’s behavioral requirements
For our robots, learning from feedback needs to carry over to new tasks. We tested this in EmbodiedEval, our internal benchmark, where the agent carries out requests through tool calls in conversations with a simulated user.
We trained Qwen3.5-4B and evaluated it on 240 held-out tasks across 12 suites. A task passes when the evaluation judge detects no violation of its requirements. We report the mean pass rate across suites.
We reduced the mean failure rate by roughly a third compared with the strongest baseline.
Conclusion
A useful correction can identify a mistake, explain a requirement, supply a missing fact, or describe the next attempt. Our method uses the teacher’s response to that information to construct a training target, and feedback-guided continuations to teach the student how to carry out the correction. The student can then use what it learned without receiving the critique at inference time.
For Alquist, the appeal is practical: the person supervising a robot can explain what should change in the language they already use. We want those explanations to become part of how the robot behaves on future tasks, so it can keep learning as its tasks and requirements change.
You can find the full derivation, proofs, benchmark details, and additional experiments in our paper.
Notes and references
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Back to text
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. Back to text
- On-Policy Distillation. Back to text
- Work such as RLCSD shows how giving a teacher extra information can shift the training signal toward style rather than the task. Back to text
- See RLCSD and W2S-OPD. Back to text
- We compare with SDPO and W2S-OPD, adapted to learn from the judge’s feedback. Back to text
- Faint lines show the original evaluation results. Bold curves show an exponential moving average with a smoothing factor of 0.35. Back to text
- Confirmation as Partial Entailment: A Representation Theorem in Inductive Logic. Back to text
- Error bars show 95% confidence intervals for the mean pass rate. Back to text
