I recently had the opportunity to read a number of machine learning papers on the problem of designing a good motion planner in robotics. Many of these papers came from the autonomous vehicle industry, especially Waymo. The set of problems that arise in the design of a good planner seems quite distinctive, and in many respects different from other machine learning problems. This note summarizes what I learned.

Traditionally, autonomous driving systems have been organized as a pipeline: perception first estimates the current state of the world from sensor data, prediction then forecasts the future behavior of other agents, and planning finally chooses the ego vehicle’s trajectory in light of those forecasts. This decomposition is intuitive and modular, and for many years it provided a useful way to structure both engineering teams and research problems. More recently, however, it has become increasingly clear that the last two stages—prediction and planning—cannot be cleanly separated, because the future behavior of other agents often depends on what the ego vehicle itself will do. For this reason, modern work has begun to couple prediction and planning much more tightly, either through joint models, planning-aware objectives, or unified generative formulations. I will return to this point below in the section titled Joint prediction of others and self.

Core problem

At a high level, the planning problem can be described as follows: at the current time \(t=0\), the robot must choose a future trajectory

where \(t_1,\ldots,t_H\) are nearby future time horizons, and \(s_{\text{ego},t}\) denotes the future state of the ego robot at time \(t\). The chosen trajectory should make progress toward some route-level goal or intermediate waypoint, while remaining safe, feasible, and socially compliant [9,12,13].

The challenge is that the robot is not free to move arbitrarily. It must move within a constrained geometry (e.g. a road network), obey traffic rules (lane boundaries, speed limits, red lights, right-of-way conventions), satisfy its own dynamics and actuation constraints, and avoid collisions with other agents whose future behavior is itself uncertain and often multi-modal. This makes planning a constrained, stochastic, multi-agent decision problem rather than a simple regression problem. Modern autonomous driving stacks therefore typically couple planning with prediction: the ego must reason not only about where it can go, but also about what others are likely to do [5,6,9,11,12].

A common abstract formulation is:

where \(x\) denotes the observed scene context, \(\mathcal{T}_{\text{feasible}}\) is the set of dynamically feasible ego trajectories, \(\tau_{1:M}^{\text{others}}\) are the future trajectories of the other \(M\) agents, and \(J\) is a cost that includes terms for collision risk, route progress, comfort, legality, and other preferences. In practice, exact optimization is generally intractable, so learned systems tend to generate candidate plans and score them, or directly emit a small set of plausible future ego trajectories together with scores [9,12,13].

Terminology

• Ego / ego-vehicle: The robot itself, i.e. the self-driving car.

• Agent: Any intelligent entity capable of motion, such as a car, cyclist, or pedestrian. The ego is sometimes also called the ego-agent.

• Behavior prediction / trajectory prediction: Prediction of the future trajectory of one or more agents, often together with uncertainty estimates. A key difficulty is that the future is usually multi-modal: an agent may brake, continue, turn, yield, or accelerate depending on latent intent and on interaction with other agents [3,4,5,7,8,11].

• State: A vector summarizing the kinematic configuration of an agent, e.g.

with position, heading, speed, and sometimes acceleration.

• Scene context: A representation of the local world state, usually including map geometry, lane connectivity, traffic light state, past trajectories of nearby agents, and possibly raw sensor-derived features. Recent models differ considerably in how they fuse these heterogeneous inputs [4,5,6,10].

In what follows, I summarize a few key ideas and challenges in this space. Some of them apply to robotic planning more broadly, while others are especially characteristic of autonomous driving.

Imitation learning

A natural way to teach a robot what to do in a given situation is to collect demonstrations from an expert and train a model to imitate them. In autonomous driving, one may encode the current state, road structure, traffic control information, and nearby agents into a context representation \(x\), and learn a mapping \(\pi\)

where \(a^\star\) is the expert future trajectory. When the learned policy is deployed directly in this way, the method is usually called behavior cloning (BC). BC is part of the broader family of imitation learning methods [1,2].

A different imitation-learning paradigm is inverse reinforcement learning (IRL). Instead of directly copying the expert’s action, IRL attempts to infer a cost function \(J\) under which the expert behavior is approximately optimal, and then plans at inference time by minimizing expected cost. Schematically,

Conceptually, IRL is attractive because it attempts to recover the principles underlying the expert’s behavior rather than merely mimicking observed outputs. But it is typically harder to formulate, optimize, and scale than plain supervised imitation. As a result, straightforward end-to-end IRL appears less common in deployed industrial driving stacks than one might guess from its conceptual appeal [1,9].

The classic weakness of pure BC is covariate shift or distribution drift. The training set contains expert-visited states, but at test time the learned policy visits states induced by its own past actions. A small action error can move the system into an off-distribution state, where the policy makes a larger error, leading to compounding deviations over time. This issue is fundamental in sequential decision making and was formalized clearly in the DAgger line of work [1].

This is one reason interactive imitation-learning variants such as DAgger are important conceptually: they augment the training set with expert labels on states visited by the learner, rather than only on nominal expert rollouts. In practice, however, online expert relabeling is expensive in real-world autonomous driving, so deployed systems often rely on offline approximations, simulation, perturbation-based data synthesis, or auxiliary losses rather than textbook DAgger itself [1,2].

Behavior cloning plus auxiliary losses

In practice, autonomous driving systems rarely rely on a pure “predict the expert’s exact future” loss alone. A more realistic formulation is to use behavior cloning as one component of a richer objective. For example, one may train a model to produce one or several candidate ego trajectories while also penalizing undesirable outcomes such as collisions, leaving the drivable area, violating route intent, or producing physically implausible motions. ChauffeurNet [2] is an influential example of this philosophy: it combines imitation with perturbations and with additional losses designed to penalize collisions and off-road behavior, thereby exposing the learner to states not commonly seen in expert data.

A generic training loss might look like

This formulation already hints at something important: the planner is not merely a function approximator, but a system whose outputs are judged by structured properties of a rollout. In other words, there is almost always a gap between “match the demonstrated trajectory” and “produce a good trajectory.” Auxiliary losses partly bridge that gap [2] is a good reference that goes into details of several auxiliary losses.

Even when a learned model is used mainly to propose candidate trajectories, a downstream scoring module usually evaluates them against criteria such as:

• probability of collision with others,
• route progress,
• comfort and smoothness,
• lane and traffic-rule compliance,
• consistency over time, to avoid plan jitter.

This candidate-generation-plus-scoring architecture is common because it separates proposal diversity from policy selection, and because explicit scoring provides a place to encode hard-won system knowledge. It also explains why many “prediction” papers are in fact deeply relevant to planning: a planner is only as good as the futures it reasons against.

Multi-modality in trajectory prediction

A central fact about driving data is that the future is rarely single-valued. At an intersection, a car may go straight, turn left, turn right, or slow to yield. Even if the observed past is identical, several futures may all be reasonable. This means that minimizing a unimodal regression loss such as mean squared error can be a bad idea: the model may average across several valid futures and produce a trajectory that no agent would actually take. There have been a few different methods to generate multi-modal predictions (NB: the term ‘multi-modal’ has two meanings – the first is the one we are discussing here, namely have more than one mode or cluster of probability density. The other meaning is data with multiple modalities, i.e. text, audio, images, video – this context is prevalent when talking about modern LLMs).

One early method to produce multi-modal trajectory predictions for agents is the use of anchors. In MultiPath [3,4], the model first predicts a discrete distribution over a finite set of canonical/template future trajectories:

and then regresses a residual offset around each anchor. Formally,

Here the anchors represent coarse behavioral modes, while the residual captures local refinement [3].

This is attractive for several reasons. First, it provides a compact parametric representation of a multi-modal distribution. Second, it often makes training easier by turning part of the problem into classification over a small set of modes. Third, it provides a natural interface to a planner, which often wants a modest number of high-level hypotheses rather than thousands of Monte Carlo samples. MultiPath++ extends this style of modeling with stronger scene encoding and information fusion [3,4].

Another classical way to model multi-modality is to let the network output the parameters of a Gaussian mixture model (GMM), or more generally to use a mixture density network. In this approach, the model predicts mixture weights, means, and covariances for several Gaussian components, thereby representing multiple plausible futures in a compact parametric form. MultiPath can be viewed partly in this way: its anchor probabilities act like mixture weights, while the anchor-conditioned residual and uncertainty predictions define the component distributions. Earlier driving-behavior work also used MDNs more directly to model distributions over future actions or trajectories.

A different response to multi-modality is to model the future trajectory distribution with a generative model. MotionDiffuser [7] is a notable example: it uses a diffusion model to represent the joint distribution of future trajectories over multiple agents, captures highly multi-modal futures without relying on a fixed anchor set, and supports controllable sampling via differentiable cost functions. This makes it interesting not only as a forecasting model, but also as a bridge between probabilistic prediction and planning-oriented sampling.

The drawback is that anchors impose an inductive bias: the true distribution must be representable, at least approximately, by a finite library of modes. If the anchor set is too small or poorly chosen, the model may miss rare but important futures. This tension between tractability and expressiveness is a recurring theme in motion forecasting. Newer diffusion-based, tokenized, or autoregressive approaches try to bypass hand-designed anchor sets while still retaining tractable multi-modal prediction.

Marginal versus joint prediction of agents

A major conceptual distinction is whether we predict each other agent independently, or whether we model all agents jointly. In marginal prediction, one predicts, for each agent \(i\),

often using the full scene context \(x\), but without explicitly constructing a coherent joint distribution over all agents’ futures. This is easier to scale and historically very common. However, it can lead to inconsistencies. For example, one predicted future may assume that agent A yields to B, while another independently predicted future for B assumes the reverse. Each marginal forecast can look plausible in isolation, yet the combination may correspond to no realistic joint scene evolution.

In joint prediction, by contrast, one attempts to model

or at least a structured approximation to it. This is far more aligned with the planner’s needs, because the ego ultimately interacts with joint futures of the scene. Scene Transformer [5] is a good example of a model that explicitly targets joint forecasting, producing scene-level consistent futures rather than separate independent guesses for each agent. MotionDiffuser also fits naturally in this category, since it models a joint multi-agent future distribution using diffusion, while MotionLM emphasizes joint multi-agent rollouts via autoregressive decoding [5,7,8].

The practical tradeoff is that joint prediction is harder. The dimension of the output space grows with the number of agents, and the space of possible interactions grows combinatorially. Nonetheless, from the perspective of planning, joint prediction is conceptually cleaner because it reflects the inherently interactive nature of traffic. This is one reason recent work increasingly blurs the boundary between prediction and planning: once one is generating joint futures anyway, it is natural to ask whether the ego trajectory should be generated as part of the same process.

Behavior prediction using language modeling

A striking recent trend is to treat motion forecasting as a kind of sequence modeling problem analogous to language modeling. The basic idea is to discretize motion into tokens, then train an autoregressive model to predict future tokens conditioned on past scene context. MotionLM is a prominent example: rather than using explicit anchors or latent-variable optimization, it tokenizes trajectories and models them with a language-model-style objective.

This perspective has several appealing features. First, autoregressive factorization naturally expresses multi-modality:

where \(z_h\) are motion tokens. Multi-modal futures emerge because the model can branch probabilistically at each step, rather than committing early to a single anchor. Second, the same machinery naturally extends to joint multi-agent prediction by interleaving or otherwise structuring the token stream across agents. Third, the approach leverages the remarkable scaling properties and flexible conditioning mechanisms of transformer-based sequence models [8].

More broadly, this line of work suggests that motion planning and forecasting may benefit from the same general lesson seen in other domains: once the representation and tokenization are chosen well, large generic sequence models can absorb much of the modeling burden that older systems handled through hand-designed structure. That said, the analogy to language should not be pushed too far. Unlike text, motion planning is safety-critical, physically grounded, and constrained by geometry and dynamics. A planner cannot merely generate a “plausible continuation”; it must generate one that is safe and executable.

Joint prediction of others and self

This brings us to what I think is one of the most interesting developments in the area: the growing view that prediction of others and planning for self should perhaps not be treated as separate problems at all.

In the traditional modular architecture, one first predicts the futures of other agents, then plans the ego trajectory against these predicted futures. This decomposition is intuitive and engineering-friendly, but it also introduces a mismatch: the futures of other agents often depend on what the ego itself will do. A pedestrian may wait if the ego advances, or begin crossing if the ego slows. A car may merge if the ego yields but not otherwise. In such cases, “predict others first, plan ego second” is conceptually awkward because the prediction target is itself conditional on the ego decision [5,9,12].

A more coherent view is to model a joint distribution

and then select or guide the ego component according to planning objectives. Recent papers in planning-oriented and unified motion frameworks move in this direction. Planning-Oriented Autonomous Driving [9] argues that perception, prediction, and planning should be optimized in a more coordinated way, rather than as loosely coupled modules. More recent unified generative approaches model ego and others jointly and then apply guidance or scoring to obtain a final plan [9,12,13].

Conceptually, this is appealing for two reasons. First, it allows interaction to be represented natively rather than bolted on afterward. Second, it aligns learning more closely with the actual downstream decision problem. The hard part, of course, is preserving the advantages of modularity: interpretability, controllability, ease of debugging, and the ability to impose hard constraints. The current state of research seems to me to be a search for the right middle ground: not a completely hand-engineered modular stack, but not an unconstrained end-to-end black box either.

References

1. Ross, S., Gordon, G., and Bagnell, D. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS, 2011.
2. Bansal, M., Krizhevsky, A., and Ogale, A. ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst. RSS, 2019.
3. Chai, Y., Sapp, B., Bansal, M., and Anguelov, D. MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction. CoRL, 2020.
4. Varadarajan, B. et al. MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction. arXiv, 2021.
5. Ngiam, J. et al. Scene Transformer: A Unified Architecture for Predicting Multiple Agent Trajectories. ICLR, 2022.
6. Nayakanti, N. et al. Wayformer: Motion Forecasting via Simple & Efficient Attention Networks. ICRA, 2023.
7. Jiang, C. M., Cornman, A., Park, C., Sapp, B., Zhou, Y., and Anguelov, D. MotionDiffuser: Controllable Multi-Agent Motion Prediction Using Diffusion. CVPR, 2023.
8. Seff, A. et al. MotionLM: Multi-Agent Motion Forecasting as Language Modeling. ICCV, 2023.
9. Hu, Y. et al. Planning-Oriented Autonomous Driving. CVPR, 2023.
10. Shi, S. et al. MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying. arXiv, 2023.
11. Madjid, N. A. et al. Trajectory Prediction for Autonomous Driving. Survey, 2025.
12. A Unified Motion Framework for Simulation, Prediction and Planning. arXiv, 2026.
13. Li, A. et al. End-to-End Predictive Planner for Autonomous Driving with Consistency Models. arXiv, 2025.