Deep Reinforcement Learning solution for pickup and delivery routing problems with time window and capacity constraints

arXiv:2608.14156 · cs.LG · Submitted 2026-08-14 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Deep Reinforcement Learning solution for pickup and delivery routing problems with time window and capacity constraints".

Jane: The paper was written by Andrew Soroka, Alex Meshcheryakov and Sergey Gerasimov from Moscow State University and Space Research Institute of the Russian Academy of Sciences.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: We are starting today with a paper titled "Deep Reinforcement Learning solution for pickup and delivery routing problems with time window and capacity constraints."

Jane: It’s quite a complex title, Tom, but the researchers at Moscow State University are looking at something that affects almost every city on earth.

Tom: They're focusing on the logistics of moving goods, specifically when you have to pick things up and drop them off in a very tight schedule.

Jane: Right, because it isn't just about going from point A to point B; you have to worry about how much space is left in the truck and exactly what time you arrive at each door.

Meng: That capacity part is where things get messy for an engineer because every single item added changes the entire math of the route. If you pick up a heavy crate early on, you might find yourself unable to fulfill a later delivery because the truck is physically full.

Lu: I think we should look at this as the first step toward a truly intelligent, synchronized global supply web. If we can solve these constraints with AI, we aren't just moving boxes; we are creating a living system that breathes with the city's needs.

Jane: That sounds like a massive leap, Lu, but how do they actually handle those strict time windows?

Meng: They have to treat every minute as a precious resource, ensuring the driver doesn't arrive too early or too late, which would break the whole chain of deliveries.

Lalam: There is also a profound environmental benefit to this kind of precision. When routes are optimized this well, we see fewer vehicles idling in traffic and much quieter, cleaner streets for everyone living in those urban centers.

Tom: It seems like a massive coordination puzzle, so let's look at the specific model they built to solve it.

Summary: Tom: Moving into the mechanics of "Deep Reinforcement Learning solution for pickup and delivery routing problems with time window and capacity constraints," the authors used a modified JAMPR model.

Jane: They basically took an existing architecture that uses self-attention and tweaked it to respect all those real-world rules we just talked about.

Tom: So, instead of a simple list of stops, the AI is constantly evaluating the entire context of the problem?

Jane: Exactly, you can imagine it like a driver who isn't just looking at the next street corner, but is constantly scanning every package in the back and every upcoming time window on their dashboard simultaneously.

Meng: I found their "soft cost" approach particularly clever for dealing with those impossible situations. Instead of the whole system crashing if a constraint is violated, they apply a mathematical penalty to the total cost, which allows the model to keep learning even from its mistakes.

Lu: They also used something called masking to act as guardrails for the AI. This ensures the model doesn't even attempt an illegal move, like trying to deliver a package before it has actually been picked up from a depot.

Jane: That sounds much more stable than just letting the model wander around blindly, doesn't it?

Lu: It definitely is, because that masking allows the neural network to build a deep, contextual understanding of the entire problem space in real-time.

Lalam: This reminds me of how we navigate our own lives. We don't just react to what is directly in front of us; we sense the broader context and adjust our path accordingly to find the best outcome.

Tom: That connection between machine logic and human intuition is fascinating, but let's see if this actually works when the problems get huge.

Improvements: Tom: We are now looking at how this model actually performs when you scale it up, specifically in "Deep Reinforcement Learning solution for pickup and delivery routing problems with time window and capacity constraints."

Jane: The researchers tested it on everything from small batches of fifty points to massive sets of one thousand points.

Tom: And the results show a really interesting split depending on that scale.

Jane: For those smaller or medium-sized tasks, the JAMPR model is incredibly fast and actually produces better costs than the traditional heuristic methods people usually rely on.

Meng: I have to bring up the massive elephant in the room, though, which is the training time. We are talking about days or even weeks of running high-end GPUs just to get this model ready for use, which is a huge hurdle for any company wanting to deploy this tomorrow.

Lu: I think that upfront cost is worth it when you consider how robust the model is compared to current tools like OR-Tools. The paper shows that while traditional solvers often fail to find any solution at all for large, complex tasks, this neural model always finds a way through.

Jane: So it's more about being reliable and finding a good path quickly rather than being perfectly optimal for every single massive dataset?

Lu: Precisely, and that reliability is what allows us to build truly autonomous systems that don't break when the real world gets messy or unpredictable.

Lalam: That kind of stability is exactly what builds public trust in new technology. If people know the systems managing our city's infrastructure are reliable and won't just fail during a peak hour, they will embrace them much more readily.

Tom: It sounds like we are trading massive computational effort upfront for incredible reliability and speed once the system is live.

Conclusion: Tom: We have reached the end of our discussion on "Deep Reinforcement Learning solution for pickup and delivery routing problems with time window and capacity constraints."

Jane: It really highlights that tension between heavy development work and nearly instant execution in the field.

Tom: I agree, Jane, because you are essentially paying with GPU hours during training to buy incredible speed during actual operations.

Lu: Looking forward, I am excited to see if they can combine these reinforcement learning models with partitioning techniques or even mixed-integer programming to tackle even larger scales.

Meng: I'll be watching closely to see if researchers can find ways to shrink those training cycles, because that is the real key to making this practical for the logistics industry.

Lalam: Regardless of the training hurdles, this work brings us closer to a world where urban movement is seamless and almost invisible in its efficiency.

Tom: That is a beautiful note to end on, Lalam.

Jane: Thanks for joining us today, everyone; we've had a great time breaking this one down with the whole team.

Tom: We will be back very soon with a brand new paper that takes us from the streets of a city straight into the microscopic world of protein folding!

Moscow State University · Space Research Institute of the Russian Academy of Sciences

cs.LG

Submitted: 2026-08-14

Updated: 2026-08-14

DOI: 10.1134/S1054661823020165

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 71/100

The gist: This paper proposes a deep reinforcement learning solution for the Pickup and Delivery problem with Capacity and Time Window constraints (CPDPTW).

Key concepts

Modified JAMPR model: An architecture that utilizes self-attention to evaluate the entire context of a routing problem at once. Rather than just focusing on the next stop, the AI simultaneously scans all available packages and upcoming time windows to make informed decisions about the most efficient route.
Soft cost approach: A method used to handle constraint violations by applying a mathematical penalty to the total cost. This prevents the system from crashing when a mistake occurs, allowing the reinforcement learning model to continue learning and improving from its errors.
Masking: A technique that acts as guardrails for the AI to prevent illegal moves. For example, it ensures the model does not attempt to deliver a package before it has been picked up from a depot, helping the neural network build a stable understanding of the problem.

Terminology

Reinforcement Learning — 13× in this paper · explained in 4 episodes
Deep Reinforcement Learning — 8× in this paper · explained in 2 episodes
Robustness — 8× in this paper · explained in 5 episodes
Vehicle Routing Problem — 8× in this paper
Combinatorial Optimization — 4× in this paper · explained in 2 episodes
Transformer — 2× in this paper · explained in 2 episodes
Action Space — 1× in this paper · explained in 1 episode
Attention Model — 1× in this paper
Contextual Understanding — 1× in this paper · explained in 2 episodes
Embeddings — 1× in this paper · explained in 2 episodes
Joint Attention — 1× in this paper · explained in 2 episodes
Objective Function — 1× in this paper · explained in 2 episodes

Summary

This paper proposes a deep reinforcement learning solution for the Pickup and Delivery problem with Capacity and Time Window constraints (CPDPTW). It addresses the critical need for a fast (or realtime) route optimizer capable of handling medium-to-large scale problems involving complex real-world restrictions, which often cause classical heuristic solvers to suffer from substantial manual labor or excessive computational loads.

The Problem and Motivation

The Vehicle Routing Problem (VRP) is an NP-hard combinatorial optimization task. In practical logistics, this is complicated by the pickup and delivery (PDP) restriction, vehicle Capacity, and specific Time Windows. While existing tools like Google OR-Tools or the HGS heuristic can solve certain variations, they often struggle with flexibility or feasibility. For instance, powerful solvers like LKH-3 may take over an hour to solve a single task of size 2000, which is inappropriate for many applications such as large courier or municipal services. The authors seek to develop a neural solver that can handle these high-dimensional problems more efficiently.

The Proposed Model

The researchers implemented a modified JAMPR model, which utilizes an encoder-decoder architecture with self-attention. This model treats route optimization as a sequential decision problem modeled as a Markov decision process. To adapt the standard architecture for CPDPTW, the authors modified the feasible function to include an additional mask that limits available customers according to a specific delivery order.

The implementation includes several key modifications:

The use of an additional mask to handle PDP restrictions and delivery orders.
A SOFT setting for cases where visiting all customers is not feasible, using a cost function that is a linear combination of the distance traveled and the number of missed clients.
Empirically chosen coefficients of 13 for distance and 10 for missed clients.

Experimental Performance

The model was tested on problems ranging from 50 to 1000 points, with training times often measured in days. The results demonstrate that performance is highly dependent on problem scale and optimization time.

The findings indicate the following:

For small and medium-sized problems (50–200 points), the model provides a fast optimal solution or a fast suboptimal solution that can outperform metaheuristics in the initial seconds of optimization.
For large-scale problems (400–1000 points), JAMPR provides a much more accurate suboptimal solution than OR-Tools in the first few minutes, although metaheuristics eventually overtake it given enough time.

Robustness and Distribution Stability

A key finding of this research is the superior robustness of the neural approach compared to traditional metaheuristics. The authors evaluated the algorithm's ability to maintain stable results across different task instances and distributions.

The study highlights several aspects of model robustness:

The JAMPR model maintains a zero failure rate, finding solutions for all problem instances regardless of size, whereas the number of unsolved problems for OR-Tools increases as task size grows.
The model is robust to changes in data distribution, maintaining a fast suboptimal solution even when up to 25% of the data points are drawn from a different normal distribution.
The model's predictive ability remains consistent regardless of whether the Euclidean measure or the Manhattan measure is used for distance calculations.

Improvements for AI systems

1. Neuro-Symbolic Hybrid Routing Engine

Improvement: Integrate the modified JAMPR reinforcement learning policy as a warm-start generator for Mixed-Integer Programming (MIP) solvers. Instead of using simple heuristics like PCA, the RL model provides a high-quality, feasible initial solution and a set of candidate edges to prune the branch-and-bound search tree.
Capability: This system will achieve the mathematical optimality guarantees of MIP while drastically reducing the computational time required to find the first feasible solution in large-scale CPDPTW instances, effectively bridging the gap between fast suboptimality and slow optimality.

2. Hierarchical Partitioning Transformer (HPT)

Improvement: Implement a two-tier architecture consisting of a Global Partitioning Transformer and Local JAMPR Agents. The first tier uses a learned partitioning policy to segment high-dimensional node sets into spatially and temporally coherent sub-clusters; the second tier applies the modified JAMPR model to solve each sub-cluster independently.
Capability: This system will scale to extremely high-dimensional logistics problems (>5,000 nodes) that currently exceed the quadratic memory/complexity limits of standard self-attention mechanisms, providing near-optimal routing for massive municipal or global delivery networks in real-time.

3. Hypernetwork-based Multi-Objective Policy

Improvement: Replace the fixed linear cost combination (distance vs. missed clients) with a Hypernetwork that modulates the JAMPR decoder weights based on a continuous preference vector omega.
Capability: This system will allow end-users to dynamically adjust the trade-off between operational cost (distance) and service level (minimizing missed clients) at inference time via a real-time slider, without needing to retrain the model for different business priorities.

4. Adversarial Distributional Robustness Training (ADRT)

Improvement: Incorporate an adversarial training loop where a Disturber network generates non-uniform spatial distributions (e.g., extreme clustering, heavy-tailed outliers, or skewed densities) to challenge the JAMPR encoder during the training phase.
Capability: This system will maintain high routing accuracy and stability in highly unpredictable urban environments where customer demand deviates significantly from uniform or normal distributions, ensuring reliability during sudden shifts in market density or localized demand surges.

Abstract

The task of constructing vehicles optimal routes for pickup and delivery of goods is one of most promising tasks in the context of global urban population growth. Although this kind of problems with small size can be solved by various classical approaches, a fast (or realtime) route optimizer under the constraints of the real world (such as capacity and time windows constraints) for medium-large size problems still remains a highly challenging task. In this work we, for the first time, successfully applied a deep Reinforcing Learning approach (modified JAMPR model) to solve Pickup and Delivery problem with Capacity and Time Window constraints (CPDPTW). We obtained a robust model that gives a fast optimal solution for problems of small and medium size, and gives fast suboptimal solution for problems of larger (> 200) size.