Reoptimization Algorithms for Contextual Bandits with Knapsack Constraints

The paper Reoptimization Algorithms for Contextual Bandits with Knapsack Constraints by Zhen Xu (University of Liverpool) studies new algorithms for Contextual Bandits with Knapsack (CBwK) problems. In these problems, there are finitely many types of customers, products, and resources. Each product is made from a fixed combination of resources, and resources have finite capacity. A decision maker must assign each arriving customer one out of a set of multiple possible products. Every assignment of a customer to a product generates a random reward, which equals an unknown linear function of customer and product features, plus a noise term. The objective is to jointly learn the mean reward function and make online assignments to minimize expected revenue loss relative to an optimal policy that knows the reward function.

The paper proposes a natural and simple extension of the Upper-Confidence-Bound (UCB) family of algorithms and applies re-optimization techniques. The algorithm bridges bandit learning with classical network revenue management by proposing a UCB-guided linear programming (LP) re-solving heuristic. By periodically re-optimizing a fluid LP that is updated with upper confidence bounds of the unknown reward parameters, the algorithm explicitly captures the opportunity cost of capacity via time-varying dual shadow prices.

The main theoretical result is that the algorithm achieves an average regret of O((ln T) cubed / T), where T is the horizon length. The paper states: "We show that by taking advantage of re-optimization, our algorithm achieves an average regret of O((ln T) cubed / T) where T is the horizon length. Our bound significantly reduces the O(1/√T) bound in the literature for closely related dynamic-pricing problems that are based on re-optimization." The regret bound is also compared to other CBwK literature in Table 1, showing the proposed work achieves O(ln T / T) under proportional capacity scaling, improving upon prior bounds of O(√(ln T / T)) or O(√(ln2 T / T)).

The model assumes known and fixed resource consumption per action, but rewards are an unknown linear function of contexts. The paper notes: "we focus on a structured, highly practical variation of the problem: a contextual bandit setting where the resource consumption of each action is fixed and known, but the rewards remain an unknown linear function of the contexts." The objective is to jointly learn the mean reward function and make online assignment decisions to minimize expected revenue loss relative to an optimal full-information policy.

The algorithm (called ON) works as follows: it maintains confidence balls CB t around estimators Û it of the unknown reward parameter U i for each customer type i. In each period t, it solves an optimization problem (5) to obtain probabilities x t, using upper confidence bounds to deal with uncertainty. The feasible region D t is defined by the actual average remaining capacity b kt. After assigning a customer to a product, it collects the reward and updates the estimator and matrix M it via least-squares updates. The capacity allocated to period t+1 is updated via b kt+1 = b kt - (d kt - b kt)/(T - t), where d kt is the realized usage of resource k in period t.

The benchmark is an optimal solution x* to problem (10), which uses the true reward parameters and the true average capacity B/T. The paper proves in Proposition 1 that the average reward per period of any online algorithm is bounded above by R u, the optimal value of (10).

Key assumptions include Assumption 1: There is a unique dual optimal solution to (10). Moreover, the variables (µ* 1,..., µ* K) corresponding to the capacity constraints (12) in this solution are strictly positive. This assumption enables quantifying the regret.

The regret analysis decomposes the total regret into two parts: regret from the estimator of U (bounded in Proposition 7) and regret from re-solving (bounded in Proposition 8). Proposition 7 shows that if each D s is well-approximated and U ∈ CB s for all 1 ≤ s ≤ t, then the sum of pseudo regrets is bounded by 12IJβ t ln(t)/Δ. Proposition 8 bounds the expected number of periods after the stopping time τ as E[T - τ] ≤ 1 + KC2/ϵ2 + KC2/ϵ2 ln(T).

The main theorem (Theorem 1) states: The regret of the online algorithm ON is at most 1+I+ (K3C2C µ2)/Δ2 ln(T) + (252IJ2)/Δ ln3(T). The proof conditions on the event S* = 1 U ∈ CB t, ∀t=1,...,T that the true U is in each confidence ball, which holds with probability at least 1 - δ by Proposition 9.

Numerical experiments compare four algorithms: Re-UCB (the proposed algorithm), UCB (online updating without re-solving), Re-SEP (separated learning and exploitation with re-solving), and SEP (separated learning and exploitation without re-solving). The experiments use I=3 customer types, J=5 products, K=4 resources, with horizon T=10000 and 1000 simulations per instance. The results show that the total expected loss under the reoptimization policy grows sublinearly with T and appears to scale approximately logarithmically. The paper notes: Although the theoretical upper bound is O(ln3 T), the numerical results suggest a much milder growth rate, close to linear in ln T. The average loss per period decreases steadily as T increases and converges to zero, confirming asymptotic optimality on a per-period basis. The results also demonstrate that reoptimization plays a critical role in controlling capacity allocation, while the learning component governs the residual loss.

The paper concludes by noting several directions for future work, including extending the reoptimization methodology to nonlinear reward structures, nonparametric models, or richer action spaces. It also acknowledges that mathematical generality is indeed limited by Assumption 1. Unfortunately, the techniques in this paper do not allow us to relax this assumption.

Improvements to AI Systems:

1. Adaptive Resource-Constrained Decision-Making with Reoptimization:

The AI system can implement a UCB-guided linear programming (LP) re-solving heuristic for sequential decision-making under capacity constraints. Instead of static allocation, the system periodically re-solves a fluid LP using updated upper confidence bounds on unknown reward parameters, capturing opportunity costs via time-varying dual shadow prices. This yields an average regret of O((ln T)3 / T), significantly outperforming prior O(1/√T) bounds. The improved system can handle finite resources (e.g., inventory, bandwidth, budget) while learning customer preferences online, making it suitable for e-commerce pricing, ad allocation, or cloud resource management.

2. Provably Efficient Learning with Known Resource Consumption:

The AI system can exploit the structure where resource consumption per action is fixed and known, but rewards are unknown linear functions of contexts. By maintaining confidence balls around least-squares estimators and updating them via online matrix inversions, the system achieves logarithmic regret in horizon T (O(ln T / T) under proportional capacity scaling). This enables the AI to jointly learn reward functions and make near-optimal assignments, even with limited data, reducing exploration costs in high-stakes settings like personalized healthcare or logistics.

3. Regret Decomposition for Robust Performance Guarantees:

The system can decompose total regret into estimation error (from unknown parameters) and re-solving error (from periodic LP updates). By bounding each component separately (e.g., Proposition 7 for estimation, Proposition 8 for re-solving), the AI can provide tight theoretical guarantees and adjust its reoptimization frequency dynamically. This improves reliability in long-horizon tasks, ensuring the system converges to optimality even when capacity constraints are tight or dual solutions are unique (Assumption 1).

4. Dual Shadow Price–Aware Exploration:

The AI system can use dual variables from the fluid LP to guide exploration, prioritizing actions that consume scarce resources more efficiently. This explicit capacity-awareness reduces wasteful exploration of low-value actions, leading to faster learning and lower cumulative loss. The system can also handle non-stationary resource availability by updating capacity constraints in each period (b kt+1 formula), making it robust to dynamic environments.

5. Scalable Numerical Performance with Logarithmic Growth:

The improved AI system can achieve near-logarithmic average loss per period in practice, as demonstrated in experiments (T=10,000, 1,000 simulations). By combining reoptimization with UCB learning, the system exhibits sublinear total loss that scales approximately as ln T, confirming asymptotic optimality per period. This makes it practical for real-world deployments where T is large (e.g., millions of customer interactions), offering both theoretical rigor and empirical efficiency.

What the Improved AI System Can Do:

• Dynamic Pricing with Inventory Constraints: Assign prices to customers to maximize revenue while respecting limited stock, learning demand curves online.

• Online Ad Allocation: Allocate impressions to advertisers with budget constraints, learning click-through rates in real time.

• Cloud Resource Provisioning: Allocate compute or storage to users with unknown utility functions, respecting capacity limits.

• Personalized Treatment Assignment: Assign treatments to patients with limited medical resources, learning treatment efficacy from contextual features.

• Network Revenue Management: Manage seat or room inventory in airlines/hotels, updating prices dynamically based on demand learning.

The system provides a principled, provably efficient framework for any application where actions consume finite resources and rewards depend on unknown linear functions of contexts, with reoptimization enabling near-optimal performance over long horizons.