Differentiable Hybrid-Action Neural Feedback Control for District Heating Networks

arXiv:2610.01822 · eess.SY, cs.SY · Submitted 2026-10-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "Differentiable Hybrid-Action Neural Feedback Control for District Heating Networks".

Rosa: Many cyber-physical systems require control policies that combine continuous setpoints with discrete operational decisions, such as equipment switching, mode selection, or resource scheduling.

Dev: First, who's behind it and why it matters.

Title and authors: Rosa: So let's talk about the title and who wrote this paper; it’s "Differentiable Hybrid-Action Neural Feedback Control for District Heating Networks." This title immediately tells us we’re dealing with a control problem that has both continuous targets and discrete decisions, which is a common hurdle in complex cyber-physical systems.

Dev: The authors are Kirsch, Sgadari, La Bella, and Ferrari-Trecate. Their background seems to be in control systems and hierarchical architectures, which makes sense given the topic of addressing these mixed control problems.

Taro: I'm interested in the authors' perspective on why they chose this specific architecture; do they think a single neural network structure is inherently better than separate continuous and discrete solvers?

Rosa: They argue that conventional methods either model dynamics in detail while keeping a purely continuous formulation or try to incorporate integer variables using simplified system models, but these approaches often require repeatedly solving complex mixed-integer nonlinear programs online, which becomes computationally prohibitive for real-time deployment.

Dev: That’s the pain point; the computational burden associated with repeatedly solving those complex mixed-integer nonlinear programs online is a major challenge when you need quick responses.

Taro: So, their main thrust seems to be moving away from that iterative optimization approach and towards an alternative strategy based on offline learning, which I think is where they're making their contribution.

Rosa: Exactly; they propose the Hybrid-Action Neural Controller, HANC, which leverages differentiable categorical relaxations for end-to-end training through closed-loop trajectories so the policy can be trained via backpropagation through time over those rollouts.

Dev: So, it's about using that mechanism to allow us to train policies directly from data or simulation rollouts without needing constant online MINLP solvers.

Taro: I’m hoping this means we can achieve a much more flexible control system that can handle the nuances of real-world behavior without being limited by the specific assumptions of a simplified model.

Rosa: That flexibility is what they aim for; they demonstrate its application to a complex, full-scale district heating network, showing that it works in practice even on systems with detailed dynamics and time-varying energy prices.

Dev: It’s impressive that they managed to apply this framework to such a specific and complex system like a DHN where integer decisions and nonlinear dynamics are usually tackled separately.

Taro: The scope of the application itself suggests that this isn't just a theoretical exercise; it has real-world potential for industrial applications in energy management.

Rosa: It’s definitely promising; we need to see how this translates from a simulation environment to actual operational systems where things are more messy than a controlled lab setting.

The paper's summary: Dev: So, the paper summarizes their main findings by explaining that they developed the Hybrid-Action Neural Controller, HANC which jointly outputs categorical actions and continuous setpoints. Basically, it shows how this structure can satisfy complex actuator constraints by design.

Rosa: They explain that they use a continuous branch for generating real-valued internal control variables using a neural network, N Nc, which enforces box constraints by construction using the sigmoid function: c,t = sigma(v t)(u-u) + u (four).

Taro: That continuous branch seems to handle the smooth aspects of the control, ensuring that we always stay within physical bounds defined by those box constraints, which is a good starting point for any physical system.

Dev: And then they introduce a discrete branch and an assembly layer A that maps these internal variables to the plant actuator inputs u, creating a unified policy output.

Rosa: The real innovation here is how they handle the categorical decisions using differentiable relaxations, specifically employing the Straight-Through Gumbel estimator to enable training by backpropagation through time over full closed-loop rollouts.

Taro: So, it means they’ve figured out a way to make the discrete choices trainable via gradient methods, which is a significant technical hurdle in hybrid systems research.

Dev: And they use this differentiable relaxation to compute gradients through the relaxed action j,t = delta j j,t using the Gumbel-Softmax relaxation sample j,t, which allows end-to-end training of the full hybrid policy.

Rosa: So in essence, they’ve combined these pieces to create a controller that can generate both the continuous setpoints and categorical decisions together while satisfying those complex constraints by design.

Taro: It sounds like the core contribution is showing how to move from intractable online optimization problems to a tractable end-to-end training framework using this hybrid approach.

Dev: That’s the main takeaway: they developed a policy that operates as a causal feedback policy at deployment, requiring only current measurements and internal memory, which is very practical for real-time use.

Rosa: It really shows how sophisticated the training process can be when you integrate these components into one cohesive architecture.

The paper's improvements: Taro: Moving on to the specific enhancements they propose, I’m curious if there are any specific tweaks they suggest that would make this hybrid action neural controller even better in practice for a real-world setting.

Dev: They discuss several key training loss components that are weighted sums, including economic cost terms to minimize a normalized version of operating cost defined in a regret-like fashion relative to per-scenario envelopes. This allows arbitrage opportunities to be visible where switching between energy sources can result in cost < zero.

Rosa: Beyond the economic aspect, they also include physical-consistency penalties designed to penalize trajectories that are physically inconsistent or those lying outside the plant’s realizable operating envelope using smooth one-sided penalties for things like heat delivery and storage realizability.

Taro: Those consistency terms are vital because they ensure that even if the AI tries to learn something weird, it't forced to stay within what the plant can physically do, which is a necessary safeguard against learning impossible behaviors.

Dev: They also have operative constraints enforced through penalties to keep supply temperatures above a minimum threshold like T sup. And then there’s switching regularization, sw, applied to the discrete second-order difference of the relaxed weights for storage and electric boiler gates.

Rosa: That switching regularization term is particularly interesting because it actively discourages rapid oscillations and actuator chattering on the real plant, which addresses a very common issue in physical systems that we see when you deploy these types of controllers.

Taro: So, they aren't just focusing on the main performance metric; they are explicitly building in mechanisms to ensure operational stability during deployment, which is smart engineering practice.

Dev: It’s interesting that they also showed that injecting Gumbel noise during training improves categorical decision margins by producing a bimodal logit distribution whose two modes sit far from the boundary, indicating substantially more confident decisions compared to the deterministic straight-through estimator.

Rosa: That suggests that noise injection isn't just a trick; it’s a way to train the system to be more robust and less sensitive near switching boundaries in real-world operation.

Taro: So, by incorporating these regularization terms and training techniques, they are addressing not just performance but also stability issues that arise when you try to deploy these complex policies into a noisy physical environment.

Conclusion: Rosa: To wrap up our discussion on the Differentiable Hybrid-Action Neural Feedback Control for District Heating Networks, we’ve seen how this HANC framework successfully integrates continuous setpoints and discrete operational decisions using novel differentiable training techniques.

Dev: The main results were a thirty percent operating-cost reduction over an industrial rule-based baseline when tested on a high-fidelity simulation model distinct from the training model.

Taro: I think the most significant implication is that this method provides a robust way to generate hybrid control actions that are guaranteed to satisfy complex actuator constraints by construction, which simplifies deployment significantly.

Rosa: It also shows that training via backpropagation through time over full closed-loop rollouts allows for end-to-end policy learning directly from data or simulation rollouts without needing expensive online solvers.

Dev: The controller operates causally using only current measurements and internal memory, which means it doesn't need any external forecasts at deployment.

Taro: So, the paper’s conclusion is that this method offers a practical pathway to deploy complex AI control systems in physical environments by focusing on guaranteed constraint satisfaction and robustness through careful training design.

Rosa: It’s definitely a framework worth following for anyone working on hybrid control policies in energy management because it shows how to handle these tricky decisions effectively.

Dev: I'm looking forward to seeing how this methodology scales up from the simulation environment to a live district heating network, keeping the loop rate and latency in mind.

Taro: And I’ll be watching closely for extensions into scenarios where things go completely out of control, because understanding its limits is just as important as seeing its successes.

Nicolas Kirsch, Corrado Sgadari, Alessio La Bella, Giancarlo Ferrari-Trecate

eess.SY, cs.SY

Submitted: 2026-10-01

Updated: 2026-10-01

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: Many cyber-physical systems require control policies that combine continuous setpoints with discrete operational decisions, such as equipment switching, mode selection, or resource scheduling.

Key concepts

Hybrid Action Neural Controller (HANC)
A controller designed to output both continuous control signals (like temperature setpoints) and discrete decisions (like switching equipment). It combines a continuous branch for real numbers and a discrete branch for categorical choices, which are then assembled into the final actuator inputs.
Differentiable Training Mechanism
This technique allows the neural network to be trained using backpropagation over full closed-loop simulations. It uses methods like the Straight-Through Gumbel estimator to approximate discrete selections with continuous gradients, enabling learning of complex hybrid policies without relying on hard, non-differentiable steps.
Straight-Through Gumbel Estimator
A specific mathematical tool used to handle the discrete decisions in the network. It replaces a hard one-hot encoding with a relaxed sampling operation that allows gradients to flow backward through the selection layer during training, making it possible to optimize the entire hybrid system.
Switching Regularization ($\ell_{sw}$)
A penalty term added to the loss function that discourages rapid changes or 'chattering' in discrete actions. This prevents the controller from causing excessive wear or instability on physical equipment by penalizing large, sudden differences between consecutive categorical decisions.

Terminology

Summary

Many cyber-physical systems require control policies that combine continuous setpoints with discrete operational decisions, such as equipment switching, mode selection, or resource scheduling. This research proposes a hybrid-action neural controller (HANC) that jointly generates continuous commands and categorical decisions to satisfy complex actuator constraints by construction.

The gist

The proposed framework is deployed on a district heating network (DHN) featuring multiple heat generation units and stratified thermal energy storage. Its performance is evaluated on a simulation of a real DHN located at RSE SpA in Italy, where the learned controller reduces operating cost by 30% compared to a rule-based industrial baseline under dynamic electricity pricing.

Hybrid Action Neural Controller (HANC) Architecture

The HANC architecture is designed to jointly output categorical actions and continuous setpoints. It comprises three main components: a continuous branch, a discrete branch, and a differentiable assembly layer A that maps these internal variables to the plant actuator inputs u. The continuous branch generates real-valued internal control variables u˜c,t using a neural network NNc, which enforces box constraints by construction using the sigmoid function: u˜c,t = σ(vt)(umin − umin) + umin (4).

Differentiable Training Mechanism

To enable backpropagation through time (BPTT) over full closed-loop rollouts, the discrete decisions are handled using differentiable relaxations. The paper employs the Straight-Through Gumbel estimator, which replaces the hard one-hot encoding in the forward pass with a Gumbel-max sampling operation (10) to return a hard one-hot encoding s˜j t, but computes gradients through the relaxed action ˆδ j t = δ j⊤sˆ j t using the relaxed sample sˆ j t from the Gumbel-Softmax relaxation (11). This allows gradients to propagate through the selection layer and assembly mappings, enabling end-to-end training of the full hybrid policy.

Application to District Heating Networks (DHN)

The HANC controller is applied to a complex DHN benchmark featuring a gas boiler (GB), an electric boiler (EB), and stratified thermal energy storage (TES). The system modeling uses a topology-guided physics-informed approach, where the network is decomposed into interconnected component models, including GRU-based submodels for loads and distribution pipes. The training is performed via empirical optimization on sampled scenarios from a set S, replacing the expected cost with its empirical counterpart (13), allowing for unconstrained stochastic program solving via BPTT.

Performance and Noise Injection Analysis

The HANC controller achieves a 30% operating-cost reduction over an industrial rule-based baseline when tested on a high-fidelity simulation model distinct from the training model, demonstrating robustness to model mismatch. Furthermore, injecting Gumbel noise during training is shown to improve categorical decision margins; the Gumbel ST controller produces a bimodal logit distribution whose two modes sit far from the boundary, indicating substantially more confident decisions compared to the deterministic straight-through estimator, which exhibits significantly more switching behavior in the electric boiler power control.

Key Training Loss Components

The total training loss L is a weighted sum of several dimensionless terms:

  1. Economic cost (15): This term minimizes a normalized version of operating cost, defined in a regret-like fashion relative to per-scenario envelopes, allowing arbitrage opportunities to be visible where switching between energy sources can result in "lcost < 0."

  2. Physical-consistency penalties: These penalize trajectories that are physically inconsistent or lie outside the plant’s realizable operating envelope using smooth one-sided penalties (e.g., for heat delivery and storage realizability).

  3. Operative constraints: Penalties are used to enforce constraints such as keeping supply temperatures above a minimum threshold (e.g., Tmin sup).

  4. Switching regularization: A penalty term, lsw, is applied to the discrete second-order difference of the relaxed weights of the storage categorical and of the two electric boiler gates, which discourages rapid oscillations and actuator chattering on the real plant.

Conclusion

The HANC framework successfully generates hybrid control actions satisfying prescribed input constraints by design, reducing Problem 1 to an unconstrained optimization over θ. The method demonstrates that noise injection during training makes the learned controller more robust to changes in operating conditions at deployment. The controller is causal and operates using only current measurements and internal memory, requiring no forecasts at deployment.

How it works

The HANC architecture jointly outputs categorical actions and continuous setpoints by combining a continuous branch (NNc) and a discrete branch (NNd), mapped by an assembly layer A. The continuous branch enforces box constraints using the sigmoid function: u˜c,t = σ(vt)(umin − umin) + umin.

Improvements for AI systems

Here are the specific improvements and capabilities that can be derived from the proposed Hybrid-Action Neural Feedback Control (HANC) system for AI systems:

The HANC architecture, leveraging differentiable categorical relaxations (like the Straight-Through Gumbel estimator), allows AI systems to transition from purely continuous control policies to complex, hybrid decision-making frameworks in dynamic environments.

Here are the specific improvements and capabilities:

  1. textbfEnd-to-End Hybrid Control Synthesis (The HANC Architecture):

  2. The system can jointly generate continuous setpoints (e.g., temperature targets) and discrete operational decisions (e.g., equipment switching, mode selection) from a single neural network structure. This eliminates the need for separate, complex optimization layers or explicit mixed-integer formulations during real-time operation.

  3. textbf Guaranteed Constraint Satisfaction by Design:

  4. The assembly layer (A) is specifically designed to map internal continuous and categorical variables into the final plant input in a way that satisfies complex actuator constraints (e.g., logical dependencies like AND gates, or physical bounds like box constraints). This means the controller is guaranteed to produce physically realizable commands, reducing the risk of deploying an infeasible control policy.

  5. textbf Training via Backpropagation Through Time (BPTT) over Closed-Loop Trajectories:

  6. The system can be trained using standard deep learning optimization techniques (gradient descent via BPTT) by employing differentiable relaxations (like the Gumbel-Softmax trick). This enables end-to-end policy learning directly from data or simulation rollouts, bypassing the need for computationally expensive online Mixed-Integer Nonlinear Programming (MINLP) solvers.

  7. textbf Enhanced Robustness to Model Mismatch:

  8. Because the training leverages a physics-informed surrogate model (like a GRU/PhysicsNet surrogate), and the architecture is trained via BPTT over long closed-loop trajectories, the resulting policy exhibits superior transferability to slightly different, more accurate plant models compared to purely open-loop schedule methods.

  9. textbf Optimal Exploitation of Time-Varying External Information (Context Awareness):

10.The controller can ingest real-time contextual information (e.g., dynamic electricity pricing, meteorological data) directly as input features, allowing it to adapt its hybrid strategy based on current market conditions or demand forecasts without needing a separate prediction layer for the control policy itself.

11.textbf Superior Arbitrage Identification:

  1. The system is explicitly trained to identify and exploit time-based arbitrage opportunities (e.g., charging storage during low-price windows and discharging during high-price windows) by minimizing a loss function specifically designed to capture these cost differentials, leading to significant operational cost reductions (up to 30% in the DHN case).

13.textbf Increased Decision Margin Confidence:

  1. By using Gumbel noise injection during training, the system learns policies that are less sensitive to small perturbations near switching boundaries. The resulting learned logit distributions show larger decision margins, meaning the policy is more confident in its discrete choices, leading to smoother and more predictable actuator behavior compared to deterministic methods that exhibit high-frequency chattering.

In summary, an AI system utilizing HANC can perform:

  1. Jointly optimize complex physical processes (continuous setpoints + discrete switches).

  2. Generate guaranteed physically feasible control actions at deployment.

  3. Learn sophisticated, long-horizon strategies (like energy arbitrage) by training on simulation trajectories using differentiable methods.

  4. Produce a controller that is robust to model inaccuracies and more stable against noise during real-world operation due to learned decision margins rather than purely hard thresholds.

Abstract

Many cyber-physical systems require control policies that combine continuous setpoints with discrete operational de- cisions, such as equipment switching, mode selection, or resource scheduling. Discrete actions are not differentiable, which ob- structs gradient-based policy training, while conventional mixed- integer formulations remain costly to solve online. This paper proposes a hybrid-action neural controller (HANC) in which a continuous branch, a categorical branch and a differentiable assembly layer jointly generate commands that satisfy complex actuator constraints by construction. Categorical decisions are handled using a straight-through Gumbel estimator, enabling the policy to be trained by backpropagation through time over full closed-loop rollouts. The proposed framework is deployed on a district heating network (DHN) featuring multiple heat generation units and stratified thermal energy storage. Its performance is evaluated on a simulation of a real DHN located at RSE SpA in Italy. The resulting policy jointly learns switching decisions and continu- ous operating setpoints. Under dynamic electricity pricing, the learned controller reduces operating cost by 30% compared to a rule-based industrial baseline. We also show that, compared with a deterministic straight-through relaxation, injecting noise during training achieves similar cost while reducing hard switching by an order of magnitude, and attribute this difference to the wider decision margins of the resulting policy.

Sources

Related papers