Deep reinforcement learning for optimal trading with partial information

The gist The quality and structure of the information supplied to the agent are crucial for improving both profitability and robustness of reinforcement learning–based trading strategies

Optimal Trading Problem Formulation

The optimal trading problem involves a risk-neutral investor aiming to maximize expected discounted profits from trading a signal (St)t≥0 The signal St follows an Ornstein-Uhlenbeck process with time varying parameters, satisfying the stochastic differential equation dSt = κt(θt − St) dt + σt dWt To overcome the issue of latent parameters, the approach takes from partial information stochastic control and filters θt, κt, σt from observations of (Su)u∈[0,t] The reward rt from a trading action at time t is defined as rt(qt, St, St+1, λ):= It+1 (St+1 − St) − λ qt The trader's performance criterion is the expected future discounted sum of rewards E"X∞ t=0 γ t rt(qt, St, St+1, λ) I0, S0 >

Learning Algorithms and Architectures

The problem faced by the algorithm is two-fold: using observed data to filter latent parameters and finding the optimal action given current estimates The approach considers three main strategies for this optimization

1. A one-step algorithm, which directly encodes hidden states from the GRU into the RL trader

2. A two-step approach where a model determines posterior probabilities for the latent θt parameter from signal paths, and then these are used to find optimal trading action

3. A third benchmark case where filtering forecasts the next value of the trading signal, which is then used as a feature in the RL algorithm

The GRU network is employed to capture temporal dependencies in the signal The Deep Deterministic Policy Gradient (DDPG) algorithm, an Actor-Critic approach, is used to approximate the optimal trading policy The state of the environment for training includes tuples of (St, It, ot), where ot is the output of a Recurrent Neural Network that takes as input a collection of past values of the signal from time t − W up to time t

Comparison of DDPG Approaches

Extensive simulations and empirical applications reveal performance differences between the proposed algorithms

1. hid-DDPG, which directly encodes hidden states from the GRU into the RL trader, offers intermediate performance with less interpretable strategies

2. prob-DDPG, which makes use of posterior regime probability estimates, achieves superior cumulative rewards and exhibits more interpretable strategies

3. reg-DDPG, which relies on forecasts of the next signal value, provides limited benefits compared to the other methods The results show that embedding probabilistic insights into latent regimes substantially improves both profitability and robustness of reinforcement learning–based trading strategies

Real Market Application

The paper applies these algorithms to a real market pair-trading problem involving two co-integrated assets, Intel (INTC) and Merril Lynch semi-conductors ETF (SMH) The signal is modeled as a stationary process S˜t = α1Ssmh t + α2Sintc t, which is assumed to follow dS˜t = ˜κ(θ − S˜t) dt + ˜σ dWt The prob-DDPG strategy outperforms the others in the test set by achieving the highest average reward, ultimately resulting in the highest cumulative rewards through the selected policy This suggests that providing structured and interpretable information to a RL algorithm significantly enhances its ability to approximate solutions to inter-temporal optimisation problems

Conclusion on Information Quality

It is concluded that when tackling optimisation problems which do not result in closed-form solutions, the quality of information provided to the RL algorithm is crucial for the quality of the solution The approach that consistently works best is to first gather insights into the dynamics of the data-generating process and only then incorporate this information into the features that the agent uses in the RL algorithm Specifically, providing estimates of next signal values does not yield any significant improvement, suggesting that it is more effective to supply general information about the process – as in the hid-DDPG approach – while ensuring that this information remains closely related to actual parameters governing underlying dynamics, as in the prob-DDPG approach The latter proves to be the most effective, delivering the highest rewards when tested on both synthetic and real data

Appendix Details

The appendix provides visual representations of the neural network architectures for all three approaches Figure 14 shows policies chosen by the DDPG agent per level of inventory I and signal St for the prob-DDPG approach Table 4 summarizes the average cumulative rewards and their standard deviation after n = 2,000 trades from the M = 500 test episodes for each approach used The results are consistent with outcomes from numerical experiments discussed earlier

References

Briola, Antonio et al. (2023). Deep Reinforcement Learning for Active High Frequency Trading. arXiv: 2101.07107 [cs.LG] Cartea, Álvaro and Sebastian Jaimungal (2016). “Incorporating order-flow into optimal execution”. In: Mathematics and Financial Economics 10, pp. 339–364 Cartea, Álvaro, Sebastian Jaimungal, and José Penalva (2015). Algorithmic and high-frequency trading. Cambridge University Press Casgrain, Philippe and Sebastian Jaimungal (2019). “Trading algorithms with learning in latent alpha models”. In: Mathematical Finance 29.3, pp. 735–772 Casgrain, Philippe, Brian Ning, and Sebastian Jaimungal (2022). “Deep Q-learning for Nash equilibria: Nash-DQN”. In: Applied Mathematical Finance 29.1, pp. 62–78 Chan, Ernie (2013). Algorithmic trading: winning strategies and their rationale. John Wiley & Sons Cho, Kyunghyun et al. (2014). “Learning phrase representations using RNN encoder-decoder for statistical machine translation”. In: arXiv preprint arXiv:1406.1078 Gârleanu, Nicolae and Lasse Heje Pedersen (2013). “Dynamic trading with predictable returns and transaction costs”. In: The Journal of Finance 68.6, pp. 2309–2340 Hambly, Ben, Renyuan Xu, and Huining Yang (2023). “Recent advances in reinforcement learning in finance”. In: Mathematical Finance 33.3, pp. 437–503 Hamilton, James D. (1989). “A new approach to the economic analysis of nonstationary time series and the business cycle”. In: Econometrica 57.2, pp. 357–384 Kolm, Petter N, Jeremy Turiel, and Nicholas Westray (2023). “Deep order flow imbalance: Extracting alpha at multiple horizons from the limit order book”. In: Mathematical Finance 33.4, pp. 1044–1081 Lillicrap, Timothy P et al. (2015). “Continuous control with deep reinforcement learning”. In: arXiv preprint arXiv:1509.02971 Millea, Adrian (2021). “Deep Reinforcement Learning for Trading—A Critical Survey”. In: Data 6.11. issn: 2306-5729. doi: 10.3390/data6110119 Ning, Brian, Franco Ho Ting Lin, and Sebastian Jaimungal (2021). “Double deep q-learning for optimal execution”. In: Applied Mathematical Finance 28.4, pp. 361–380 Sirignano, Justin and Rama Cont (2021). “Universal features of price formation in financial markets: perspectives from deep learning”. In: Machine Learning and AI in Finance. Routledge, pp. 5–15 Tsantekidis, Avraam et al. (2020). “Using deep learning for price prediction by exploiting stationary limit order book features”. In: Applied Soft Computing 93, p. 106401 Yang, Hongyang et al. (2020). “Deep reinforcement learning for automated stock trading: An ensemble strategy”. In: Proceedings of the first ACM international conference on AI in finance, pp.

1. Bold Header: Posterior Probability Integration in Policy Selection

The improved system can utilize posterior probability estimates of mean reversion regimes as features for optimal trading decisions, as this is shown to yield superior cumulative rewards in prob-DDPG, suggesting the agent will better condition its actions on the most likely regime.

2. Bold Header: Regime Filtering via GRU Network

The system can employ a Gated Recurrent Unit (GRU) network to approximate the posterior probability of a latent parameter, specifically by using it as the function approximant for estimating the probability of being mean reverting to a specific regime, thereby extracting temporal dependencies in the signal and providing structured and interpretable information.

3. Bold Header: Dual-Stage Learning Architecture

The improved system can adopt a two-step learning procedure where it first learns to estimate the latent state (regime probabilities) and then uses these estimates as features to solve the optimal trading problem, separating the filtering and optimal trading tasks for potentially better performance than single-step methods.

4. Bold Header: Robustness Against Market Complexity

By incorporating information about the quality and structure of the information supplied to the agent, the system can achieve greater robustness; specifically, embedding these insights into latent regimes substantially improves both profitability and robustness of reinforcement learning–based trading strategies.

5. Bold Header: Adaptability to Complex Parameter Dynamics

The system can be designed to handle highly complex signal dynamics by incorporating independent Markov chains for the speed of mean reversion and the volatility of the trading signal, which, when modeled as independent Markov chains, allows the agent to learn policies that are more effective in environments where parameters are not constant.