We present a reinforcement learning (RL) approach for robust optimisation of risk-aware performance criteria. To allow agents to express a wide variety of risk-reward profiles, we assess the value of a policy using rank dependent expected utility (RDEU). RDEU allows the agent to seek gains, while simultaneously protecting themselves against downside events. To robustify optimal policies against model uncertainty, we assess a policy not by its distribution, but rather, by the worst possible distribution that lies within a Wasserstein ball around it. Thus, our problem formulation may be viewed as an actor choosing a policy (the outer problem), and the adversary then acting to worsen the performance of that strategy (the inner problem). We develop explicit policy gradient formulae for the inner and outer problems, and show its efficacy on three prototypical financial problems: robust portfolio allocation, optimising a benchmark, and statistical arbitrage
We propose a hybrid method for generating arbitrage-free implied volatility (IV) surfaces consistent with historical data by combining model-free Variational Autoencoders (VAEs) with continuous time stochastic differential equation (SDE) driven models. We focus on two classes of SDE models: regime switching models and Levy additive processes. By projecting historical surfaces onto the space of SDE model parameters, we obtain a distribution on the parameter subspace faithful to the data on which we then train a VAE. Arbitrage-free IV surfaces are then generated by sampling from the posterior distribution on the latent space, decoding to obtain SDE model parameters, and finally mapping those parameters to IV surfaces.
Trading frictions are stochastic. They are, moreover, in many instances fast-mean reverting. Here, we study how to optimally trade in a market with stochastic price impact and study approximations to the resulting optimal control problem using singular perturbation methods. We prove, by constructing sub- and super-solutions, that the approximations are accurate to the specified order. Finally, we perform some numerical experiments to illustrate the effect that stochastic trading frictions have on optimal trading.
We study the problem of active portfolio management where an investor aims to outperform a benchmark strategys risk profile while not deviating too far from it. Specifically, an investor considers alternative strategies whose terminal wealth lie within a Wasserstein ball surrounding a benchmarks -- being distributionally close -- and that have a specified dependence/copula -- tying state-by-state outcomes -- to it. The investor then chooses the alternative strategy that minimises a distortion risk measure of terminal wealth. In a general (complete) market model, we prove that an optimal dynamic strategy exists and provide its characterisation through the notion of isotonic projections. We further propose a simulation approach to calculate the optimal strategys terminal wealth, making our approach applicable to a wide range of market models. Finally, we illustrate how investors with different copula and risk preferences invest and improve upon the benchmark using the Tail Value-at-Risk, inverse S-shaped, and lower- and upper-tail distortion risk measures as examples. We find that investors optimal terminal wealth distribution has larger probability masses in regions that reduce their risk measure relative to the benchmark while preserving the benchmarks structure.
We study a general class of entropy-regularized multi-variate LQG mean field games (MFGs) in continuous time with $K$ distinct sub-population of agents. We extend the notion of actions to action distributions (exploratory actions), and explicitly derive the optimal action distributions for individual agents in the limiting MFG. We demonstrate that the optimal set of action distributions yields an $epsilon$-Nash equilibrium for the finite-population entropy-regularized MFG. Furthermore, we compare the resulting solutions with those of classical LQG MFGs and establish the equivalence of their existence.
We develop the optimal trading strategy for a foreign exchange (FX) broker who must liquidate a large position in an illiquid currency pair. To maximize revenues, the broker considers trading in a currency triplet which consists of the illiquid pair and two other liquid currency pairs. The liquid pairs in the triplet are chosen so that one of the pairs is redundant. The broker is risk-neutral and accounts for model ambiguity in the FX rates to make her strategy robust to model misspecification. When the broker is ambiguity neutral (averse) the trading strategy in each pair is independent (dependent) of the inventory in the other two pairs in the triplet. We employ simulations to illustrate how the robust strategies perform. For a range of ambiguity aversion parameters, we find the mean Profit and Loss (P&L) of the strategy increases and the standard deviation of the P&L decreases as ambiguity aversion increases.
We address the Merton problem of maximizing the expected utility of terminal wealth using techniques from variational analysis. Under a general continuous semimartingale market model with stochastic parameters, we obtain a characterization of the optimal portfolio for general utility functions in terms of a forward-backward stochastic differential equation (FBSDE) and derive solutions for a number of well-known utility functions. Our results complement a previous studies conducted on optimal strategies in markets driven by Brownian noise with random drift and volatility parameters.
Solar Renewable Energy Certificate (SREC) markets are a market-based system that incentivizes solar energy generation. A regulatory body imposes a lower bound on the amount of energy each regulated firm must generate via solar means, providing them with a tradeable certificate for each MWh generated. Firms seek to navigate the market optimally by modulating their SREC generation and trading rates. As such, the SREC market can be viewed as a stochastic game, where agents interact through the SREC price. We study this stochastic game by solving the mean-field game (MFG) limit with sub-populations of heterogeneous agents. Market participants optimize costs accounting for trading frictions, cost of generation, non-linear non-compliance costs, and generation uncertainty. Moreover, we endogenize SREC price through market clearing. We characterize firms optimal controls as the solution of McKean-Vlasov (MV) FBSDEs and determine the equilibrium SREC price. We establish the existence and uniqueness of a solution to this MV-FBSDE, and prove that the MFG strategies form an $epsilon$-Nash equilibrium for the finite player game. Finally, we develop a numerical scheme for solving the MV-FBSDEs and conduct a simulation study.
Latency (i.e., time delay) in electronic markets affects the efficacy of liquidity taking strategies. During the time liquidity takers process information and send marketable limit orders (MLOs) to the exchange, the limit order book (LOB) might undergo updates, so there is no guarantee that MLOs are filled. We develop a latency-optimal trading strategy that improves the marksmanship of liquidity takers. The interaction between the LOB and MLOs is modelled as a marked point process. Each MLO specifies a price limit so the order can receive worse prices and quantities than those the liquidity taker targets if the updates in the LOB are against the interest of the trader. In our model, the liquidity taker balances the tradeoff between missing trades and the costs of walking the book. We employ techniques of variational analysis to obtain the optimal price limit of each MLO the agent sends. The price limit of a MLO is characterized as the solution to a new class of forward-backward stochastic differential equations (FBSDEs) driven by random measures. We prove the existence and uniqueness of the solution to the FBSDE and numerically solve it to illustrate the performance of the latency-optimal strategies.
A risk-averse agent hedges her exposure to a non-tradable risk factor $U$ using a correlated traded asset $S$ and accounts for the impact of her trades on both factors. The effect of the agents trades on $U$ is referred to as cross-impact. By solving the agents stochastic control problem, we obtain a closed-form expression for the optimal strategy when the agent holds a linear position in $U$. When the exposure to the non-tradable risk factor $psi(U_T)$ is non-linear, we provide an approximation to the optimal strategy in closed-form, and prove that the value function is correctly approximated by this strategy when cross-impact and risk-aversion are small. We further prove that when $psi(U_T)$ is non-linear, the approximate optimal strategy can be written in terms of the optimal strategy for a linear exposure with the size of the position changing dynamically according to the exposures Delta under a particular probability measure.
