Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function

We consider non-convex stochastic optimization problems where the objective functions have super-linearly growing and discontinuous stochastic gradients. In such a setting, we provide a non-asymptotic analysis for the tamed unadjusted stochastic Langevin algorithm (TUSLA) introduced in Lovas et al. (2021). In particular, we establish non-asymptotic error bounds for the TUSLA algorithm in Wasserstein-1 and Wasserstein-2 distances. The latter result enables us to further derive non-asymptotic estimates for the expected excess risk. To illustrate the applicability of the main results, we consider an example from transfer learning with ReLU neural networks, which represents a key paradigm in machine learning. Numerical experiments are presented for the aforementioned example which supports our theoretical findings. Hence, in this setting, we demonstrate both theoretically and numerically that the TUSLA algorithm can solve the optimization problem involving neural networks with ReLU activation function. Besides, we provide simulation results for synthetic examples where popular algorithms, e.g. ADAM, AMSGrad, RMSProp, and (vanilla) SGD, may fail to find the minimizer of the objective functions due to the super-linear growth and the discontinuity of the corresponding stochastic gradient, while the TUSLA algorithm converges rapidly to the optimal solution.

Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks

We present a new class of adaptive stochastic optimization algorithms, which overcomes many of the known shortcomings of popular adaptive optimizers that are currently used for the fine tuning of artificial neural networks (ANNs). Its underpinning theory relies on advances of Eulers polygonal approximations for stochastic differential equations (SDEs) with monotone coefficients. As a result, it inherits the stability properties of tamed algorithms, while it addresses other known issues, e.g. vanishing gradients in ANNs. In particular, we provide an nonasymptotic analysis and full theoretical guarantees for the convergence properties of an algorithm of this novel class, which we named TH$varepsilon$O POULA (or, simply, TheoPouLa). Finally, several experiments are presented with different types of ANNs, which show the superior performance of TheoPouLa over many popular adaptive optimization algorithms.

A fully data-driven approach to minimizing CVaR for portfolio of assets via SGLD with discontinuous updating

A new approach in stochastic optimization via the use of stochastic gradient Langevin dynamics (SGLD) algorithms, which is a variant of stochastic gradient decent (SGD) methods, allows us to efficiently approximate global minimizers of possibly complicated, high-dimensional landscapes. With this in mind, we extend here the non-asymptotic analysis of SGLD to the case of discontinuous stochastic gradients. We are thus able to provide theoretical guarantees for the algorithms convergence in (standard) Wasserstein distances for both convex and non-convex objective functions. We also provide explicit upper estimates of the expected excess risk associated with the approximation of global minimizers of these objective functions. All these findings allow us to devise and present a fully data-driven approach for the optimal allocation of weights for the minimization of CVaR of portfolio of assets with complete theoretical guarantees for its performance. Numerical results illustrate our main findings.

Higher Order Langevin Monte Carlo Algorithm

A new (unadjusted) Langevin Monte Carlo (LMC) algorithm with improved rates in total variation and in Wasserstein distance is presented. All these are obtained in the context of sampling from a target distribution $pi$ that has a density $hat{pi}$ on $mathbb{R}^d$ known up to a normalizing constant. Moreover, $-log hat{pi}$ is assumed to have a locally Lipschitz gradient and its third derivative is locally H{o}lder continuous with exponent $beta in (0,1]$. Non-asymptotic bounds are obtained for the convergence to stationarity of the new sampling method with convergence rate $1+ beta/2$ in Wasserstein distance, while it is shown that the rate is 1 in total variation even in the absence of convexity. Finally, in the case where $-log hat{pi}$ is strongly convex and its gradient is Lipschitz continuous, explicit constants are provided.

On explicit order 1.5 approximations with varying coefficients: the case of super-linear diffusion coefficients

A conjecture appears in cite{milsteinscheme}, in the form of a remark, where it is stated that it is possible to construct, in a specified way, any high order explicit numerical schemes to approximate the solutions of SDEs with superlinear coefficients. We answer this conjecture affirmatively for the case of order 1.5 approximations and show that the suggested methodology works. Moreover, we explore the case of having H{o}lder continuous derivatives for the diffusion coefficients.

General Price Bounds for Guaranteed Annuity Options

In this paper, we are concerned with the valuation of Guaranteed Annuity Options (GAOs) under the most generalised modelling framework where both interest and mortality rates are stochastic and correlated. Pricing these type of options in the correlated environment is a challenging task and no closed form solution exists in the literature. We employ the use of doubly stochastic stopping times to incorporate the randomness about the time of death and employ a suitable change of measure to facilitate the valuation of survival benefit, there by adapting the payoff of the GAO in terms of the payoff of a basket call option. We derive general price bounds for GAOs by utilizing a conditioning approach for the lower bound and arithmetic-geometric mean inequality for the upper bound. The theory is then applied to affine models to present some very interesting formulae for the bounds under the affine set up. Numerical examples are furnished and benchmarked against Monte Carlo simulations to estimate the price of a GAO for a variety of affine processes governing the evolution of mortality and the interest rate.

On Explicit Approximations for Levy Driven SDEs with Super-linear Diffusion Coefficients

Motivated by the results of cite{sabanis2015}, we propose explicit Euler-type schemes for SDEs with random coefficients driven by Levy noise when the drift and diffusion coefficients can grow super-linearly. As an application of our results, one can construct explicit Euler-type schemes for SDEs with delays (SDDEs) which are driven by Levy noise and have super-linear coefficients. Strong convergence results are established and their rate of convergence is shown to be equal to that of the classical Euler scheme. It is proved that the optimal rate of convergence is achieved for $mathcal{L}^2$-convergence which is consistent with the corresponding results available in the literature.

On Milstein approximations with varying coefficients: the case of super-linear diffusion coefficients

A new class of explicit Milstein schemes, which approximate stochastic differential equations (SDEs) with superlinearly growing drift and diffusion coefficients, is proposed in this article. It is shown, under very mild conditions, that these explicit schemes converge in $mathcal L^p$ to the solution of the corresponding SDEs with optimal rate.