Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userRobust Neural Networks inspired by Strong Stability Preserving Runge-Kutta methods
108
0
0.0
(
0
)
Added by
Byungjoo Kim
Publication date
2020
fields
Informatics Engineering
and research's language is
English
Ask ChatGPT about the research
No Arabic abstract
Deep neural networks have achieved state-of-the-art performance in a variety of fields. Recent works observe that a class of widely used neural networks can be viewed as the Euler method of numerical discretization. From the numerical discretization perspective, Strong Stability Preserving (SSP) methods are more advanced techniques than the explicit Euler method that produce both accurate and stable solutions. Motivated by the SSP property and a generalized Runge-Kutta method, we propose Strong Stability Preserving networks (SSP networks) which improve robustness against adversarial attacks. We empirically demonstrate that the proposed networks improve the robustness against adversarial examples without any defensive methods. Further, the SSP networks are complementary with a state-of-the-art adversarial training scheme. Lastly, our experiments show that SSP networks suppress the blow-up of adversarial perturbations. Our results open up a way to study robust architectures of neural networks leveraging rich knowledge from numerical discretization literature.
rate research
Read More
Strong stability preserving (SSP) Runge-Kutta methods are often desired when evolving in time problems that have two components that have very different time scales. Where the SSP property is needed, it has been shown that implicit and implicit-explicit methods have very restrictive time-steps and are therefore not efficient. For this reason, SSP integrating factor methods may offer an attractive alternative to traditional time-stepping methods for problems with a linear component that is stiff and a nonlinear component that is not. However, the strong stability properties of integrating factor Runge-Kutta methods have not been established. In this work we show that it is possible to define explicit integrating factor Runge-Kutta methods that preserve the desired strong stability properties satisfied by each of the two components when coupled with forward Euler time-stepping, or even given weaker conditions. We define sufficient conditions for an explicit integrating factor Runge--Kutta method to be SSP, namely that they are based on explicit SSP Runge--Kutta methods with non-decreasing abscissas. We find such methods of up to fourth order and up to ten stages, analyze their SSP coefficients, and prove their optimality in a few cases. We test these methods to demonstrate their convergence and to show that the SSP time-step predicted by the theory is generally sharp, and that the non-decreasing abscissa condition is needed in our test cases. Finally, we show that on typical total variation diminishing linear and nonlinear test-cases our new explicit SSP integrating factor Runge-Kutta methods out-perform the corresponding explicit SSP Runge-Kutta methods, implicit-explicit SSP Runge--Kutta methods, and some well-known exponential time differencing methods.
Problems that feature significantly different time scales, where the stiff time-step restriction comes from a linear component, implicit-explicit (IMEX) methods alleviate this restriction if the concern is linear stability. However, where the SSP property is needed, IMEX SSP Runge-Kutta (SSP-IMEX) methods have very restrictive time-steps. An alternative to SSP-IMEX schemes is to adopt an integrating factor approach to handle the linear component exactly and step the transformed problem forward using some time-evolution method. The strong stability properties of integrating factor Runge--Kutta methods were previously established, where it was shown that it is possible to define explicit integrating factor Runge-Kutta methods that preserve strong stability properties satisfied by each of the two components when coupled with forward Euler time-stepping. It was proved that the solution will be SSP if the transformed problem is stepped forward with an explicit SSP Runge-Kutta method that has non-decreasing abscissas. However, explicit SSP Runge-Kutta methods have an order barrier of p=4, and sometimes higher order is desired. In this work we consider explicit SSP two-step Runge--Kutta integrating factor methods to raise the order. We show that strong stability is ensured if the two-step Runge-Kutta method used to evolve the transformed problem is SSP and has non-decreasing abscissas. We find such methods up to eighth order and present their SSP coefficients. Adding a step allows us to break the fourth order barrier on explicit SSP Runge-Kutta methods; furthermore, our explicit SSP two-step Runge--Kutta methods with non-decreasing abscissas typically have larger SSP coefficients than the corresponding one-step methods.
Strong stability preserving (SSP) Runge-Kutta methods are desirable when evolving in time problems that have discontinuities or sharp gradients and require nonlinear non-inner-product stability properties to be satisfied. Unlike the case for L2 linear stability, implicit methods do not significantly alleviate the time-step restriction when the SSP property is needed. For this reason, when handling problems with a linear component that is stiff and a nonlinear component that is not, SSP integrating factor Runge--Kutta methods may offer an attractive alternative to traditional time-stepping methods. The strong stability properties of integrating factor Runge--Kutta methods where the transformed problem is evolved with an explicit SSP Runge--Kutta method with non-decreasing abscissas was recently established. In this work, we consider the use of downwinded spatial operators to preserve the strong stability properties of integrating factor Runge--Kutta methods where the Runge--Kutta method has some decreasing abscissas. We present the SSP theory for this approach and present numerical evidence to show that such an approach is feasible and performs as expected. However, we also show that in some cases the integrating factor approach with explicit SSP Runge--Kutta methods with non-decreasing abscissas performs nearly as well, if not better, than with explicit SSP Runge--Kutta methods with downwinding. In conclusion, while the downwinding approach can be rigorously shown to guarantee the SSP property for a larger time-step, in practice using the integrating factor approach by including downwinding as needed with optimal explicit SSP Runge--Kutta methods does not necessarily provide significant benefit over using explicit SSP Runge--Kutta methods with non-decreasing abscissas.
When evolving in time the solution of a hyperbolic partial differential equation, it is often desirable to use high order strong stability preserving (SSP) time discretizations. These time discretizations preserve the monotonicity properties satisfied by the spatial discretization when coupled with the first order forward Euler, under a certain time-step restriction. While the allowable time-step depends on both the spatial and temporal discretizations, the contribution of the temporal discretization can be isolated by taking the ratio of the allowable time-step of the high order method to the forward Euler time-step. This ratio is called the strong stability coefficient. The search for high order strong stability time-stepping methods with high order and large allowable time-step had been an active area of research. It is known that implicit SSP Runge-Kutta methods exist only up to sixth order. However, if we restrict ourselves to solving only linear autonomous problems, the order conditions simplify and we can find implicit SSP Runge-Kutta methods of any linear order. In the current work we aim to find very high linear order implicit SSP Runge-Kutta methods that are optimal in terms of allowable time-step. Next, we formulate an optimization problem for implicit-explicit (IMEX) SSP Runge-Kutta methods and find implicit methods with large linear stability regions that pair with known explicit SSP Runge-Kutta methods of orders plin=3,4,6 as well as optimized IMEX SSP Runge-Kutta pairs that have high linear order and nonlinear orders p=2,3,4. These methods are then tested on sample problems to verify order of convergence and to demonstrate the sharpness of the SSP coefficient and the typical behavior of these methods on test problems.
In this work we present a class of high order unconditionally strong stability preserving (SSP) implicit multi-derivative Runge--Kutta schemes, and SSP implicit-explicit (IMEX) multi-derivative Runge--Kutta schemes where the time-step restriction is independent of the stiff term. The unconditional SSP property for a method of order $p>2$ is unique among SSP methods, and depends on a backward-in-time assumption on the derivative of the operator. We show that this backward derivative condition is satisfied in many relevant cases where SSP IMEX schemes are desired. We devise unconditionally SSP implicit Runge--Kutta schemes of order up to $p=4$, and IMEX Runge--Kutta schemes of order up to $p=3$. For the multi-derivative IMEX schemes, we also derive and present the order conditions, which have not appeared previously. The unconditional SSP condition ensures that these methods are positivity preserving, and we present sufficient conditions under which such methods are also asymptotic preserving when applied to a range of problems, including a hyperbolic relaxation system, the Broadwell model, and the Bhatnagar-Gross-Krook (BGK) kinetic equation. We present numerical results to support the theoretical results, on a variety of problems.
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
