Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userAdaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
171
0
0.0
(
0
)
Authors
Taiji Suzuki
Ask ChatGPT about the research
No Arabic abstract
Deep learning has shown high performances in various types of tasks from visual recognition to natural language processing, which indicates superior flexibility and adaptivity of deep learning. To understand this phenomenon theoretically, we develop a new approximation and estimation error analysis of deep learning with the ReLU activation for functions in a Besov space and its variant with mixed smoothness. The Besov space is a considerably general function space including the Holder space and Sobolev space, and especially can capture spatial inhomogeneity of smoothness. Through the analysis in the Besov space, it is shown that deep learning can achieve the minimax optimal rate and outperform any non-adaptive (linear) estimator such as kernel ridge regression, which shows that deep learning has higher adaptivity to the spatial inhomogeneity of the target function than other estimators such as linear ones. In addition to this, it is shown that deep learning can avoid the curse of dimensionality if the target function is in a mixed smooth Besov space. We also show that the dependency of the convergence rate on the dimensionality is tight due to its minimax optimality. These results support high adaptivity of deep learning and its superior ability as a feature extractor.
rate research
Read More
Deep learning has exhibited superior performance for various tasks, especially for high-dimensional datasets, such as images. To understand this property, we investigate the approximation and estimation ability of deep learning on {it anisotropic Besov spaces}. The anisotropic Besov space is characterized by direction-dependent smoothness and includes several function classes that have been investigated thus far. We demonstrate that the approximation error and estimation error of deep learning only depend on the average value of the smoothness parameters in all directions. Consequently, the curse of dimensionality can be avoided if the smoothness of the target function is highly anisotropic. Unlike existing studies, our analysis does not require a low-dimensional structure of the input data. We also investigate the minimax optimality of deep learning and compare its performance with that of the kernel method (more generally, linear estimators). The results show that deep learning has better dependence on the input dimensionality if the target function possesses anisotropic smoothness, and it achieves an adaptive rate for functions with spatially inhomogeneous smoothness.
We continue our investigations on pointwise multipliers for Besov spaces of dominating mixed smoothness. This time we study the algebra property of the classes $S^r_{p,q}B(mathbb{R}^d)$ with respect to pointwise multiplication. In addition if $pleq q$, we are able to describe the space of all pointwise multipliers for $S^r_{p,q}B(mathbb{R}^d)$.
We prove thatthe Banach space $(oplus_{n=1}^infty ell_p^n)_{ell_q}$, which is isomorphic to certain Besov spaces, has a greedy basis whenever $1leq p leqinfty$ and $1<q<infty$. Furthermore, the Banach spaces $(oplus_{n=1}^infty ell_p^n)_{ell_1}$, with $1<ple infty$, and $(oplus_{n=1}^infty ell_p^n)_{c_0}$, with $1le p<infty$ do not have a greedy bases. We prove as well that the space $(oplus_{n=1}^infty ell_p^n)_{ell_q}$ has a 1-greedy basis if and only if $1leq p=qle infty$.
This paper is devoted to giving definitions of Besov spaces on an arbitrary open set of $mathbb R^n$ via the spectral theorem for the Schrodinger operator with the Dirichlet boundary condition. The crucial point is to introduce some test function spaces on $Omega$. The fundamental properties of Besov spaces are also shown, such as embedding relations and duality, etc. Furthermore, the isomorphism relations are established among the Besov spaces in which regularity of functions is measured by the Dirichlet Laplacian and the Schrodinger operators.
We study the embeddings of (homogeneous and inhomogeneous) anisotropic Besov spaces associated to an expansive matrix $A$ into Sobolev spaces, with focus on the influence of $A$ on the embedding behaviour. For a large range of parameters, we derive sharp characterizations of embeddings.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
