An Optimal Resource Allocator of Elastic Training for Deep Learning Jobs on Cloud

Cloud training platforms, such as Amazon Web Services and Huawei Cloud provide users with computational resources to train their deep learning jobs. Elastic training is a service embedded in cloud training platforms that dynamically scales up or down the resources allocated to a job. The core technique of an elastic training system is to best allocate limited resources among heterogeneous jobs in terms of shorter queueing delay and higher training efficiency. This paper presents an optimal resource allocator for elastic training system that leverages a mixed-integer programming (MIP) model to maximize the training progress of deep learning jobs. We take advantage of the real-world job data obtained from ModelArts, the deep learning training platform of Huawei Cloud and conduct simulation experiments to compare the optimal resource allocator with a greedy one as benchmark. Numerical results show that the proposed allocator can reduce queuing time by up to 32% and accelerate training efficiency by up to 24% relative to the greedy resource allocator, thereby greatly improving user experience with Huawei ModelArts and potentially enabling the realization of higher profits for the product. Also, the optimal resource allocator is fast in decision-making, taking merely 0.4 seconds on average.

Photoelectron Transportation Dynamics in GaAs Photocathodes

We report here a general theory describing photoelectron transportation dynamics in GaAs semiconductor photocathodes. Gradient doping is incorporated in the model through the inclusion of directional carrier drift. The time-evolution of electron concentration in the active layer upon the injection of an excitation pulse is solved both numerically and analytically. The predictions of the model are compared with experiments via carrier-induced transient reflectivity change, which is measured for gradient-doped and uniform-doped photocathodes using femtosecond pump-probe reflectometry. Excellent agreement is found between the experiments and the theory, leading to the characterization of key device parameters such as diffusion constant and electron decay rates. Comparisons are also made between uniform doping and gradient doping for their characteristics in photoelectron transportation. Doping gradient is found to be able to accelerate electron accumulation on the device surface. These results offer new insights into the dynamics of III-V photocathodes and potentially open a new avenue toward experimental characterization of device parameters.

Edwards-Anderson parameter and local Ising-nematicity in FeSe revealed via NMR spectral broadening

The NMR spectrum of FeSe shows a dramatic broadening on cooling towards the bulk nematic phase at $T_s=90$ K, due to the formation of a quasi-static, short-range-ordered nematic domain structure. However, a quantitative understanding of the NMR broadening and its relationship to the nematic susceptibility is still lacking. Here, we show that the temperature and pressure dependence of the broadening is in quantitative agreement with the mean-field Edwards-Anderson parameter of an Ising-nematic model in the presence of random-field disorder introduced by non-magnetic impurities. Furthermore, these results reconcile the interpretation of NMR and Raman spectroscopy data in FeSe under pressure.

On distributions of velocity random fields in turbulent flows

The purpose of the present paper is to derive a partial differential equation (PDE) for the single-time single-point probability density function (PDF) of the velocity field of a turbulent flow. The PDF PDE is a highly non-linear parabolic-transport equation, which depends on two conditional statistical numerics of important physical significance. The PDF PDE is a general form of the classical Reynolds mean flow equation, and is a precise formulation of the PDF transport equation. The PDF PDE provides us with a new method for modelling turbulence. An explicit example is constructed, though the example is seemingly artificial, but it demonstrates the PDF method based on the new PDF PDE.

Decentralized Composite Optimization in Stochastic Networks: A Dual Averaging Approach with Linear Convergence

Decentralized optimization, particularly the class of decentralized composite convex optimization (DCCO) problems, has found many applications. Due to ubiquitous communication congestion and random dropouts in practice, it is highly desirable to design decentralized algorithms that can handle stochastic communication networks. However, most existing algorithms for DCCO only work in time-invariant networks and cannot be extended to stochastic networks because they inherently need knowledge of network topology $textit{a priori}$. In this paper, we propose a new decentralized dual averaging (DDA) algorithm that can solve DCCO in stochastic networks. Under a rather mild condition on stochastic networks, we show that the proposed algorithm attains $textit{global linear convergence}$ if each local objective function is strongly convex. Our algorithm substantially improves the existing DDA-type algorithms as the latter were only known to converge $textit{sublinearly}$ prior to our work. The key to achieving the improved rate is the design of a novel dynamic averaging consensus protocol for DDA, which intuitively leads to more accurate local estimates of the global dual variable. To the best of our knowledge, this is the first linearly convergent DDA-type decentralized algorithm and also the first algorithm that attains global linear convergence for solving DCCO in stochastic networks. Numerical results are also presented to support our design and analysis.

Optimal Non-Convex Exact Recovery in Stochastic Block Model via Projected Power Method

In this paper, we study the problem of exact community recovery in the symmetric stochastic block model, where a graph of $n$ vertices is randomly generated by partitioning the vertices into $K ge 2$ equal-sized communities and then connecting each pair of vertices with probability that depends on their community memberships. Although the maximum-likelihood formulation of this problem is discrete and non-convex, we propose to tackle it directly using projected power iterations with an initialization that satisfies a partial recovery condition. Such an initialization can be obtained by a host of existing methods. We show that in the logarithmic degree regime of the considered problem, the proposed method can exactly recover the underlying communities at the information-theoretic limit. Moreover, with a qualified initialization, it runs in $mathcal{O}(nlog^2n/loglog n)$ time, which is competitive with existing state-of-the-art methods. We also present numerical results of the proposed method to support and complement our theoretical development.

One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction

Traditional industrial recommenders are usually trained on a single business domain and then serve for this domain. However, in large commercial platforms, it is often the case that the recommenders need to make click-through rate (CTR) predictions for multiple business domains. Different domains have overlapping user groups and items. Thus, there exist commonalities. Since the specific user groups have disparity and the user behaviors may change in various business domains, there also have distinctions. The distinctions result in domain-specific data distributions, making it hard for a single shared model to work well on all domains. To learn an effective and efficient CTR model to handle multiple domains simultaneously, we present Star Topology Adaptive Recommender (STAR). Concretely, STAR has the star topology, which consists of the shared centered parameters and domain-specific parameters. The shared parameters are applied to learn commonalities of all domains, and the domain-specific parameters capture domain distinction for more refined prediction. Given requests from different business domains, STAR can adapt its parameters conditioned on the domain characteristics. The experimental result from production data validates the superiority of the proposed STAR model. Since 2020, STAR has been deployed in the display advertising system of Alibaba, obtaining averaging 8.0% improvement on CTR and 6.0% on RPM (Revenue Per Mille).

CAN: Revisiting Feature Co-Action for Click-Through Rate Prediction

Inspired by the success of deep learning, recent industrial Click-Through Rate (CTR) prediction models have made the transition from traditional shallow approaches to deep approaches. Deep Neural Networks (DNNs) are known for its ability to learn non-linear interactions from raw feature automatically, however, the non-linear feature interaction is learned in an implicit manner. The non-linear interaction may be hard to capture and explicitly model the textit{co-action} of raw feature is beneficial for CTR prediction. textit{Co-action} refers to the collective effects of features toward final prediction. In this paper, we argue that current CTR models do not fully explore the potential of feature co-action. We conduct experiments and show that the effect of feature co-action is underestimated seriously. Motivated by our observation, we propose feature Co-Action Network (CAN) to explore the potential of feature co-action. The proposed model can efficiently and effectively capture the feature co-action, which improves the model performance while reduce the storage and computation consumption. Experiment results on public and industrial datasets show that CAN outperforms state-of-the-art CTR models by a large margin. Up to now, CAN has been deployed in the Alibaba display advertisement system, obtaining averaging 12% improvement on CTR and 8% on RPM.

OpenAttack: An Open-source Textual Adversarial Attack Toolkit

Textual adversarial attacking has received wide and increasing attention in recent years. Various attack models have been proposed, which are enormously distinct and implemented with different programming frameworks and settings. These facts hinder quick utilization and apt comparison of attack models. In this paper, we present an open-source textual adversarial attack toolkit named OpenAttack. It currently builds in 12 typical attack models that cover all the attack types. Its highly inclusive modular design not only supports quick utilization of existing attack models, but also enables great flexibility and extensibility. OpenAttack has broad uses including comparing and evaluating attack models, measuring robustness of a victim model, assisting in developing new attack models, and adversarial training. Source code, built-in models and documentation can be obtained at https://github.com/thunlp/OpenAttack.

Solving High-Order Portfolios via Successive Convex Approximation Algorithms

The first moment and second central moments of the portfolio return, a.k.a. mean and variance, have been widely employed to assess the expected profit and risk of the portfolio. Investors pursue higher mean and lower variance when designing the portfolios. The two moments can well describe the distribution of the portfolio return when it follows the Gaussian distribution. However, the real world distribution of assets return is usually asymmetric and heavy-tailed, which is far from being a Gaussian distribution. The asymmetry and the heavy-tailedness are characterized by the third and fourth central moments, i.e., skewness and kurtosis, respectively. Higher skewness and lower kurtosis are preferred to reduce the probability of extreme losses. However, incorporating high-order moments in the portfolio design is very difficult due to their non-convexity and rapidly increasing computational cost with the dimension. In this paper, we propose a very efficient and convergence-provable algorithm framework based on the successive convex approximation (SCA) algorithm to solve high-order portfolios. The efficiency of the proposed algorithm framework is demonstrated by the numerical experiments.