Improving Gradient-based Adversarial Training for Text Classification by Contrastive Learning and Auto-Encoder

Recent work has proposed several efficient approaches for generating gradient-based adversarial perturbations on embeddings and proved that the models performance and robustness can be improved when they are trained with these contaminated embeddings. While they paid little attention to how to help the model to learn these adversarial samples more efficiently. In this work, we focus on enhancing the models ability to defend gradient-based adversarial attack during the models training process and propose two novel adversarial training approaches: (1) CARL narrows the original sample and its adversarial sample in the representation space while enlarging their distance from different labeled samples. (2) RAR forces the model to reconstruct the original sample from its adversarial representation. Experiments show that the proposed two approaches outperform strong baselines on various text classification datasets. Analysis experiments find that when using our approaches, the semantic representation of the input sentence wont be significantly affected by adversarial perturbations, and the models performance drops less under adversarial attack. That is to say, our approaches can effectively improve the robustness of the model. Besides, RAR can also be used to generate text-form adversarial samples.

Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks

Loading models pre-trained on the large-scale corpus in the general domain and fine-tuning them on specific downstream tasks is gradually becoming a paradigm in Natural Language Processing. Previous investigations prove that introducing a further pre-training phase between pre-training and fine-tuning phases to adapt the model on the domain-specific unlabeled data can bring positive effects. However, most of these further pre-training works just keep running the conventional pre-training task, e.g., masked language model, which can be regarded as the domain adaptation to bridge the data distribution gap. After observing diverse downstream tasks, we suggest that different tasks may also need a further pre-training phase with appropriate training tasks to bridge the task formulation gap. To investigate this, we carry out a study for improving multiple task-oriented dialogue downstream tasks through designing various tasks at the further pre-training phase. The experiment shows that different downstream tasks prefer different further pre-training tasks, which have intrinsic correlation and most further pre-training tasks significantly improve certain target tasks rather than all. Our investigation indicates that it is of great importance and effectiveness to design appropriate further pre-training tasks modeling specific information that benefit downstream tasks. Besides, we present multiple constructive empirical conclusions for enhancing task-oriented dialogues.

Propensity Score Adapted Covariate Selection for Causal Inference

In this paper, we propose a propensity score adapted variable selection procedure to select covariates for inclusion in propensity score models, in order to eliminate confounding bias and improve statistical efficiency in observational studies. Our variable selection approach is specially designed for causal inference, it only requires the propensity scores to be $sqrt{n}$-consistently estimated through a parametric model and need not correct specification of potential outcome models. By using estimated propensity scores as inverse probability treatment weights in performing an adaptive lasso on the outcome, it successfully excludes instrumental variables, and includes confounders and outcome predictors. We show its oracle properties under the linear association conditions. We also perform some numerical simulations to illustrate our propensity score adapted covariate selection procedure and evaluate its performance under model misspecification. Comparison to other covariate selection methods is made using artificial data as well, through which we find that it is more powerful in excluding instrumental variables and spurious covariates.

Variance Reduction for Causal Inference

Propensity score methods have been shown to be powerful in obtaining efficient estimators of average treatment effect (ATE) from observational data, especially under the existence of confounding factors. When estimating, deciding which type of covariates need to be included in the propensity score function is important, since incorporating some unnecessary covariates may amplify both bias and variance of estimators of ATE. In this paper, we show that including additional instrumental variables that satisfy the exclusion restriction for outcome will do harm to the statistical efficiency. Also, we prove that, controlling for covariates that appear as outcome predictors, i.e. predict the outcomes and are irrelevant to the exposures, can help reduce the asymptotic variance of ATE estimation. We also note that, efficiently estimating the ATE by non-parametric or semi-parametric methods require the estimated propensity score function, as described in Hirano et al. (2003)cite{Hirano2003}. Such estimation procedure usually asks for many regularity conditions, Rothe (2016)cite{Rothe2016} also illustrated this point and proposed a known propensity score (KPS) estimator that requires mild regularity conditions and is still fully efficient. In addition, we introduce a linearly modified (LM) estimator that is nearly efficient in most general settings and need not estimation of the propensity score function, hence convenient to calculate. The construction of this estimator borrows idea from the interaction estimator of Lin (2013)cite{Lin2013}, in which regression adjustment with interaction terms are applied to deal with data arising from a completely randomized experiment. As its name suggests, the LM estimator can be viewed as a linear modification on the IPW estimator using known propensity scores. We will also investigate its statistical properties both analytically and numerically.

The sharp lower bound of the first Dirichlet eigenvalue for geodesic balls

On complete noncompact Riemannian manifolds with non-negative Ricci curvature, Li-Schoen proved the uniform Poincare inequality for any ge odesic ball. In this note, we obtain the sharp lower bound of the first Dirichlet eigenvalue of such geodesic balls, which implies the sharp Poincare inequality for geodesic balls.

SimpModeling: Sketching Implicit Field to Guide Mesh Modeling for 3D Animalmorphic Head Design

Head shapes play an important role in 3D character design. In this work, we propose SimpModeling, a novel sketch-based system for helping users, especially amateur users, easily model 3D animalmorphic heads - a prevalent kind of heads in character design. Although sketching provides an easy way to depict desired shapes, it is challenging to infer dense geometric information from sparse line drawings. Recently, deepnet-based approaches have been taken to address this challenge and try to produce rich geometric details from very few strokes. However, while such methods reduce users workload, they would cause less controllability of target shapes. This is mainly due to the uncertainty of the neural prediction. Our system tackles this issue and provides good controllability from three aspects: 1) we separate coarse shape design and geometric detail specification into two stages and respectively provide different sketching means; 2) in coarse shape designing, sketches are used for both shape inference and geometric constraints to determine global geometry, and in geometric detail crafting, sketches are used for carving surface details; 3) in both stages, we use the advanced implicit-based shape inference methods, which have strong ability to handle the domain gap between freehand sketches and synthetic ones used for training. Experimental results confirm the effectiveness of our method and the usability of our interactive system. We also contribute to a dataset of high-quality 3D animal heads, which are manually created by artists.

Transient Analysis for Resonant Beam Charging and Communication

High communication speed and sufficient energy supply are the directions of technological development. Energy and information available anywhere and anytime has always been peoples good wishes. On this basis, resonant beam system (RBS) has demonstrated its unique superiority in meeting the needs for energy and communication. The previous work has mostly focused on the analysis of charging performance of RBS and its steady-state characteristics. In order to analyze the communication performance of RBS more thoroughly, we propose a resonant beam charging and communication (RBCC) system and use the equivalent circuit analysis method to conduct transient analysis on it. The equivalent circuit reveals the dynamic establishment process of the resonant beam from scratch, which facilitates the analysis of the relaxation oscillation process and a deeper understanding of the energy transmission and communication performance. In addition, we explore the energy transmission and communication performance of the RBCC under different energy allocation strategies.

Similarity-Aware Fusion Network for 3D Semantic Segmentation

In this paper, we propose a similarity-aware fusion network (SAFNet) to adaptively fuse 2D images and 3D point clouds for 3D semantic segmentation. Existing fusion-based methods achieve remarkable performances by integrating information from multiple modalities. However, they heavily rely on the correspondence between 2D pixels and 3D points by projection and can only perform the information fusion in a fixed manner, and thus their performances cannot be easily migrated to a more realistic scenario where the collected data often lack strict pair-wise features for prediction. To address this, we employ a late fusion strategy where we first learn the geometric and contextual similarities between the input and back-projected (from 2D pixels) point clouds and utilize them to guide the fusion of two modalities to further exploit complementary information. Specifically, we employ a geometric similarity module (GSM) to directly compare the spatial coordinate distributions of pair-wise 3D neighborhoods, and a contextual similarity module (CSM) to aggregate and compare spatial contextual information of corresponding central points. The two proposed modules can effectively measure how much image features can help predictions, enabling the network to adaptively adjust the contributions of two modalities to the final prediction of each point. Experimental results on the ScanNetV2 benchmark demonstrate that SAFNet significantly outperforms existing state-of-the-art fusion-based approaches across various data integrity.

Room-temperature continuous-wave Dirac-vortex topological lasers on silicon

Robust laser sources are a fundamental building block for contemporary information technologies. Originating from condensed-matter physics, the concept of topology has recently entered the realm of optics, offering fundamentally new design principles for lasers with enhanced robustness. In analogy to the well-known Majorana fermions in topological superconductors, Dirac-vortex states have recently been investigated in passive photonic systems and are now considered as a promising candidate for single-mode large-area lasers. Here, we experimentally realize the first Dirac-vortex topological lasers in InAs/InGaAs quantum-dot materials monolithically grown on a silicon substrate. We observe room-temperature continuous-wave single-mode linearly polarized vertical laser emission at a telecom wavelength. Most importantly, we confirm that the wavelength of the Dirac-vortex laser is topologically robust against variations in the cavity size, and its free spectral range defies the universal inverse scaling law with the cavity size. These lasers will play an important role in CMOS-compatible photonic and optoelectronic systems on a chip.

Even-Odd Layer-Dependent Anomalous Hall Effect in Topological Magnet MnBi2Te4 Thin Films

A central theme in condensed matter physics is to create and understand the exotic states of matter by incorporating magnetism into topological materials. One prime example is the quantum anomalous Hall (QAH) state. Recently, MnBi2Te4 has been demonstrated to be an intrinsic magnetic topological insulator and the QAH effect was observed in exfoliated MnBi2Te4 flakes. Here, we used molecular beam epitaxy (MBE) to grow MnBi2Te4 films with thickness down to 1 septuple layer (SL) and performed thickness-dependent transport measurements. We observed a non-square hysteresis loop in the antiferromagnetic state for films with thickness greater than 2 SL. The hysteresis loop can be separated into two AH components. Through careful analysis, we demonstrated that one AH component with the larger coercive field is from the dominant MnBi2Te4 phase, while the other AH component with the smaller coercive field is from the minor Mn-doped Bi2Te3 phase in the samples. The extracted AH component of the MnBi2Te4 phase shows a clear even-odd layer-dependent behavior, a signature of antiferromagnetic thin films. Our studies reveal insights on how to optimize the MBE growth conditions to improve the quality of MnBi2Te4 films, in which the QAH and other exotic states are predicted.