Dynamic Modeling of Hand-Object Interactions via Tactile Sensing

Tactile sensing is critical for humans to perform everyday tasks. While significant progress has been made in analyzing object grasping from vision, it remains unclear how we can utilize tactile sensing to reason about and model the dynamics of hand-object interactions. In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model, which can then be used on its own during the test time. The tactile model aims to predict the 3d locations of both the hand and the object purely from the touch data by combining a predictive model and a contrastive learning module. This framework can reason about the interaction patterns from the tactile data, hallucinate the changes in the environment, estimate the uncertainty of the prediction, and generalize to unseen objects. We also provide detailed ablation studies regarding different system designs as well as visualizations of the predicted trajectories. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing, which opens the door for future applications in activity learning, human-computer interactions, and imitation learning for robotics.

Neural Network Branch-and-Bound for Neural Network Verification

Many available formal verification methods have been shown to be instances of a unified Branch-and-Bound (BaB) formulation. We propose a novel machine learning framework that can be used for designing an effective branching strategy as well as for computing better lower bounds. Specifically, we learn two graph neural networks (GNN) that both directly treat the network we want to verify as a graph input and perform forward-backward passes through the GNN layers. We use one GNN to simulate the strong branching heuristic behaviour and another to compute a feasible dual solution of the convex relaxation, thereby providing a valid lower bound. We provide a new verification dataset that is more challenging than those used in the literature, thereby providing an effective alternative for testing algorithmic improvements for verification. Whilst using just one of the GNNs leads to a reduction in verification time, we get optimal performance when combining the two GNN approaches. Our combined framework achieves a 50% reduction in both the number of branches and the time required for verification on various convolutional networks when compared to several state-of-the-art verification methods. In addition, we show that our GNN models generalize well to harder properties on larger unseen networks.

Efficient Spatialtemporal Context Modeling for Action Recognition

Contextual information plays an important role in action recognition. Local operations have difficulty to model the relation between two elements with a long-distance interval. However, directly modeling the contextual information between any two points brings huge cost in computation and memory, especially for action recognition, where there is an additional temporal dimension. Inspired from 2D criss-cross attention used in segmentation task, we propose a recurrent 3D criss-cross attention (RCCA-3D) module to model the dense long-range spatiotemporal contextual information in video for action recognition. The global context is factorized into sparse relation maps. We model the relationship between points in the same line along the direction of horizon, vertical and depth at each time, which forms a 3D criss-cross structure, and duplicate the same operation with recurrent mechanism to transmit the relation between points in a line to a plane finally to the whole spatiotemporal space. Compared with the non-local method, the proposed RCCA-3D module reduces the number of parameters and FLOPs by 25% and 30% for video context modeling. We evaluate the performance of RCCA-3D with two latest action recognition networks on three datasets and make a thorough analysis of the architecture, obtaining the optimal way to factorize and fuse the relation maps. Comparisons with other state-of-the-art methods demonstrate the effectiveness and efficiency of our model.

Kinetic ratchet effect as a non-equilibrium design principle for selective channels

In living cells, ion channels passively allow for ions to flow through as the concentration gradient relaxes to thermal equilibrium. Most ion channels are selective, only allowing one type of ion to go through while blocking another. One salient example is KcsA, which allows for larger $text{K}^+$ ions through but blocks the smaller $text{Na}^+$ ions. This counter-intuitive selectivity has been explained by two distinct theories that both focus on equilibrium properties: particle-channel affinity and particle-solvent affinity. However, ion channels operate far from equilibrium. By constructing minimal kinetic models of channels, we discover a ubiquitous kinetic ratchet effect as a non-equilibrium mechanism to explain such selectivity. We find that a multi-site channel kinetically couples the competing flows of two types of particles, where one particles flow could suppress or even invert the flow of another type. At the inversion point (transition between the ratchet and dud modes), the channel achieves infinite selectivity. We have applied our theory to obtain general design principles of artificial selective channels.

System measurement of Intel AEP Optane DIMM

In recent years, memory wall has been a great performance bottleneck of computer system. To overcome it, Non-Volatile Main Memory (NVMM) technology has been discussed widely to provide a much larger main memory capacity. Last year, Intel released AEP Optane DIMM, which provides hundreds of GB capacity as a promising replacement of traditional DRAM memory. But as most key parameters of AEP is not open to users, there is a need to get to know them because they will guide a direction of further NVMM research. In this paper, we focus on measuring performance and architecture features of AEP DIMM. Together, we explore the design of DRAM cache which is an important part of DRAM-AEP hybrid memory system. As a result, we estimate the write latency of AEP DIMM which has not been measured accurately. And, we discover the current design parameters of DRAM cache, such as tag organization, cache associativity and set index mapping. All of these features are first published on academic paper which are greatly helpful to future NVMM optimizations.

System Outage Probability of PS-SWIPT Enabled Two-Way AF Relaying with Hardware Impairments

In this paper, we investigate the system outage probability of a simultaneous wireless information and power transfer (SWIPT) based two-way amplify-and-forward (AF) relay network considering transceiver hardware impairments (HIs), where the energy-constrained relay node processes the received signals based on a power splitting protocol and the two terminals employ a selection combining (SC) scheme to exploit the signals from the direct and relaying links. Assuming independent but non-identically distributed Nakagami-m fading channels, we derive the system outage probability in a closed-form, which enables us to identify two crucial ceiling effects on the system outage probability caused by HIs in the high data rate regions, i.e., relay cooperation ceiling (RCC) and overall system ceiling (OSC). Specifically, the RCC prevents the relaying link from participating in cooperative communications, while the OSC leaves the overall system in outage. Furthermore, we derive the achievable diversity gain of the considered network, which shows that the diversity gain equals either the shape parameter of the direct link or zero. Computer simulations are provided to validate the correctness of our analytical results, and study the effects of various system parameters on the system outage performance and the optimal power splitting ratio, as well as the energy efficiency.

Multibeam Optimization for Joint Communication and Radio Sensing Using Analog Antenna Arrays

Multibeam technology enables the use of two or more subbeams for joint communication and radio sensing, to meet different requirements of beamwidth and pointing directions. Generating and optimizing multibeam subject to the requirements is critical and challenging, particularly for systems using analog arrays. This paper develops optimal solutions to a range of multibeam design problems, where both communication and sensing are considered. We first study the optimal combination of two pre-generated subbeams, and their beamforming vectors, using a combining phase coefficient. Closed-form optimal solutions are derived to the constrained optimization problems, where the received signal powers for communication and the beamforming waveforms are alternatively used as the objective and constraint functions. We also develop global optimization methods which directly find optimal solutions for a single beamforming vector. By converting the original intractable complex NP-hard global optimization problems to real quadratically constrained quadratic programs, near-optimal solutions are obtained using semidefinite relaxation techniques. Extensive simulations validate the effectiveness of the proposed constrained multibeam generation and optimization methods.

Reveal of Domain Effect: How Visual Restoration Contributes to Object Detection in Aquatic Scenes

Underwater robotic perception usually requires visual restoration and object detection, both of which have been studied for many years. Meanwhile, data domain has a huge impact on modern data-driven leaning process. However, exactly indicating domain effect, the relation between restoration and detection remains unclear. In this paper, we generally investigate the relation of quality-diverse data domain to detection performance. In the meantime, we unveil how visual restoration contributes to object detection in real-world underwater scenes. According to our analysis, five key discoveries are reported: 1) Domain quality has an ignorable effect on within-domain convolutional representation and detection accuracy; 2) low-quality domain leads to higher generalization ability in cross-domain detection; 3) low-quality domain can hardly be well learned in a domain-mixed learning process; 4) degrading recall efficiency, restoration cannot improve within-domain detection accuracy; 5) visual restoration is beneficial to detection in the wild by reducing the domain shift between training data and real-world scenes. Finally, as an illustrative example, we successfully perform underwater object detection with an aquatic robot.

Neural Network Branching for Neural Network Verification

Formal verification of neural networks is essential for their deployment in safety-critical areas. Many available formal verification methods have been shown to be instances of a unified Branch and Bound (BaB) formulation. We propose a novel framework for designing an effective branching strategy for BaB. Specifically, we learn a graph neural network (GNN) to imitate the strong branching heuristic behaviour. Our framework differs from previous methods for learning to branch in two main aspects. Firstly, our framework directly treats the neural network we want to verify as a graph input for the GNN. Secondly, we develop an intuitive forward and backward embedding update schedule. Empirically, our framework achieves roughly $50%$ reduction in both the number of branches and the time required for verification on various convolutional networks when compared to the best available hand-designed branching strategy. In addition, we show that our GNN model enjoys both horizontal and vertical transferability. Horizontally, the model trained on easy properties performs well on properties of increased difficulty levels. Vertically, the model trained on small neural networks achieves similar performance on large neural networks.

Branch and Bound for Piecewise Linear Neural Network Verification

The success of Deep Learning and its potential use in many safety-critical applications has motivated research on formal verification of Neural Network (NN) models. In this context, verification involves proving or disproving that an NN model satisfies certain input-output properties. Despite the reputation of learned NN models as black boxes, and the theoretical hardness of proving useful properties about them, researchers have been successful in verifying some classes of models by exploiting their piecewise linear structure and taking insights from formal methods such as Satisifiability Modulo Theory. However, these methods are still far from scaling to realistic neural networks. To facilitate progress on this crucial area, we exploit the Mixed Integer Linear Programming (MIP) formulation of verification to propose a family of algorithms based on Branch-and-Bound (BaB). We show that our family contains previous verification methods as special cases. With the help of the BaB framework, we make three key contributions. Firstly, we identify new methods that combine the strengths of multiple existing approaches, accomplishing significant performance improvements over previous state of the art. Secondly, we introduce an effective branching strategy on ReLU non-linearities. This branching strategy allows us to efficiently and successfully deal with high input dimensional problems with convolutional network architecture, on which previous methods fail frequently. Finally, we propose comprehensive test data sets and benchmarks which includes a collection of previously released testcases. We use the data sets to conduct a thorough experimental comparison of existing and new algorithms and to provide an inclusive analysis of the factors impacting the hardness of verification problems.