While we have made significant progress on understanding hand-object interactions in computer vision, it is still very challenging for robots to perform complex dexterous manipulation. In this paper, we propose a new platform and pipeline, DexMV (Dex
terous Manipulation from Videos), for imitation learning to bridge the gap between computer vision and robot learning. We design a platform with: (i) a simulation system for complex dexterous manipulation tasks with a multi-finger robot hand and (ii) a computer vision system to record large-scale demonstrations of a human hand conducting the same tasks. In our new pipeline, we extract 3D hand and object poses from the videos, and convert them to robot demonstrations via motion retargeting. We then apply and compare multiple imitation learning algorithms with the demonstrations. We show that the demonstrations can indeed improve robot learning by a large margin and solve the complex tasks which reinforcement learning alone cannot solve. Project page with video: https://yzqin.github.io/dexmv
Periodically driven systems, or Floquet systems, exhibit many novel dynamics and interesting out-of-equilibrium phases of matter. Those phases arising with the quantum systems symmetries, such as global $U(1)$ symmetry, can even show dynamical stabil
ity with symmetry-protection. Here we experimentally demonstrate a $U(1)$ symmetry-protected prethermal phase, via performing a digital-analog quantum simulation on a superconducting quantum processor. The dynamical stability of this phase is revealed by its robustness against external perturbations. We also find that the spin glass order parameter in this phase is stabilized by the interaction between the spins. Our work reveals a promising prospect in discovering emergent quantum dynamical phases with digital-analog quantum simulators.
The Wigner crystal state, first predicted by Eugene Wigner in 1934, has fascinated condensed matter physicists for nearly 90 years2-14. Studies of two-dimensional (2D) electron gases first revealed signatures of the Wigner crystal in electrical trans
port measurements at high magnetic fields2-4. More recently optical spectroscopy has provided evidence of generalized Wigner crystal states in transition metal dichalcogenide (TMDC) moire superlattices. Direct observation of the 2D Wigner crystal lattice in real space, however, has remained an outstanding challenge. Scanning tunneling microscopy (STM) in principle has sufficient spatial resolution to image a Wigner crystal, but conventional STM measurements can potentially alter fragile Wigner crystal states in the process of measurement. Here we demonstrate real-space imaging of 2D Wigner crystals in WSe2/WS2 moire heterostructures using a novel non-invasive STM spectroscopy technique. We employ a graphene sensing layer in close proximity to the WSe2/WS2 moire superlattice for Wigner crystal imaging, where local STM tunneling current into the graphene sensing layer is modulated by the underlying electron lattice of the Wigner crystal in the WSe2/WS2 heterostructure. Our measurement directly visualizes different lattice configurations associated with Wigner crystal states at fractional electron fillings of n = 1/3, 1/2, and 2/3, where n is the electron number per site. The n=1/3 and n=2/3 Wigner crystals are observed to exhibit a triangle and a honeycomb lattice, respectively, in order to minimize nearest-neighbor occupations. The n = 1/2 state, on the other hand, spontaneously breaks the original C3 symmetry and forms a stripe structure in real space. Our study lays a solid foundation toward the fundamental understanding of rich Wigner crystal states in WSe2/WS2 moire heterostructures.
Estimating 3D hand and object pose from a single image is an extremely challenging problem: hands and objects are often self-occluded during interactions, and the 3D annotations are scarce as even humans cannot directly label the ground-truths from a
single image perfectly. To tackle these challenges, we propose a unified framework for estimating the 3D hand and object poses with semi-supervised learning. We build a joint learning framework where we perform explicit contextual reasoning between hand and object representations by a Transformer. Going beyond limited 3D annotations in a single image, we leverage the spatial-temporal consistency in large-scale hand-object videos as a constraint for generating pseudo labels in semi-supervised learning. Our method not only improves hand pose estimation in challenging real-world dataset, but also substantially improve the object pose which has fewer ground-truths per instance. By training with large-scale diverse videos, our model also generalizes better across multiple out-of-domain datasets. Project page and code: https://stevenlsw.github.io/Semi-Hand-Object
Transition metal dichalcogenide (TMD) moire heterostructures provide an ideal platform to explore the extended Hubbard model1 where long-range Coulomb interactions play a critical role in determining strongly correlated electron states. This has led
to experimental observations of Mott insulator states at half filling2-4 as well as a variety of extended Wigner crystal states at different fractional fillings5-9. Microscopic understanding of these emerging quantum phases, however, is still lacking. Here we describe a novel scanning tunneling microscopy (STM) technique for local sensing and manipulation of correlated electrons in a gated WS2/WSe2 moire superlattice that enables experimental extraction of fundamental extended Hubbard model parameters. We demonstrate that the charge state of local moire sites can be imaged by their influence on STM tunneling current, analogous to the charge-sensing mechanism in a single-electron transistor. In addition to imaging, we are also able to manipulate the charge state of correlated electrons. Discharge cascades of correlated electrons in the moire superlattice are locally induced by ramping the STM bias, thus enabling the nearest-neighbor Coulomb interaction (UNN) to be estimated. 2D mapping of the moire electron charge states also enables us to determine onsite energy fluctuations at different moire sites. Our technique should be broadly applicable to many semiconductor moire systems, offering a powerful new tool for microscopic characterization and control of strongly correlated states in moire superlattices.
Moire superlattices in transition metal dichalcogenide (TMD) heterostructures can host novel correlated quantum phenomena due to the interplay of narrow moire flat bands and strong, long-range Coulomb interactions1-5. However, microscopic knowledge o
f the atomically-reconstructed moire superlattice and resulting flat bands is still lacking, which is critical for fundamental understanding and control of the correlated moire phenomena. Here we quantitatively study the moire flat bands in three-dimensional (3D) reconstructed WSe2/WS2 moire superlattices by comparing scanning tunneling spectroscopy (STS) of high quality exfoliated TMD heterostructure devices with ab initio simulations of TMD moire superlattices. A strong 3D buckling reconstruction accompanied by large in-plane strain redistribution is identified in our WSe2/WS2 moire heterostructures. STS imaging demonstrates that this results in a remarkably narrow and highly localized K-point moire flat band at the valence band edge of the heterostructure. A series of moire flat bands are observed at different energies that exhibit varying degrees of localization. Our observations contradict previous simplified theoretical models but agree quantitatively with ab initio simulations that fully capture the 3D structural reconstruction. Here the strain redistribution and 3D buckling dominate the effective moire potential and result in moire flat bands at the Brillouin zone K points.
The quest for biologically plausible deep learning is driven, not just by the desire to explain experimentally-observed properties of biological neural networks, but also by the hope of discovering more efficient methods for training artificial netwo
rks. In this paper, we propose a new algorithm named Variational Probably Flow (VPF), an extension of minimum probability flow for training binary Deep Boltzmann Machines (DBMs). We show that weight updates in VPF are local, depending only on the states and firing rates of the adjacent neurons. Unlike contrastive divergence, there is no need for Gibbs confabulations; and unlike backpropagation, alternating feedforward and feedback phases are not required. Moreover, the learning algorithm is effective for training DBMs with intra-layer connections between the hidden nodes. Experiments with MNIST and Fashion MNIST demonstrate that VPF learns reasonable features quickly, reconstructs corrupted images more accurately, and generates samples with a high estimated log-likelihood. Lastly, we note that, interestingly, if an asymmetric version of VPF exists, the weight updates directly explain experimental results in Spike-Timing-Dependent Plasticity (STDP).
Gaussian latent tree models, or more generally, Gaussian latent forest models have Fisher-information matrices that become singular along interesting submodels, namely, models that correspond to subforests. For these singularities, we compute the rea
l log-canonical thresholds (also known as stochastic complexities or learning coefficients) that quantify the large-sample behavior of the marginal likelihood in Bayesian inference. This provides the information needed for a recently introduced generalization of the Bayesian information criterion. Our mathematical developments treat the general setting of Laplace integrals whose phase functions are sums of squared differences between monomials and constants. We clarify how in this case real log-canonical thresholds can be computed using polyhedral geometry, and we show how to apply the general theory to the Laplace integrals associated with Gaussian latent tree and forest models. In simulations and a data example, we demonstrate how the mathematical knowledge can be applied in model selection.
The image of the principal minor map for n x n-matrices is shown to be closed. In the 19th century, Nansen and Muir studied the implicitization problem of finding all relations among principal minors when n=4. We complete their partial results by con
structing explicit polynomials of degree 12 that scheme-theoretically define this affine variety and also its projective closure in $PP^{15}$. The latter is the main component in the singular locus of the 2 x 2 x 2 x 2-hyperdeterminant.
