Most existing video tasks related to human focus on the segmentation of salient humans, ignoring the unspecified others in the video. Few studies have focused on segmenting and tracking all humans in a complex video, including pedestrians and humans
of other states (e.g., seated, riding, or occluded). In this paper, we propose a novel framework, abbreviated as HVISNet, that segments and tracks all presented people in given videos based on a one-stage detector. To better evaluate complex scenes, we offer a new benchmark called HVIS (Human Video Instance Segmentation), which comprises 1447 human instance masks in 805 high-resolution videos in diverse scenes. Extensive experiments show that our proposed HVISNet outperforms the state-of-the-art methods in terms of accuracy at a real-time inference speed (30 FPS), especially on complex video scenes. We also notice that using the center of the bounding box to distinguish different individuals severely deteriorates the segmentation accuracy, especially in heavily occluded conditions. This common phenomenon is referred to as the ambiguous positive samples problem. To alleviate this problem, we propose a mechanism named Inner Center Sampling to improve the accuracy of instance segmentation. Such a plug-and-play inner center sampling mechanism can be incorporated in any instance segmentation models based on a one-stage detector to improve the performance. In particular, it gains 4.1 mAP improvement on the state-of-the-art method in the case of occluded humans. Code and data are available at https://github.com/IIGROUP/HVISNet.
Current methods of multi-person pose estimation typically treat the localization and the association of body joints separately. It is convenient but inefficient, leading to additional computation and a waste of time. This paper, however, presents a n
ovel framework PoseDet (Estimating Pose by Detection) to localize and associate body joints simultaneously at higher inference speed. Moreover, we propose the keypoint-aware pose embedding to represent an object in terms of the locations of its keypoints. The proposed pose embedding contains semantic and geometric information, allowing us to access discriminative and informative features efficiently. It is utilized for candidate classification and body joint localization in PoseDet, leading to robust predictions of various poses. This simple framework achieves an unprecedented speed and a competitive accuracy on the COCO benchmark compared with state-of-the-art methods. Extensive experiments on the CrowdPose benchmark show the robustness in the crowd scenes. Source code is available.
In informal learning scenarios the popularity of multimedia content, such as video tutorials or lectures, has significantly increased. Yet, the users interactions, navigation behavior, and consequently learning outcome, have not been researched exten
sively. Related work in this field, also called search as learning, has focused on behavioral or text resource features to predict learning outcome and knowledge gain. In this paper, we investigate whether we can exploit features representing multimedia resource consumption to predict of knowledge gain (KG) during Web search from in-session data, that is without prior knowledge about the learner. For this purpose, we suggest a set of multimedia features related to image and video consumption. Our feature extraction is evaluated in a lab study with 113 participants where we collected data for a given search as learning task on the formation of thunderstorms and lightning. We automatically analyze the monitored log data and utilize state-of-the-art computer vision methods to extract features about the seen multimedia resources. Experimental results demonstrate that multimedia features can improve KG prediction. Finally, we provide an analysis on feature importance (text and multimedia) for KG prediction.
Open source development, to a great extent, is a type of social movement in which shared ideologies play critical roles. For participants of open source development, ideology determines how they make sense of things, shapes their thoughts, actions, a
nd interactions, enables rich social dynamics in their projects and communities, and hereby realizes profound impacts at both individual and organizational levels. While software engineering researchers have been increasingly recognizing ideologys importance in open source development, the notion of ideology has shown significant ambiguity and vagueness, and resulted in theoretical and empirical confusion. In this article, we first examine the historical development of ideologys conceptualization, and its theories in multiple disciplines. Then, we review the extant software engineering literature related to ideology. We further argue the imperatives of developing an empirical theory of ideology in open source development, and propose a research agenda for developing such a theory. How such a theory could be applied is also discussed.
Galaxy angular momentum directions (spins) are observable, well described by the Lagrangian tidal torque theory, and proposed to probe the primordial universe. They trace the spins of dark matter halos, and are indicators of protohalos properties in
Lagrangian space. We define a Lagrangian spin parameter and tidal twist parameters and quantify their influence on the spin conservation and predictability in the spin mode reconstruction in $N$-body simulations. We conclude that protohalos in a more tidal twisting environments are preferentially more rotation-supported, and more likely to conserve their spin direction through the cosmic evolution. These tidal environments and spin magnitudes are predictable by a density reconstruction in Lagrangian space, and such predictions can improve the correlation between galaxy spins and the initial conditions in the study of constraining the primordial universe by spin mode reconstruction.
Feature pyramid has been an efficient method to extract features at different scales. Development over this method mainly focuses on aggregating contextual information at different levels while seldom touching the inter-level correlation in the featu
re pyramid. Early computer vision methods extracted scale-invariant features by locating the feature extrema in both spatial and scale dimension. Inspired by this, a convolution across the pyramid level is proposed in this study, which is termed pyramid convolution and is a modified 3-D convolution. Stacked pyramid convolutions directly extract 3-D (scale and spatial) features and outperforms other meticulously designed feature fusion modules. Based on the viewpoint of 3-D convolution, an integrated batch normalization that collects statistics from the whole feature pyramid is naturally inserted after the pyramid convolution. Furthermore, we also show that the naive pyramid convolution, together with the design of RetinaNet head, actually best applies for extracting features from a Gaussian pyramid, whose properties can hardly be satisfied by a feature pyramid. In order to alleviate this discrepancy, we build a scale-equalizing pyramid convolution (SEPC) that aligns the shared pyramid convolution kernel only at high-level feature maps. Being computationally efficient and compatible with the head design of most single-stage object detectors, the SEPC module brings significant performance improvement ($>4$AP increase on MS-COCO2017 dataset) in state-of-the-art one-stage object detectors, and a light version of SEPC also has $sim3.5$AP gain with only around 7% inference time increase. The pyramid convolution also functions well as a stand-alone module in two-stage object detectors and is able to improve the performance by $sim2$AP. The source code can be found at https://github.com/jshilong/SEPC.
Galaxy spins can be predicted from the initial conditions in the early Universe through the tidal tensor twist. In simulations, their directions are well preserved through cosmic time, consistent with expectations of angular momentum conservation. We
report a $sim 3 sigma$ detection of correlation between observed oriented directions of galaxy angular momenta and their predictions based on the initial density field reconstructed from the positions of SDSS galaxies. The detection is driven by a group of spiral galaxies classified by the Galaxy Zoo as (anti-)clockwise, with a modest improvement from adding galaxies from MaNGA and SAMI surveys. This is the first such detection of the oriented galaxy spin direction, which opens a way to use measurements of galaxy spins to probe fundamental physics in the early Universe.
N-body simulations are essential tools in physical cosmology to understand the large-scale structure (LSS) formation of the Universe. Large-scale simulations with high resolution are important for exploring the substructure of universe and for determ
ining fundamental physical parameters like neutrino mass. However, traditional particle-mesh (PM) based algorithms use considerable amounts of memory, which limits the scalability of simulations. Therefore, we designed a two-level PM algorithm CUBE towards optimal performance in memory consumption reduction. By using the fixed-point compression technique, CUBE reduces the memory consumption per N-body particle toward 6 bytes, an order of magnitude lower than the traditional PM-based algorithms. We scaled CUBE to 512 nodes (20,480 cores) on an Intel Cascade Lake based supercomputer with $simeq$95% weak-scaling efficiency. This scaling test was performed in Cosmo-$pi$ -- a cosmological LSS simulation using $simeq$4.4 trillion particles, tracing the evolution of the universe over $simeq$13.7 billion years. To our best knowledge, Cosmo-$pi$ is the largest completed cosmological N-body simulation. We believe CUBE has a huge potential to scale on exascale supercomputers for larger simulations.
The cosmic neutrino background is an important component of the Universe that is difficult to include in cosmological simulations due to the extremely large velocity dispersion of neutrino particles. We develop a new approach to simulate cosmic neutr
inos that decomposes the Fermi-Dirac phase space into shells of constant speed and then evolves those shells using hydrodynamic equations. These collisionless hydrodynamic equations are chosen to match linear theory, free particle evolution and allow for superposition. We implement this method into the information-optimized cosmological $N$-body code CUBE and demonstrate that neutrino perturbations can be accurately resolved to at least $ksim1 h/$Mpc. This technique allows for neutrino memory requirements to be decreased by up to $sim 10^3$ compared to traditional $N$-body methods.
Chiral symmetry is maximally violated in weak interactions, and such microscopic asymmetries in the early Universe might leave observable imprints on astrophysical scales without violating the cosmological principle. In this Letter, we propose a heli
city measurement to detect primordial chiral violation. We point out that observations of halo-galaxy angular momentum directions (spins), which are frozen in during the galaxy formation process, provide a fossil chiral observable. From the clustering mode of large scale structure of the Universe, we construct a spin mode in Lagrangian space and show in simulations that it is a good probe of halo-galaxy spins. In standard model, a strong symmetric correlation between the left and right helical components of this spin mode and galaxy spins is expected. Measurements of these correlations will be sensitive to chiral breaking, providing a direct test of chiral symmetry breaking in the early Universe.
