We compare the flow-like correlations in high multiplicity proton-nucleus ($p+A$) and nucleus-nucleus ($A+A$) collisions. At fixed multiplicity, the correlations in these two colliding systems are strikingly similar, although the system size is smaller in $p+A$. Based on an independent cluster model and a simple conformal scaling argument, where the ratio of the mean free path to the system size stays constant at fixed multiplicity, we argue that flow in $p+A$ emerges as a collective response to the fluctuations in the position of clusters, just like in $A+A$ collisions. With several physically motivated and parameter free rescalings of the recent LHC data, we show that this simple model captures the essential physics of elliptic and triangular flow in $p+A$ collisions. We also explore the implications of the model for jet energy loss in $p+A$, and predict slightly larger transverse momentum broadening in $p+A$ than in $A+A$ at the same multiplicity.