We present the first measurements of long-range angular correlations and the transverse momentum dependence of elliptic flow $v_2$ in high-multiplicity $p$$+$Au collisions at $sqrt{s_{_{NN}}}=200$ GeV. A comparison of these results with previous measurements in high-multiplicity $d$$+$Au and $^3{rm He}$$+$Au collisions demonstrates a relation between $v_2$ and the initial collision eccentricity $varepsilon_2$, suggesting that the observed momentum-space azimuthal anisotropies in these small systems have a collective origin and reflect the initial geometry. Good agreement is observed between the measured $v_2$ and hydrodynamic calculations for all systems, and an argument disfavoring theoretical explanations based on momentum-space domain correlations is presented. The set of measurements presented here allows us to leverage the distinct intrinsic geometry of each of these systems to distinguish between different theoretical descriptions of the long-range correlations observed in small collision systems.