Cold dark matter (CDM) could be composed of primordial black holes (PBH) in addition to or instead of more orthodox weakly interacting massive particle dark matter (PDM). We study the formation of the first structures in such $Lambda$PBH cosmologies using $N$-body simulations evolved from deep in the radiation era to redshift 99. When PBH are only a small component of the CDM, they are clothed by PDM to form isolated halos. On the other hand, when PBH make most of the CDM, halos can also grow via clustering of many PBH. We find that the halo mass function is well modelled via Poisson statistics assuming random initial conditions. We quantify the nonlinear velocities induced by structure formation and find that they are too small to significantly impact CMB constraints. A chief challenge is how best to extrapolate our results to lower redshifts relevant for some observational constraints.