Context: Planet formation by pebble accretion is an alternative to planetesimal-driven core accretion. In this scenario, planets grow by accreting cm-to-m-sized pebbles instead of km-sized planetesimals. One of the main differences with planetesimal-driven core accretion is the increased thermal ablation experienced by pebbles. This provides early enrichment to the planets envelope, which changes the process of core growth. Aims: We aim to predict core masses and envelope compositions of planets that form by pebble accretion and compare mass deposition of pebbles to planetesimals. Methods: We model the early growth of a proto-planet by calculating the structure of its envelope, taking into account the fate of impacting pebbles or planetesimals. The region where high-Z material can exist in vapor form is determined by the vapor pressure. We include enrichment effects by locally modifying the mean molecular weight. Results: In the pebble case, three phases of core growth can be identified. In the first phase, pebbles impact the core without significant ablation. During the second phase, ablation becomes increasingly severe. A layer of high-Z vapor starts to form around the core that absorbs a small fraction of the ablated mass. The rest either rains out to the core or mixes outwards instead, slowing core growth. In the third phase, the vapor inner region expands outwards, absorbing an increasing fraction of the ablated material as vapor. Rainout ends before the core mass reaches 0.6 M_Earth, terminating direct core growth. In the case of icy H2O pebbles, this happens before 0.1 M_Earth. Conclusions: Our results indicate that pebble accretion can directly form rocky cores up to only 0.6 M_Earth, and is unable to form similarly sized icy cores. Subsequent core growth can proceed indirectly when the planet cools, provided it is able to retain its high-Z material.