Recent observations point to the presence of structured dust grains in the discs surrounding young brown dwarfs, thus implying that the first stages of planet formation take place also in the sub-stellar regime. Here, we investigate the potential for planet formation around brown dwarfs and very low mass stars according to the sequential core accretion model of planet formation. We find that, for a brown dwarfs of mass 0.05M_{odot}, our models predict a maximum planetary mass of ~5M_{oplus}, orbiting with semi-major axis ~1AU. However, we note that the predictions for the mass - semi-major axis distribution are strongly dependent upon the models chosen for the disc surface density profiles and the assumed distribution of disc masses. In particular, if brown dwarf disc masses are of the order of a few Jupiter masses, Earth-mass planets might be relatively frequent, while if typical disc masses are only a fraction of Jupiter mass, we predict that planet formation would be extremely rare in the sub-stellar regime. As the observational constraints on disc profiles, mass dependencies and their distributions are poor in the brown dwarf regime, we advise caution in validating theoretical models only on stars similar to the Sun and emphasise the need for observational data on planetary systems around a wide range of stellar masses. We also find that, unlike the situation around solar-like stars, Type-II migration is totally absent from the planet formation process around brown dwarfs, suggesting that any future observations of planets around brown dwarfs would provide a direct measure of the role of other types of migration.