Small objects can swim by generating around them fields or gradients which in turn induce fluid motion past their surface by phoretic surface effects. We quantify for arbitrary swimmer shapes and surface patterns, how efficient swimming requires both surface ``activity to generate the fields, and surface ``phoretic mobility. We show in particular that (i) swimming requires symmetry breaking in either or both of the patterns of activity and ``mobility, and (ii) for a given geometrical shape and surface pattern, the swimming velocity is size-independent. In addition, for given available surface properties, our calculation framework provides a guide for optimizing the design of swimmers.