We derive upper and lower bounds on the Casimir--Polder force between an anisotropic dipolar body and a macroscopic body separated by vacuum via algebraic properties of Maxwells equations. These bounds require only a coarse characterization of the system---the material composition of the macroscopic object, the polarizability of the dipole, and any convenient partition between the two objects---to encompass all structuring possibilities. We find that the attractive Casimir--Polder force between a polarizable dipole and a uniform planar semi-infinite bulk medium always comes within 10% of the lower bound, implying that nanostructuring is of limited use for increasing attraction. In contrast, the possibility of repulsion is observed even for isotropic dipoles, and is routinely found to be several orders of magnitude larger than any known design, including recently predicted geometries involving conductors with sharp edges. Our results have ramifications for the design of surfaces to trap, suspend, or adsorb ultracold gases.