Many cuprate superconductors possess an unusual charge-ordered phase that is characterized by an approximate $d_{x^2-y^2}$ intra-unit cell form factor and a finite modulation wavevector $bq^ast$. We study the effects impurities on this charge ordered phase via a single-band model in which bond order is the analogue of charge order in the cuprates. Impurities are assumed to be pointlike and are treated within the self-consistent t-matrix approximation (SCTMA). We show that suppression of bond order by impurities occurs through the local disruption of the $d_{x^2-y^2}$ form factor near individual impurities. Unlike $d$-wave superconductors, where the sensitivity of $T_c$ to impurities can be traced to a vanishing average of the $d_{x^2-y^2}$ order parameter over the Fermi surface, the response of bond order to impurities is dictated by a few Fermi surface hotspots. The bond order transition temperature $T_mathrm{bo}$ thus follows a different universal dependence on impurity concentration $n_i$ than does the superconducting $T_c$. In particular, $T_mathrm{bo}$ decreases more rapidly than $T_c$ with increasing $n_i$ when there is a nonzero Fermi surface curvature at the hotspots. Based on experimental evidence that the pseudogap is insensitive to Zn doping, we conclude that a direct connection between charge order and the pseudogap is unlikely. Furthermore, the enhancement of stripe correlations in the La-based cuprates by Zn doping is evidence that this charge order is also distinct from stripes.