The distinguishing number of $G leqslant sym(Omega)$ is the smallest size of a partition of $Omega$ such that only the identity of $G$ fixes all the parts of the partition. Extending earlier results of Cameron, Neumann, Saxl and Seress on the distinguishing number of finite primitive groups, we show that all imprimitive quasiprimitive groups have distinguishing number two, and all non-quasiprimitive semiprimitive groups have distinguishing number two, except for $mathrm{GL}(2, 3)$ acting on the eight non-zero vectors of $mathbb F_2^3$, which has distinguishing number three.