Given two symmetric and positive semidefinite square matrices $A, B$, is it true that any matrix given as the product of $m$ copies of $A$ and $n$ copies of $B$ in a particular sequence must be dominated in the spectral norm by the ordered matrix product $A^m B^n$? For example, is $$ | AABAABABB | leq | AAAAABBBB | ? $$ Drury has characterized precisely which disordered words have the property that an inequality of this type holds for all matrices $A,B$. However, the $1$-parameter family of counterexamples Drury constructs for these characterizations is comprised of $3 times 3$ matrices, and thus as stated the characterization applies only for $N times N$ matrices with $N geq 3$. In contrast, we prove that for $2 times 2$ matrices, the general rearrangement inequality holds for all disordered words. We also show that for larger $N times N$ matrices, the general rearrangement inequality holds for all disordered words, for most $A,B$ (in a sense of full measure) that are sufficiently small perturbations of the identity.