In an ideal two-component two-dimensional electron system, particle-hole symmetry dictates that the fractional quantum Hall states around $ u = 1/2$ are equivalent to those around $ u = 3/2$. We demonstrate that composite fermions (CFs) around $ u = 1/2$ in AlAs possess a valley degree of freedom like their counterparts around $ u = 3/2$. However, focusing on $ u = 2/3$ and 4/3, we find that the energy needed to completely valley polarize the CFs around $ u = 1/2$ is considerably smaller than the corresponding value for CFs around $ u = 3/2$ thus betraying a particle-hole symmetry breaking.