Millisecond pulsars (MSPs) and normal non-recycled pulsars are both detected in $gamma$-rays. However, it appears that a much larger fraction of known energetic and nearby MSPs are detected in $gamma$-rays, in comparison with normal pulsars, thereby making undetected $gamma$-ray MSPs exceptions. In this paper, we demonstrate that the viewing angles (i.e. between the pulsar spin axis and the line of sight) are well described by the orbital inclination angles which, for binary MSPs with helium white dwarf companions, can be determined using the relationship between the orbital period and the white dwarf mass. We use the predicted viewing angles, in complement with values obtained from other constraints when available, to identify the causes of non-detection of energetic and nearby MSPs from the point of view of beaming geometry and orientation. We find evidence for slightly different viewing angle distributions, and postulate that energetic and nearby MSPs are mainly undetected in $gamma$-rays simply because they are seen under unfavourable (i.e. small) viewing angles. We finally discuss the magnetic fields of $gamma$-ray detected pulsars and show that pulsars which are efficient at converting their rotational energy into $gamma$-ray emission may have overestimated dipolar magnetic field strengths.