We report high-resolution capacitance dilatometry studies on the uniaxial length changes in a NdB$_4$ single crystal. The evolution of magnetically ordered phases below $T_{rm N}$= 17.2~K (commensurate antiferromagnetic phase, cAFM), $T_{rm IT}$= 6.8~K (intermediate incommensurate phase, IT), and $T_{rm LT}$= 4.8~K (low-temperature phase, LT) is associated with pronounced anomalies in the thermal expansion coefficients. The data imply significant magneto-elastic coupling and evidence of a structural phase transition at $T_{rm LT}$ . While both cAFM and LT favor structural anisotropy $delta$ between in-plane and out-of-plane length changes, it competes with the IT-type of order, i.e., $delta$ is suppressed in that phase. Notably, finite anisotropy well above $T_{rm N}$ indicates short-range correlations which are, however, of neither cAFM, IT, nor LT-type. Gruneisen analysis of the ratio of thermal expansion coefficient and specific heat enables the derivation of uniaxial as well as hydrostatic pressure dependencies. While $alpha$/$c_{rm p}$ evidences a single dominant energy scale in LT, our data imply precursory fluctuations of a competing phase in IT and cAFM, respectively. Our results suggest the presence of orbital degrees of freedom competing with cAFM and successive evolution of a magnetically and orbitally ordered ground state.