Millimeter-wave (mmWave) and terahertz (THz) spectrum can support significantly higher data rates compared to lower frequency bands and hence are being actively considered for 5G wireless networks and beyond. These bands have high free-space path loss (FSPL) in line-of-sight (LOS) propagation due to their shorter wavelength. Moreover, in non-line-of-sight (NLOS) scenario, these two bands suffer higher penetration loss than lower frequency bands which could seriously affect the network coverage. It is therefore critical to study the NLOS penetration loss introduced by different building materials at mmWave and THz bands, to help establish link budgets for an accurate performance analysis in indoor environments. In this work, we measured the penetration loss and the attenuation of several common constructional materials at mmWave (28 and 39 GHz) and sub-THz (120 and 144 GHz) bands. Measurements were conducted using a channel sounder based on NI PXI platforms. Results show that the penetration loss changes extensively based on the frequency and the material properties, ranging from 0.401 dB for ceiling tile at 28 GHz, to 16.608 dB for plywood at 144 GHz. Ceiling tile has the lowest measured attenuation at 28 GHz, while clear glass has the highest attenuation of 27.633 dB/cm at 144 GHz. As expected, the penetration loss and attenuation increased with frequency for all the tested materials.