We used the 10.4m Gran Telescopio Canarias to search for the optical counterparts to four isolated $gamma$-ray pulsars, all detected in the X-rays by either xmm or chan but not yet in the optical. Three of them are middle-aged pulsars -- PSR, J1846+0919 (0.36 Myr), PSR, J2055+2539 (1.2 Myr), PSR, J2043+2740 (1.2 Myr) -- and one, PSR, J1907+0602, is a young pulsar (19.5 kyr). For both PSR, J1907+0602 and PSR, J2055+2539 we found one object close to the pulsar position. However, in both cases such an object cannot be a viable candidate counterpart to the pulsar. For PSR, J1907+0602, because it would imply an anomalously red spectrum for the pulsar and for PSR, J2055+2539 because the pulsar would be unrealistically bright ($r=20.34pm0.04$) for the assumed distance and interstellar extinction. For PSR, J1846+0919, we found no object sufficiently close to the expected position to claim a possible association, whereas for PSR, J2043+2740 we confirm our previous findings that the object nearest to the pulsar position is an unrelated field star. We used our brightness limits ($g approx 27$), the first obtained with a large-aperture telescope for both PSR, J1846+0919 and PSR, J2055+2539, to constrain the optical emission properties of these pulsars and investigate the presence of spectral turnovers at low energies in their multi-wavelength spectra.