Optical methods are most convenient to analyze spatially periodic patterns with wavevector $bm q$ in a thin layer of a nematic liquid crystal. In the standard experimental setup a beam of parallel light with a short wavelength $lambda ll 2 pi/q$ passes the nematic layer. Recording the transmitted light the patterns are either directly visualized by shadowgraphy or characterized more indirectly by the diffraction fringes due to the optical grating effects of the pattern. In this work we present a systematic short-wavelength analysis of these methods for the commonly used planar orientation of the optical axis of liquid crystal at the confining surfaces. Our approach covers general 3D experimental geometries with respect to the relative orientation of $bm q$ and of the wavevector $bm k$ of the incident light. In particular the importance of phase grating effects is emphasized, which are not accessible in a pure geometric optics approach. Finally, as a byproduct we present also an optical analysis of convection rolls in Rayleigh-Benard convection, where the refraction index of the fluid is isotropic in contrast to its uniaxial symmetry in nematic liquid crystals. Our analysis is in excellent agreement with an earlier physical optics approach by Trainoff and Cannell [Physics of Fluids {bf 14}, 1340 (2002)], which is restricted to a 2D geometry and technically much more demanding.