We present a theory of viscoelasticity of amorphous media, which takes into account the effects of confinement along one of three spatial dimensions. The framework is based on the nonaffine extension of lattice dynamics to amorphous systems, or nonaffine response theory. The size effects due to the confinement are taken into account via the nonaffine part of the shear storage modulus $G$. The nonaffine contribution is written as a sum over modes in $k$-space. With a rigorous argument based on the analysis of the $k$-space integral over modes, it is shown that the confinement size $L$ in one spatial dimension, e.g. the $z$ axis, leads to a infrared cut-off for the modes contributing to the nonaffine (softening) correction to the modulus that scales as $L^{-3}$. Corrections for finite sample size $D$ in the two perpendicular dimensions scale as $sim (L/D)^4$, and are negligible for $L ll D$. For liquids it is predicted that $Gsim L^{-3}$ in agreement with a previous more approximate analysis, whereas for amorphous materials $G sim G_{bulk} + beta L^{-3}$. For the case of liquids, four different experimental systems are shown to be very well described by the $L^{-3}$ law.