Since its elaboration by Whitham, almost fifty years ago, modulation theory has been known to be closely related to the stability of periodic traveling waves. However, it is only recently that this relationship has been elucidated, and that fully nonlinear results have been obtained. These only concern dissipative systems though: reaction-diffusion systems were first considered by Doelman, Sandstede, Scheel, and Schneider [Mem. Amer. Math. Soc. 2009], and viscous systems of conservation laws have been addressed by Johnson, Noble, Rodrigues, and Zumbrun [preprint 2012]. Here, only nondissipative models are considered, and a most basic question is investigated, namely the expected link between the hyperbolicity of modulated equations and the spectral stability of periodic traveling waves to sideband perturbations. This is done first in an abstract Hamiltonian framework, which encompasses a number of dispersive models, in particular the well-known (generalized) Korteweg--de Vries equation, and the less known Euler--Korteweg system, in both Eulerian coordinates and Lagrangian coordinates. The latter is itself an abstract framework for several models arising in water waves theory, superfluidity, and quantum hydrodynamics. As regards its application to compressible capillary fluids, attention is paid here to untangle the interplay between traveling waves/modulation equations in Eulerian coordinates and those in Lagrangian coordinates. In the most general setting, it is proved that the hyperbolicity of modulated equations is indeed necessary for the spectral stability of periodic traveling waves. This extends earlier results by Serre [Comm. Partial Differential Equations 2005], Oh and Zumbrun [Arch. Ration. Mech. Anal. 2003], and Johnson, Zumbrun and Bronski [Phys. D 2010]. In addition, reduced necessary conditions are obtained in the small amplitude limit. Then numerical investigations are carried out for the modulated equations of the Euler--Korteweg system with two types of pressure laws, namely the quadratic law of shallow water equations, and the nonmonotone van der Waals pressure law. Both the evolutionarity and the hyperbolicity of the modulated equations are tested, and regions of modulational instability are thus exhibited.