In this paper, we study contragredient duals and invariant bilinear forms for modular vertex algebras (in characteristic $p$). We first introduce a bialgebra $mathcal{H}$ and we then introduce a notion of $mathcal{H}$-module vertex algebra and a notion of $(V,mathcal{H})$-module for an $mathcal{H}$-module vertex algebra $V$. Then we give a modular version of Frenkel-Huang-Lepowskys theory and study invariant bilinear forms on an $mathcal{H}$-module vertex algebra. As the main results, we obtain an explicit description of the space of invariant bilinear forms on a general $mathcal{H}$-module vertex algebra, and we apply our results to affine vertex algebras and Virasoro vertex algebras.