In this paper, we explore a canonical connection between the algebra of $q$-difference operators $widetilde{V}_{q}$, affine Lie algebra and affine vertex algebras associated to certain subalgebra $mathcal{A}$ of the Lie algebra $mathfrak{gl}_{infty}$. We also introduce and study a category $mathcal{O}$ of $widetilde{V}_{q}$-modules. More precisely, we obtain a realization of $widetilde{V}_{q}$ as a covariant algebra of the affine Lie algebra $widehat{mathcal{A}^{*}}$, where $mathcal{A}^{*}$ is a 1-dimensional central extension of $mathcal{A}$. We prove that restricted $widetilde{V_{q}}$-modules of level $ell_{12}$ correspond to $mathbb{Z}$-equivariant $phi$-coordinated quasi-modules for the vertex algebra $V_{widetilde{mathcal{A}}}(ell_{12},0)$, where $widetilde{mathcal{A}}$ is a generalized affine Lie algebra of $mathcal{A}$. In the end, we show that objects in the category $mathcal{O}$ are restricted $widetilde{V_{q}}$-modules, and we classify simple modules in the category $mathcal{O}$.