We consider the affine vertex algebra at the critical level associated with the centralizer of a nilpotent element in the Lie algebra $mathfrak{gl}_N$. Due to a recent result of Arakawa and Premet, the center of this vertex algebra is an algebra of polynomials. We construct a family of free generators of the center in an explicit form. As a corollary, we obtain generators of the corresponding quantum shift of argument subalgebras and recover free generators of the center of the universal enveloping algebra of the centralizer produced earlier by Brown and Brundan.