We construct actions of the affine Yangian of type A on higher level Fock spaces by extending known actions of the Yangian of finite type A due to Uglov. This is a degenerate analog of a result by Takemura-Uglov, which constructed actions of the quantum toroidal algebra on higher level $q$-deformed Fock spaces.