Quantum speed limit (QSL) under noise has drawn considerable attention in real quantum computational processes and quantum communication. Though non-Markovian noise is proven to be able to accelerate quantum evolution for a damped Jaynes-Cummings model, in this work we show that non-Markovianity may even slow down the quantum evolution of an experimentally controllable photon system. As an important application, QSL time of a photon can be well controlled by regulating the relevant environment parameter properly, which is close to reach the currently available photonic experimental technology.