We tackle the problem of finding a suitable categorical framework for generalized functions used in mathematical physics for linear and non-linear PDEs. We are looking for a Cartesian closed category which contains both Schwartz distributions and Colombeau generalized functions as natural objects. We study Frolicher spaces, diffeological spaces and functionally generated spaces as frameworks for generalized functions. The latter are similar to Frolicher spaces, but starting from locally defined functionals. Functionally generated spaces strictly lie between Frolicher spaces and diffeological spaces, and they form a complete and cocomplete Cartesian closed category. We deeply study functionally generated spaces (and Frolicher spaces) as a framework for Schwartz distributions, and prove that in the category of diffeological spaces, both the special and the full Colombeau algebras are smooth differential algebras, with a smooth embedding of Schwartz distributions and smooth pointwise evaluations of Colombeau generalized functions.