We investigate the Yangian symmetry of scattering amplitudes in N=4 super Yang-Mills theory and show that its formulations in twistor and momentum twistor space can be interchanged. In particular we show that the full symmetry can be thought of as the Yangian of the dual superconformal algebra, annihilating the amplitude with the MHV part factored out. The equivalence of this picture with the one where the ordinary superconformal symmetry is thought of as fundamental is an algebraic expression of T-duality. Motivated by this, we analyse some recently proposed formulas, which reproduce different contributions to amplitudes through a Grassmannian integral. We prove their Yangian invariance by directly applying the generators.