We combine the Density Matrix Renormalization Group (DMRG) with Matrix Product State tangent space concepts to construct a variational algorithm for finding ground states of one dimensional quantum lattices in the thermodynamic limit. A careful comparison of this variational uniform Matrix Product State algorithm (VUMPS) with infinite Density Matrix Renormalization Group (IDMRG) and with infinite Time Evolving Block Decimation (ITEBD) reveals substantial gains in convergence speed and precision. We also demonstrate that VUMPS works very efficiently for Hamiltonians with long range interactions and also for the simulation of two dimensional models on infinite cylinders. The new algorithm can be conveniently implemented as an extension of an already existing DMRG implementation.