The differential affine velocity estimator (DAVE) developed in Schuck (2006) for estimating velocities from line-of-sight magnetograms is modified to directly incorporate horizontal magnetic fields to produce a differential affine velocity estimator for vector magnetograms (DAVE4VM). The DAVE4VMs performance is demonstrated on the synthetic data from the anelastic pseudospectral ANMHD simulations that were used in the recent comparison of velocity inversion techniques by Welsch (2007). The DAVE4VM predicts roughly 95% of the helicity rate and 75% of the power transmitted through the simulation slice. Inter-comparison between DAVE4VM and DAVE and further analysis of the DAVE method demonstrates that line-of-sight tracking methods capture the shearing motion of magnetic footpoints but are insensitive to flux emergence -- the velocities determined from line-of-sight methods are more consistent with horizontal plasma velocities than with flux transport velocities. These results suggest that previous studies that rely on velocities determined from line-of-sight methods such as the DAVE or local correlation tracking may substantially misrepresent the total helicity rates and power through the photosphere.