We study the algebra of Wilson line operators in three-dimensional N=2 supersymmetric U(M) gauge theories with a Higgs phase related to a complex Grassmannian Gr(M,N), and its connection to K-theoretic Gromov-Witten invariants for Gr(M,N). For different Chern-Simons levels, the Wilson loop algebra realizes either the quantum cohomology of Gr(M,N), isomorphic to the Verlinde algebra for U(M), or the quantum K-theoretic ring of Schubert structure sheaves studied by mathematicians, or closely related algebras.