We study algebras and correlation functions of local operators at half-BPS interfaces engineered by the stacks of D5 or NS5 branes in the 4d $mathcal{N}=4$ super Yang-Mills. The operator algebra in this sector is isomorphic to a truncation of the Yangian $mathcal{Y}(mathfrak{gl}_n)$. The correlators, encoded in a trace on the Yangian, are controlled by the inhomogeneous $mathfrak{sl}_n$ spin chain, where $n$ is the number of fivebranes: they are given in terms of matrix elements of transfer matrices associated to Verma modules, or equivalently of products of Baxters Q-operators. This can be viewed as a novel connection between the $mathcal{N}=4$ super Yang-Mills and integrable spin chains. We also remark on analogous constructions involving half-BPS Wilson lines.