The 1/2-BPS Wilson loop in $mathcal{N}=4$ supersymmetric Yang-Mills theory is an important and well-studied example of conformal defect. In particular, much work has been done for the correlation functions of operator insertions on the Wilson loop in the fundamental representation. In this paper, we extend such analyses to Wilson loops in the large-rank symmetric and antisymmetric representations, which correspond to probe D3 and D5 branes with $AdS_2 times S^2$ and $AdS_2 times S^4$ worldvolume geometries, ending at the $AdS_5$ boundary along a one-dimensional contour. We first compute the correlation functions of protected scalar insertions from supersymmetric localization, and obtain a representation in terms of multiple integrals that are similar to the eigenvalue integrals of the random matrix, but with important differences. Using ideas from the Fermi Gas formalism and the Clustering method, we evaluate their large $N$ limit exactly as a function of the t Hooft coupling. The results are given by simple integrals of polynomials that resemble the $Q$-functions of the Quantum Spectral Curve, with integration measures depending on the number of insertions. Next, we study the correlation functions of fluctuations on the probe D3 and D5 branes in AdS. We compute a selection of three- and four-point functions from perturbation theory on the D-branes, and show that they agree with the results of localization when restricted to supersymmetric kinematics. We also explain how the difference of the internal geometries of the D3 and D5 branes manifests itself in the localization computation.