We compute non-extremal three-point functions of scalar operators in $mathcal{N}=4$ super Yang-Mills at tree-level in $g_{YM}$ and at finite $N_c$, using the operator basis of the restricted Schur characters. We make use of the diagrammatic methods called quiver calculus to simplify the three-point functions. The results involve an invariant product of the generalized Racah-Wigner tensors ($6j$ symbols). Assuming that the invariant product is written by the Littlewood-Richardson coefficients, we show that the non-extremal three-point functions satisfy the large $N_c$ background independence; correspondence between the string excitations on $AdS_5 times S^5$ and those in the LLM geometry.