We study the transmission spectra of ultracold rubidium atoms coupled to a high-finesse optical cavity. Under weak probing with pi-polarized light, the linear response of the system is that of a collective spin with multiple levels coupled to a single mode of the cavity. By varying the atom number, we change the collective coupling of the system. We observe the change in transmission spectra when going from a regime where the collective coupling is much smaller than the separation of the atomic levels to a regime where both are of comparable size. The observations are in good agreement with a reduced model we developed for our system.