Pb(Fe$_{0.5}$Nb$_{0.5}$)O$_3$ (PFN), one of the few relaxor multiferroic systems, has a $G$-type antiferromagnetic transition at $T_N$ = 143 K and a ferroelectric transition at $T_C$ = 385 K. By using high-resolution neutron-diffraction experiments and a total scattering technique, we paint a comprehensive picture of the long- and short-range structures of PFN: (i) a clear sign of short-range structural correlation above $T_C$, (ii) no sign of the negative thermal expansion behavior reported in a previous study, and (iii) clearest evidence thus far of magnetoelectric coupling below $T_N$. We conclude that at the heart of the unusual relaxor multiferroic behavior lies the disorder between Fe$^{3+}$ and Nb$^{5+}$ atoms. We argue that this disorder gives rise to short-range structural correlations arising from O disorder in addition to Pb displacement.