We have prepared polycrystalline samples of LaSrRh$_{1-x}$Ga$_x$O$_4$ and LaSr$_{1-x}$Ca$_x$RhO$_4$,and have measured the x-ray diffraction, resistivity, Seebeck coefficient, magnetization and electron spin resonance in order to evaluate their electronic states. The energy gap evaluated from the resistivity and the Seebeck coefficient systematically changes with the Ga concentration, and suggests that the system changes from a small polaron insulator to a band insulator. We find that all the samples show Curie-Weiss-like susceptibility with a small Weiss temperature of the order of 1 K, which is seriously incompatible with the collective wisdom that a trivalent rhodium ion is nonmagnetic. We have determined the $g$ factor to be $g$=2.3 from the electron spin resonance, and the spin number to be $S$=1 from the magnetization-field curves by fitting with a modified Brillouin function. The fraction of the $S$=1 spins is 2--5%, which depends on the degree of disorder in the La/Sr/Ca-site, which implies that disorder near the apical oxygen is related to the magnetism of this system. A possible origin for the magnetic Rh$^{3+}$ ions is discussed.