The article addresses the possibility of alloy elements in MnBi which may modify the thermodynamic stability of the NiAs-type structure without significantly degrading the magnetic properties. The addition of small amounts of Rh and Mn provides an improvement in the thermal stability with some degradation of the magnetic properties. The small amounts of Rh and Mn additions in MnBi stabilize an orthorhombic phase whose structural and magnetic properties are closely related to the ones of the previously reported high-temperature phase of MnBi (HT~MnBi). To date, the properties of the HT~MnBi, which is stable between $613$ and $719$~K, have not been studied in detail because of its transformation to the stable low-temperature MnBi (LT~MnBi), making measurements near and below its Curie temperature difficult. The Rh-stabilized MnBi with chemical formula Mn$_{1.0625-x}$Rh$_{x}$Bi [$x=0.02(1)$] adopts a new superstructure of the NiAs/Ni$_2$In structure family. It is ferromagnetic below a Curie temperature of $416$~K. The critical exponents of the ferromagnetic transition are not of the mean-field type but are closer to those associated with the Ising model in three dimensions. The magnetic anisotropy is uniaxial; the anisotropy energy is rather large, and it does not increase when raising the temperature, contrary to what happens in LT~MnBi. The saturation magnetization is approximately $3$~$mu_B$/f.u. at low temperatures. While this exact composition may not be application ready, it does show that alloying is a viable route to modifying the stability of this class of rare-earth-free magnet alloys.