Nematic orders emerge nearly universally in iron-based superconductors, but elucidating their origins is challenging because of intimate couplings between orbital and magnetic fluctuations. The iron-based ladder material BaFe2S3, which superconducts under pressure, exhibits antiferromagnetic order below TN ~ 117K and a weak resistivity anomaly at T* ~ 180K, whose nature remains elusive. Here we report angle-resolved magnetoresistance (MR) and elastoresistance (ER) measurements in BaFe2S3, which reveal distinct changes at T*. We find that MR anisotropy and ER nematic response are both suppressed near T*, implying that an orbital order promoting isotropic electronic states is stabilized at T*. Such an isotropic state below T* competes with the antiferromagnetic order, which is evidenced by the nonmonotonic temperature dependence of nematic fluctuations. In contrast to the cooperative nematic orders in spin and orbital channels in iron pnictides, the present competing orders can provide a new platform to identify the separate roles of orbital and magnetic fluctuations.