We report on a study of electronic transport in semi-metallic Bi$_{0.96}$Sb$_{0.04}$. At zero field, the system is a very dilute Fermi liquid displaying a T$^{2}$ resistivity with an enhanced prefactor. Quantum oscillations in resistivity as well as in Hall, Nernst and Seebeck responses of the system are detectable and their period quantifies the shrinking of the Fermi surface with antimony doping. For a field along the trigonal axis, the quantum limit was found to occur at a field as low as 3T. An ultraquantum anomaly at twice this field was detected in both charge transport and Nernst response. Its origin appears to lie beyond the one-particle picture and linked to unidentified many-body effects.