The breathing pyrochlore lattice material Ba$_3$Yb$_2$Zn$_5$O$_{11}$ exists in the nearly decoupled limit, in contrast to most other well-studied breathing pyrochlore compounds. As a result, it constitutes a useful platform to benchmark theoretical calculations of exchange interactions in insulating Yb$^{3+}$ magnets. Here we study Ba$_3$Yb$_2$Zn$_5$O$_{11}$ at low temperatures in applied magnetic fields as a further probe of the physics of this model system. Experimentally, we consider the behavior of polycrystalline samples of Ba$_3$Yb$_2$Zn$_5$O$_{11}$ with a combination of inelastic neutron scattering and heat capacity measurements down to 75 mK and up to fields of 10 T. Consistent with previous work, inelastic neutron scattering finds a level crossing near 3 T, but no significant dispersion of the spin excitations is detected up to the highest applied fields. Refinement of the theoretical model previously determined at zero field can reproduce much of the inelastic neutron scattering spectra and specific heat data. A notable exception is a low temperature peak in the specific heat near 0.1 K. This may indicate the scale of interactions between tetrahedra or may reflect undetected disorder in Ba$_3$Yb$_2$Zn$_5$O$_{11}$.