We present a joint analysis of the rest-frame ultraviolet (UV) luminosity functions of continuum-selected star-forming galaxies and galaxies dominated by active galactic nuclei (AGNs) at $z sim$ 4. These 3,740 $z sim$ 4 galaxies are selected from broad-band imaging in nine photometric bands over 18 deg$^2$ in the textit{Spitzer}/HETDEX Exploratory Large Area Survey (SHELA) field. The large area and moderate depth of our survey provide a unique view of the intersection between the bright end of the galaxy UV luminosity function (M$_{AB}<-$22) and the faint end of the AGN UV luminosity function. We do not separate AGN-dominated galaxies from star-formation-dominated galaxies, but rather fit both luminosity functions simultaneously. These functions are best fit with a double power-law (DPL) for both the galaxy and AGN components, where the galaxy bright-end slope has a power-law index of $-3.80pm0.10$, and the corresponding AGN faint-end slope is $alpha_{AGN} = -1.49^{+0.30}_{-0.21}$. We cannot rule out a Schechter-like exponential decline for the galaxy UV luminosity function, and in this scenario the AGN luminosity function has a steeper faint-end slope of $-2.08^{+0.18}_{-0.11}$. Comparison of our galaxy luminosity function results with a representative cosmological model of galaxy formation suggests that the molecular gas depletion time must be shorter, implying that star formation is more efficient in bright galaxies at $z=4$ than at the present day. If the galaxy luminosity function does indeed have a power-law shape at the bright end, the implied ionizing emissivity from AGNs is not inconsistent with previous observations. However, if the underlying galaxy distribution is Schechter, it implies a significantly higher ionizing emissivity from AGNs at this epoch.