Gravitational encounters between small-scale dark matter substructure and cold stellar streams in the Milky Way halo lead to density perturbations in the latter, making streams an effective probe for detecting dark matter substructure. The Pal 5 stream is one such system for which we have some of the best data. However, Pal 5 orbits close to the center of the Milky Way and has passed through the Galactic disk many times, where its structure can be perturbed by baryonic structures such as the Galactic bar and giant molecular clouds (GMCs). In order to understand how these baryonic structures affect Pal 5s density, we present a detailed study of the effects of the Galactic bar, spiral structure, GMCs, and globular clusters on the Pal 5 stream. We estimate the effect of each perturber on the stream density by computing its power spectrum and comparing it to the power induced by a CDM-like population of dark matter subhalos. We find that the bar and GMCs can each individually create power that is comparable to the observed power on large scales, leaving little room for dark matter substructure, while spirals are subdominant on all scales. On degree scales, the power induced by the bar is small, but GMCs create small-scale density variations that are similar in amplitude to the dark-matter induced variations but otherwise indistinguishable from it. These results demonstrate that Pal 5 is a poor system for constraining the dark matter substructure fraction and that observing streams further out in the halo will be necessary to confidently detect dark matter subhalos.