We use new deep 21 cm HI observations of the moderately inclined galaxy NGC 4559 in the HALOGAS survey to investigate the properties of extra-planar gas. We use TiRiFiC to construct simulated data cubes to match the HI observations. We find that a thick disk component of scale height $sim,2,mathrm{kpc}$, characterized by a negative vertical gradient in its rotation velocity (lag) of $sim13 pm 5$ km s$^{-1}$ kpc$^{-1}$ is an adequate fit to extra-planar gas features. The tilted ring models also present evidence for a decrease in the magnitude of the lag outside of $R_{25}$, and a radial inflow of $sim 10$ km s$^{-1}$. We extracted lagging extra-planar gas through Gaussian velocity profile fitting. From both the 3D models and and extraction analyses we conclude that $sim10-20%$ of the total {HI} mass is extra-planar. Most of the extra-planar gas is spatially coincident with regions of star formation in spiral arms, as traced by H$alpha$ and GALEX FUV images, so it is likely due to star formation processes driving a galactic fountain. We also find the signature of a filament of a kinematically forbidden HI, containing $sim 1.4times 10^{6}$ M$_{odot}$ of HI, and discuss its potential relationship to a nearby HI hole. We discover a previously undetected dwarf galaxy in HI located $sim 0.4^{circ}$ ($sim 58$ kpc) from the center of NGC 4559, containing $sim 4times10^{5}$ M$_{odot}$. This dwarf has counterpart sources in SDSS with spectra typical of HII regions, and we conclude it is two merging blue compact dwarf galaxies.