High spin states of neutron deficient Trans-Lead nucleus $^{204}$At were populated up to $E_x sim 8,{rm MeV}$ through the $^{12}$C + $^{197}$Au fusion evaporation reaction. Decay of the high spin states including prompt and delayed gamma ray emission were studied to understand the underlying nuclear structure. The level scheme, which was partly known from earlier studies, was extended further through our experiment and analysis of spin and parity of the associated levels. An isomeric $16^+$ level $(tau=52(5), {rm ns})$, corresponding to $M2$ transition, was established from our measurements. Attempts were made at interpretation of the excited states based on multi quasiparticle and hole structure involving $2f_{5/2}$, $1h_{9/2}$, and $1i_{13/2}$ shell model states, along with moderate core excitation. Magnetic dipole band structure over the spin parity range:~$16^+ - 23^+$, which was found in the earlier Gammasphere study, was confirmed and explored in more detail, including the missing cross-over $E2$ transitions. Band-crossing along the shears band was observed and compared with the evidence of similar phenomena in the neighboring neutron deficient $^{202}$Bi, $^{205}$Rn isotones and the neighbouring $^{203}$At isotope. Based on comparison of the measured $B(M1)/B(E2)$ values for transitions along the band with the semiclassical model based estimates, the shears band of $^{204}$At was firmly established along with the level scheme.