The ratio of the electric and magnetic form factor of the proton, $mu_p G_E^p/G_M^p$, has been measured for elastic electron-proton scattering with polarized beam and target up to four-momentum transfer squared, $Q^2=5.66$ (GeV/c)$^2$ using the double spin asymmetry for target spin orientation aligned nearly perpendicular to the beam momentum direction. This measurement of $mu_p G_E^p/G_M^p$ agrees with the $Q^2$ dependence of previous recoil polarization data and reconfirms the discrepancy at high $Q^2$ between the Rosenbluth and the polarization-transfer method with a different measurement technique and systematic uncertainties uncorrelated to those of the recoil-polarization measurements. The form factor ratio at $Q^2$=2.06 (GeV/c)$^2$ has been measured as $mu_p G_E^p/G_M^p = 0.720 pm 0.176_{stat} pm 0.039_{sys}$, which is in agreement with an earlier measurement with the polarized target technique at similar kinematics. The form factor ratio at $Q^2$=5.66 (GeV/c)$^2$ has been determined as $mu_p G_E^p/G_M^p=0.244pm0.353_{stat}pm0.013_{sys}$, which represents the highest $Q^2$ reach with the double spin asymmetry with polarized target to date.