We present results for the measurement of $phi$ meson production via its charged kaon decay channel $phi to K^+K^-$ in Au+Au collisions at $sqrt{s_{_{NN}}}=62.4$, 130, and 200 GeV, and in $p+p$ and $d$+Au collisions at $sqrt{s_{_{NN}}}=200$ GeV from the STAR experiment at the BNL Relativistic Heavy Ion Collider (RHIC). The midrapidity ($|y|<0.5$) $phi$ meson transverse momentum ($p_{T}$) spectra in central Au+Au collisions are found to be well described by a single exponential distribution. On the other hand, the $p_{T}$ spectra from $p+p$, $d$+Au and peripheral Au+Au collisions show power-law tails at intermediate and high $p_{T}$ and are described better by Levy distributions. The constant $phi/K^-$ yield ratio vs beam species, collision centrality and colliding energy is in contradiction with expectations from models having kaon coalescence as the dominant mechanism for $phi$ production at RHIC. The $Omega/phi$ yield ratio as a function of $p_{T}$ is consistent with a model based on the recombination of thermal $s$ quarks up to $p_{T}sim 4$ GeV/$c$, but disagrees at higher transverse momenta. The measured nuclear modification factor, $R_{dAu}$, for the $phi$ meson increases above unity at intermediate $p_{T}$, similar to that for pions and protons, while $R_{AA}$ is suppressed due to the energy loss effect in central Au+Au collisions. Number of constituent quark scaling of both $R_{cp}$ and $v_{2}$ for the $phi$ meson with respect to other hadrons in Au+Au collisions at $sqrt{s_{_{NN}}}$=200 GeV at intermediate $p_{T}$ is observed. These observations support quark coalescence as being the dominant mechanism of hadronization in the intermediate $p_{T}$ region at RHIC.