We estimate the rate of inspiral for a population of stellar mass BHs in the star cluster around the super massive black hole at the center of Milky Way mass galaxies. Our approach is based on an orbit averaged Fokker Planck approach. This is then followed by a post-processing approach, which incorporates the impact of the angular momentum diffusion and the GW dissipation in the evolution of system. We make a sample of 10000 BHs with different initial semi-major and eccentricities with the distribution of $f_c(a)/a$ and $e$, respectively. Where $f_c(a)$ refers to the phase-space distribution function for cth species. Angular momentum diffusion leads to an enhancement in the eccentricity of every system in the above sample and so increases the rate of inspiral. We compute the fraction of time that every system spends in the LISA band with the signal to noise ratio $rm{SNR} geq 8$. Every system eventually approaches the loss-cone with a replenishment rate given by the diffusion rate of the cluster, $mu/ rm{Gyr}^{-1} lesssim 1 $. This small rate reduces the total rate of the inspiral for individual MW mass galaxies with an estimate $R_{obs} lesssim 10^{-5} yr^{-1}$. It is expected though that a collection of $N_{gal} simeq 10^4$ MW mass galaxies lead to an observable GW signal in the LISA band.