We study the effect of shear and bulk viscosities on the heavy quark transport coefficient within the matrix model of semi QGP. Dissipative effects are incorporated through the first-order viscous correction in the quark/antiquark and gluon distribution function. It is observed that while the shear viscosity effects reduces the drag of heavy quark the bulk viscosity effects increase the drag and the diffusion coefficients of heavy quark. For finite values of {eta}/s and {xi}/s, Polyakov loop further decreases the drag and the diffusion coefficients as compared to perturbative QCD.