We present a second-order-in-time finite difference scheme for the Cahn-Hilliard-Hele-Shaw equations. This numerical method is uniquely solvable and unconditionally energy stable. At each time step, this scheme leads to a system of nonlinear equations that can be efficiently solved by a nonlinear multigrid solver. Owing to the energy stability, we derive an $ell^2 (0,T; H_h^3)$ stability of the numerical scheme. To overcome the difficulty associated with the convection term $ abla cdot (phi boldsymbol{u})$, we perform an $ell^infty (0,T; H_h^1)$ error estimate instead of the classical $ell^infty (0,T; ell^2)$ one to obtain the optimal rate convergence analysis. In addition, various numerical simulations are carried out, which demonstrate the accuracy and efficiency of the proposed numerical scheme.