Let $G$ be a simple graph with vertex set $V(G) = {v_1 ,v_2 ,cdots ,v_n}$. The Harary matrix $RD(G)$ of $G$, which is initially called the reciprocal distance matrix, is an $n times n$ matrix whose $(i,j)$-entry is equal to $frac{1}{d_{ij}}$ if $i ot=j$ and $0$ otherwise, where $d_{ij}$ is the distance of $v_i$ and $v_j$ in $G$. In this paper, we characterize graphs with maximum spectral radius of Harary matrix in three classes of simple connected graphs with $n$ vertices: graphs with fixed matching number, bipartite graphs with fixed matching number, and graphs with given number of cut edges, respectively.