In the theoretical research of Hepatitis B virus, mathematical models of its transmission mechanism have been thoroughly investigated, while the dynamics of the immune process in vivo have not. At present, nearly all the existing models are based on integer-order differential equations. However, models restricted to integer-order cannot depict the complex dynamical behaviors of the virus. Since fractional-order models possess property of memory and memory is the main feature of immune response, we propose a fractional-order model of Hepatitis B with time-delay to further explore the dynamical characters of the HBV model. First, by using the Caputo fractional differential and its properties, we obtain the existence and uniqueness of the solution. Then by utilizing stability analysis of fractional-order system, the stability results of the disease-free equilibrium and the epidemic equilibrium are studied according to the value of the basic reproduction number. Moreover, the bifurcation diagram, the largest Lyapunov exponent diagram, phase diagram, Poincare section and frequency spectrum are employed to confirm the chaotic characteristics of the fractional HBV model and to figure out the effects of time-delay and fractional order. Further, a theory of the asymptotical stability of nonlinear autonomous system with delay on the basis of Caputo fractional differential is proved. The results of our work illustrates the rich dynamics of fractional HBV model with time-delay, and can provide theoretical guidance to virus dynamics study and clinical practice to some extent.