Background: Multinucleon transfer (MNT) and quasifission (QF) processes are dominant processes in low-energy collisions of two heavy nuclei. They are expected to be useful to produce neutron-rich unstable nuclei. Nuclear dynamics leading to these processes depends sensitively on nuclear properties such as deformation and shell structure. Purpose: We elucidate reaction mechanisms of MNT and QF processes involving heavy deformed nuclei, making detailed comparisons between microscopic time-dependent Hartree-Fock (TDHF) calculations and measurements for the $^{64}$Ni+$^{238}$U reaction. Methods: Three-dimensional Skyrme-TDHF calculations are performed. Particle-number projection method is used to evaluate MNT cross sections from the TDHF wave function after collision. Results: Fragment masses, total kinetic energy (TKE), scattering angle, contact time, and MNT cross sections are investigated for the $^{64}$Ni+$^{238}$U reaction. They show reasonable agreements with measurements. At small impact parameters, collision dynamics depends sensitively on the orientation of deformed $^{238}$U. In tip (side) collisions, we find a larger (smaller) TKE and a shorter (longer) contact time. In tip collisions, we find a strong influence of quantum shells around $^{208}$Pb. Conclusions: It is confirmed that the TDHF calculations reasonably describe both MNT and QF processes in the $^{64}$Ni+$^{238}$U reaction. Analyses of this system indicates the significance of the nuclear structure effects such as deformation and quantum shells in nuclear reaction dynamics at low energies.