Relaxor behavior and lattice dynamics have been studied for a single crystal of K$_{1-x}$Li$_x$TaO$_3$ $(x=0.05)$, where a small amount of a Ca impurity ($sim 15$~ppm) was incorporated. The dielectric measurements revealed Debye-like relaxations with Arrhenius activation energies of 80 and 240 meV that are assigned to Li$^+$ dipoles and the Li$^+$-Li$^+$ dipolar pairs, respectively. In the neutron scattering results, diffuse scattering ridges appear around the nuclear Bragg peaks below $sim 150$ K and phonon line broadening features start to appear at even higher temperatures suggesting that polar nano-regions (PNRs) start to form at these temperatures. These results are supported by the dielectric data that reveal relaxor behavior starting at $sim 200$ K on cooling. From analyses of the diffuse intensities, the displacements include a uniform phase shift of all of the atoms in addition to the atomic displacements corresponding to a polarization vector of the transverse optic soft ferroelectric mode, a finding that is analogous to that in the prototypical relaxor material Pb(Mg$_{1/3}$Nb$_{2/3}$)O$_3$.