We present a long-exposure (~10 hr) image of the supernova (SN) remnant Cassiopeia A (Cas A) obtained with the UKIRT 3.8-m telescope using a narrow band filter centered at 1.644 um emission. The passband contains [Fe II] 1.644 um and [Si I] 1.645 um lines, and our `deep [Fe II]+[Si I] image provides an unprecedented panoramic view of Cas A, showing both shocked and unshocked SN ejecta together with shocked circumstellar medium at subarcsec (~0.7 arcsec or 0.012 pc) resolution. The diffuse emission from the unshocked SN ejecta has a form of clumps, filaments, and arcs, and their spatial distribution correlates well with that of the Spitzer [Si II] infrared emission, suggesting that the emission is likely due to [Si I] line not [Fe II] line as in shocked material. The structure of the optically-invisible western area of Cas A is clearly seen for the first time. The area is filled with many Quasi-Stationary Flocculi (QSFs) and fragments of the disrupted ejecta shell. We suggest that the anomalous radio properties in this area could be due to the increased number of such dense clumps. We identified 309 knots in the deep [Fe II]+[Si I] image and classified them into QSFs and fast-moving knots (FMKs). The total H+He mass of QSFs is ~0.23 Msun, implying that the mass fraction of dense clumps in the progenitors red-supergiant wind is 4--13%. The spatial distribution of QSFs suggests that there had been a highly asymmetric mass loss $10^4$--$10^5$ yr before the SN explosion. The mass of the [Fe II] line-emitting, shocked dense Fe ejecta is ~3x$10^{-5}$ Msun. The comparison with the ionic S-line dominated Hubble Space Telescope WFC3/IR image suggests that the outermost FMKs in the southeastern area are Fe-rich.