We present a measurement of the gravitational lensing of the Cosmic Microwave Background (CMB) temperature and polarization fields obtained by cross-correlating the reconstructed convergence signal from the first season of ACTPol data at 146 GHz with Cosmic Infrared Background (CIB) fluctuations measured using the Planck satellite. Using an overlap area of 206 square degrees, we detect gravitational lensing of the CMB polarization by large-scale structure at a statistical significance of 4.5 sigma. Combining both CMB temperature and polarization data gives a lensing detection at 9.1 sigma significance. A B-mode polarization lensing signal is present with a significance of 3.2 sigma. We also present the first measurement of CMB lensing--CIB correlation at small scales corresponding to l > 2000. Null tests and systematic checks show that our results are not significantly biased by astrophysical or instrumental systematic effects, including Galactic dust. Fitting our measurements to the best-fit lensing-CIB cross power spectrum measured in Planck data, scaled by an amplitude A, gives A=1.02 +0.12/-0.18 (stat.) +/-0.06(syst.), consistent with the Planck results.