We report secondary eclipse photometry of the hot Jupiter HAT-P-32Ab, taken with Hale/WIRC in H and Ks bands and with Spitzer/IRAC at 3.6 and 4.5 micron. We carried out adaptive optics imaging of the planet host star HAT-P-32A and its companion HAT-P-32B in the near-IR and the visible. We clearly resolve the two stars from each other and find a separation of 2.923 +/- 0. 004 and a position angle 110.64 deg +/- 0.12 deg. We measure the flux ratios of the binary in g r i z and H & Ks bands, and determine Teff = 3565 +/- 82 K for the companion star, corresponding to an M1.5 dwarf. We use PHOENIX stellar atmosphere models to correct the dilution of the secondary eclipse depths of the hot Jupiter due to the presence of the M1.5 companion. We also improve the secondary eclipse photometry by accounting for the non-classical, flux-dependent nonlinearity of the WIRC IR detector in the H band. We measure planet-to-star flux ratios of 0.090 +/- 0.033%, 0.178 +/- 0.057%, 0.364 +/- 0.016%, and 0.438 +/- 0.020% in the H, Ks, 3.6 and 4.5 micron bands, respectively. We compare these with planetary atmospheric models, and find they prefer an atmosphere with a temperature inversion and inefficient heat redistribution. However, we also find that the data are equally well-described by a blackbody model for the planet with Tp = 2042 +/- 50 K. Finally, we measure a secondary eclipse timing offset of 0.3 +/- 1.3 min from the predicted mid-eclipse time, which constrains e = 0.0072 +0.0700/-0.0064 when combined with RV data and is more consistent with a circular orbit.