We measure the near-infrared properties of 42 X-ray detected sources from the Chandra Deep Fields North and South, the majority of which lie within the NICMOS Hubble Deep Field North and Ultra Deep Field. We detect all 42 Chandra sources with NICMOS, with 95% brighter than H = 24.5. We find that X-ray sources are most often in the brightest and most massive galaxies. Neither the X-ray fluxes nor hardness ratios of the sample show any correlation with near-infrared flux, color or morphology. This lack of correlation indicates there is little connection between the two emission mechanisms and is consistent with the near-infrared emission being dominated by starlight rather than a Seyfert non-stellar continuum. Near-infrared X-ray sources make up roughly half of all extremely red (J-H > 1.4) objects brighter than H > 24.5. These red X-ray sources have a range of hardness ratios similar to the rest of the sample, decreasing the likelihood of dust-obscured AGN activity as the sole explanation for their red color. Using a combination of spectroscopic and photometric redshifts, we find the red J-H objects are at high redshifts (z > 1.5), which we propose as the primary explanation for their extreme J-H color. Measurement of rest-wavelength absolute B magnitudes shows that X-ray sources are the brightest optical objects at all redshifts, which explains their dominance of the bright end of the red J-H population.