Between 1997 and 2004 several observing runs were conducted mainly with the CTIO 0.9 m to image ICRF counterparts (mostly QSOs) in order to determine accurate optical positions. Contemporary to these deep CCD images the same fields were observed with the US Naval Observatory (USNO) astrograph in the same bandpass. They provide accurate positions on the Hipparcos/Tycho-2 system for stars in the 10 to 16 magnitude range used as reference stars for the deep CCD imaging data. Here we present final optical position results of 413 sources based on reference stars obtained by dedicated astrograph observations which were reduced following 2 different procedures. These optical positions are compared to radio VLBI positions. The current optical system is not perfectly aligned to the ICRF radio system with rigid body rotation angles of 3 to 5 mas (= 3 sigma level) found between them for all 3 axes. Furthermore, statistically, the optical minus radio position differences are found to exceed the total, combined, known errors in the observations. Systematic errors in the optical reference star positions as well as physical offsets between the centers of optical and radio emissions are both identified as likely causes. A detrimental, astrophysical, random noise (DARN) component is postulated to be on about the 10 mas level. If confirmed by future observations, this could severely limit the Gaia to ICRF reference frame alignment accuracy to an error of about 0.5 mas per coordinate axis with the current number of sources envisioned to provide the link. A list of 36 ICRF sources without the detection of an optical counterpart to a limiting magnitude of about R=22 is provided as well.