Geometric Distortion Calibration with Photo-lithographic Pinhole Masks for High-Precision Astrometry

Adaptive optics (AO) systems deliver high-resolution images that may be ideal for precisely measuring positions of stars (i.e. astrometry) if the system has stable and well-calibrated geometric optical distortions. A calibration unit, equipped with back-illuminated pinhole mask, can be utilized to measure instrumental optical distortions. AO systems on the largest ground-based telescopes, such as the W. M. Keck Observatory and the Thirty Meter Telescope require pinhole positions known to 20 nm to achieve an astrometric precision of 0.001 of a resolution element. We characterize a photo-lithographic pinhole mask and explore the systematic errors that result from different experimental setups. We characterized the nonlinear geometric distortion of a simple imaging system using the mask; and we measured 857 nm RMS of optical distortion with a final residual of 39 nm (equivalent to 20 {mu}as for TMT). We use a sixth order bivariate Legendre polynomial to model the optical distortion and allow the reference positions of the individual pinholes to vary. The nonlinear deviations in the pinhole pattern with respect to the manufacturing design of a square pattern are 47.2 nm +/- 4.5 nm (random) +/- 10.8 nm (systematic) over an area of 1788 mm$^2$. These deviations reflect the additional error induced when assuming the pinhole mask is manufactured perfectly square. We also find that ordered mask distortions are significantly more difficult to characterize than random mask distortions as the ordered distortions can alias into optical camera distortion. Future design simulations for astrometric calibration units should include ordered mask distortions. We conclude that photo-lithographic pinhole masks are >10 times better than the pinhole masks deployed in first generation AO systems and are sufficient to meet the distortion calibration requirements for the upcoming thirty meter class telescopes.