We search for correlations between positions of extragalactic objects and arrival directions of Ultra-High Energy Cosmic Rays (UHECRs) with primary energy $E ge 40$ EeV as observed by the surface detector array of the Telescope Array (TA) experiment during the first 40 months of operation. We examined several public astronomical object catalogs, including the Veron-Cetty and Veron catalog of active galactic nuclei. We counted the number of TA events correlated with objects in each catalog as a function of three parameters: the maximum angular separation between a TA event and an object, the minimum energy of the events, and the maximum redshift of the objects. We determine combinations of these parameters which maximize the correlations, and calculate the chance probabilities of having the same levels of correlations from an isotropic distribution of UHECR arrival directions. No statistically significant correlations are found when penalties for scanning over the above parameters and for searching in several catalogs are taken into account.