We present a ghost handwritten digit recognition method for the unknown handwritten digits based on ghost imaging (GI) with deep neural network, where a few detection signals from the bucket detector, generated by the Cosine Transform speckle, are used as the characteristic information and the input of the designed deep neural network (DNN), and the classification is designed as the output of the DNN. The results show that the proposed scheme has a higher recognition accuracy (as high as 98.14% for the simulations, and 92.9% for the experiments ) with a smaller sampling ratio (say 12.76%). With the increase of the sampling ratio, the recognition accuracy is enhanced greatly. Compared with the traditional recognition scheme using the same DNN structure, the proposed scheme has a little better performance with a lower complexity and non-locality property. The proposed scheme provides a promising way for remote sensing.