Detecting UAVs is becoming more crucial for various industries such as airports and nuclear power plants for improving surveillance and security measures. Exploiting radio frequency (RF) based drone control and communication enables a passive way of drone detection for a wide range of environments and even without favourable line of sight (LOS) conditions. In this paper, we evaluate RF based drone classification performance of various state-of-the-art (SoA) models on a new realistic drone RF dataset. With the help of a newly proposed residual Convolutional Neural Network (CNN) model, we show that the drone RF frequency signatures can be used for effective classification. The robustness of the classifier is evaluated in a multipath environment considering varying Doppler frequencies that may be introduced from a flying drone. We also show that the model achieves better generalization capabilities under different wireless channel and drone speed scenarios. Furthermore, the newly proposed models classification performance is evaluated on a simultaneous multi-drone scenario. The classifier achieves close to 99 % classification accuracy for signal-to-noise ratio (SNR) 0 dB and at -10 dB SNR it obtains 5 % better classification accuracy compared to the existing framework.