The development of the Internet of Drones (IoD) becomes vital because of a proliferation of drone-based civilian or military applications. The IoD based technological revolution upgrades the current Internet environment into a more pervasive and ubiquitous world. IoD is capable of enhancing the state-of-the-art for drones while leveraging services from the existing cellular networks. Irrespective to a vast domain and range of applications, IoD is vulnerable to malicious attacks over open-air radio space. Due to increasing threats and attacks, there has been a lot of attention on deploying security measures for IoD networks. In this paper, critical threats and vulnerabilities of IoD are presented. Moreover, taxonomy is created to classify attacks based on the threats and vulnerabilities associated with the networking of drone and their incorporation in the existing cellular setups. In addition, this article summarizes the challenges and research directions to be followed for the security of IoD.