Understanding the dominant brown dwarf and giant planet formation processes, and finding out whether these processes rely on completely different mechanisms or share common channels represents one of the major challenges of astronomy and remains the subject of heated debates. It is the aim of this review to summarize the latest developments in this field and to address the issue of origin by confronting different brown dwarf and giant planet formation scenarios to presently available observational constraints. As examined in the review, if objects are classified as Brown Dwarfs or Giant Planets on the basis of their formation mechanism, it has now become clear that their mass domains overlap and that there is no mass limit between these two distinct populations. Furthermore, while there is increasing observational evidence for the existence of non-deuterium burning brown dwarfs, some giant planets, characterized by a significantly metal enriched composition, might be massive enough to ignite deuterium burning in their core. Deuterium burning (or lack of) thus plays no role in either brown dwarf or giant planet formation. Consequently, we argue that the IAU definition to distinguish these two populations has no physical justification and brings scientific confusion. In contrast, brown dwarfs and giant planets might bear some imprints of their formation mechanism, notably in their mean density and in the physical properties of their atmosphere. Future direct imaging surveys will undoubtedly provide crucial information and perhaps provide some clear observational diagnostics to unambiguously distinguish these different astrophysical objects.