In the context of a warped extra-dimension with Standard Model fields in the bulk, we obtain the general flavor structure of the Higgs couplings to fermions. These couplings will be generically misaligned with respect to the fermion mass matrix, producing large and potentially dangerous flavor changing neutral currents (FCNCs). As recently pointed out in [arXiv:0906.1542], a similar effect is expected from the point of view of a composite Higgs sector, which corresponds to a 4D theory dual to the 5D setup by the AdS-CFT correspondence. We also point out that the effect is independent of the geographical nature of the Higgs (bulk or brane localized), and specifically that it does not go away as the Higgs is pushed towards the IR boundary. The FCNCs mediated by a light enough Higgs (specially their contribution to $epsilon_K$) could become of comparable size as the ones coming from the exchange of Kaluza-Klein (KK) gluons. Moreover, both sources of flavor violation are complementary since they have inverse dependence on the 5D Yukawa couplings, such that we cannot decouple the flavor violation effects by increasing or decreasing these couplings. We also find that for KK scales of a few TeV, the Higgs couplings to third generation fermions could experience suppressions of up to 40% while the rest of diagonal couplings would suffer much milder corrections. Potential LHC signatures like the Higgs flavor violating decays $htomutau$ or $hto tc$, or the exotic top decay channel $tto c h$, are finally addressed.