In this article we present the case of HD 41004 AB, a system composed of a K0V star and a 3.7-magnitude fainter M-dwarf companion separated by only 0.5 arcsec. An analysis of CORALIE radial-velocity measurements has revealed a variation with an amplitude of about 50m/s and a periodicity of 1.3days. This radial-velocity signal is consistent with the expected variation induced by the presence a very low mass giant planetary companion to HD 41004 A, whose light dominates the spectra. The radial-velocity measurements were then complemented with a photometric campaign and with the analysis of the bisector of the CORALIE Cross-Correlation Function (CCF). While the former revealed no significant variations within the observational precision of 0.003-0.004 mag (except for an observed flare event), the bisector analysis showed that the line profiles are varying in phase with the radial-velocity. This latter result, complemented with a series of simulations, has shown that we can explain the observations by considering that HD 41004 B has a brown-dwarf companion orbiting with the observed 1.3-day period. If confirmed, this detection represents the first discovery of a brown dwarf in a very short period (1.3-day) orbit around an M dwarf. Finally, this case should be taken as a serious warning about the importance of analyzing the bisector when looking for planets using radial-velocity techniques.