Extending the formalism of Datta, Bharadwaj & Choudhury (2007) for detecting ionized bubbles in redshifted 21 cm maps using a matched-filtering technique, we use different simulations to analyze the impact of HI fluctuations outside the bubble on the detectability of the bubble. In the first three kinds of simulations there is a spherical bubble of comoving radius R_b, the one that we are trying to detect, located at the center, and the neutral hydrogen (HI) outside the bubble traces the underlying dark matter distribution. We consider three different possible scenarios of reionization, i.e., (i) there is a single bubble (SB) in the field of view (FoV) and the hydrogen neutral fraction is constant outside this bubble (ii) patchy reionization with many small ionized bubbles in the FoV (PR1) and (iii) many spherical ionized bubbles of the same radius $R_b$ (PR2). The fourth kind of simulation uses more realistic maps based on semi-numeric modelling (SM) of ionized regions. We find that for both the SB and PR1 scenarios the fluctuating IGM restricts bubble detection to size R_b<= 6 Mpc and R_b<= 12 Mpc for the GMRT and the MWA respectively, however large be the integration time. These results are well explained by analytical predictions. Large uncertainty due to the HI fluctuations restricts bubble detection in the PR2 scenario for neutral fraction x_HI<0.6. The matched-filter technique works well even when the targeted ionized bubble is non-spherical due to surrounding bubbles and inhomogeneous recombination (SM). We find that determining the size and positions of the bubbles is not limited by the HI fluctuations in the SB and PR1 scenario but limited by the instruments angular resolution instead, and this can be done more precisely for larger bubble (abridged).