Bound states in the continuum (BICs), a concept from quantum mechanics, are ubiquitous physical phenomena where waves will be completely locked inside physical systems without energy leaky. Such a physical phenomenon in optics will provide a platform for optical mode confinement to strengthen local field enhancement in nonlinear optics. Here we utilize an optical system consisting of asymmetric nanogratings and waveguide of thin film lithium niobate (LiNbO3) material to enhance second harmonic response near BICs. By breaking the symmetry of grating periodicity, we realize strong local field confined inside waveguide up to 25 times normalized to incident field (with dissymmetric factor of 0.2), allowing strong light-matter interaction in nonlinear material. From the numerical simulation, we theoretically demonstrate that such an optical system can greatly enhance second harmonic intensity enhancement of about 104 compared with undersigned LiNbO3 film and conversion efficiency reaching 1.53e-5 for dissymmetric factor=0.2 under illumination of 1.33 GW/(suqare cm). Surprisingly, we can predict that a giant enhancement of second harmonic conversion efficiency will exceed 8.13e-5 for dissymmetric factor=0.1 when the optical system is extremely close to BICs. We believe that such an optical system to trap local field inside is also accessible to promote the application of thin film lithium niobate in the field of integrated nonlinear optics.