A bias-free source-independent quantum random number generator scheme based on the measurement of vacuum fluctuation is proposed to realize the effective elimination of system bias and common mode noise introduced by the local oscillator. Optimal parameter settings are derived to avoid the system recording two canonically conjugate quadratures simultaneously in each measurement. In particular, it provides a new approach to investigate the performance difference between measuring two quadratures of equal and unequal intensity. It is experimentally demonstrated that the system supports 4.2 Gbps bias-free source-independent random number generation, where its common mode rejection ratio reaches 61.17 dB. Furthermore, the scheme offers an all-optical method facilitating the integration of source-independent quantum random number generators into compact chips.