The spectrum of terahertz (THz) emission in gases via ionizing two-color femtosecond pulses is analyzed by means of a semi-analytic model and finite-difference-time-domain simulations in 1D and 2D geometries. We show that produced THz signals interact with free electron trajectories and thus influence significantly further THz generation upon propagation, i.e., make the process inherently nonlocal. This self-action plays a key role in the observed strong spectral broadening of the generated THz field. Diffraction limits the achievable THz bandwidth by efficiently depleting the low frequency amplitudes in the propagating field.