This paper describes the optimisation theory on which VPFIT, a non-linear least-squares program for modelling absorption spectra, is based. Particular attention is paid to precision. Voigt function derivatives have previously been calculated using numerical finite difference approximations. We show how these can instead be computed analytically using Taylor series expansions and look-up tables. We introduce a new optimisation method for an efficient descent path to the best-fit, combining the principles used in both the Gauss-Newton and Levenberg-Marquardt algorithms. A simple practical fix for ill-conditioning is described, a common problem when modelling quasar absorption systems. We also summarise how unbiased modelling depends on using an appropriate information criterion to guard against over- or under-fitting. The methods and the new implementations introduced in this paper are aimed at optimal usage of future data from facilities such as ESPRESSO/VLT and HIRES/ELT, particularly for the most demanding applications such as searches for spacetime variations in fundamental constants and attempts to detect cosmological redshift drift.