Despite mounting evidence that materials imperfections are a major obstacle to practical applications of superconducting qubits, connections between microscopic material properties and qubit coherence are poorly understood. Here, we perform measurements of transmon qubit relaxation times $T_1$ in parallel with spectroscopy and microscopy of the thin polycrystalline niobium films used in qubit fabrication. By comparing results for films deposited using three techniques, we reveal correlations between $T_1$ and grain size, enhanced oxygen diffusion along grain boundaries, and the concentration of suboxides near the surface. Physical mechanisms connect these microscopic properties to residual surface resistance and $T_1$ through losses arising from the grain boundaries and from defects in the suboxides. Further, experiments show that the residual resistance ratio can be used as a figure of merit for qubit lifetime. This comprehensive approach to understanding qubit decoherence charts a pathway for materials-driven improvements of superconducting qubit performance.