We present a semi-analytic and asymptotically exact solution to the problem of phonon-induced decoherence in a quantum dot-microcavity system. Particular emphasis is placed on the linear polarization and optical absorption, but the approach presented herein may be straightforwardly adapted to address any elements of the exciton-cavity density matrix. At its core, the approach combines Trotters decomposition theorem with the linked cluster expansion. The effects of the exciton-cavity and exciton-phonon couplings are taken into account on equal footing, thereby providing access to regimes of comparable polaron and polariton timescales. We show that the optical decoherence is realized by real phonon-assisted transitions between different polariton states of the quantum dot-cavity system, and that the polariton line broadening is well-described by Fermis golden rule in the polariton frame. We also provide purely analytic approximations which accurately describe the system dynamics in the limit of longer polariton timescales.