We introduce a second quantization scheme based on quasinormal modes, which are the dissipative modes of leaky optical cavities and plasmonic resonators with complex eigenfrequencies. The theory enables the construction of multi-plasmon/photon Fock states for arbitrary three-dimensional dissipative resonators and gives a solid understanding to the limits of phenomenological dissipative Jaynes-Cummings models. In the general case, we show how different quasinormal modes interfere through an off-diagonal mode coupling and demonstrate how these results affect cavity-modified spontaneous emission. To illustrate the practical application of the theory, we show examples using a gold nanorod dimer and a hybrid dielectric-metal cavity structure.