We study the Casimir interaction energy due to the vacuum fluctuations of the Electromagnetic (EM) field in the presence of two mirrors, described by $2+1$-dimensional, generally nonlocal actions, which may contain both parity-conserving and parity-breaking terms. We compare the results with the ones corresponding to Chern-Simons boundary conditions, and evaluate the interaction energy for several particular situations.