Quantum-dot based parity-to-charge conversion is a promising method for reading out quantum information encoded nonlocally into pairs of Majorana zero modes. To obtain a sizable parity-to-charge visibility, it is crucial to tune the relative phase of the tunnel couplings between the dot and the Majorana modes appropriately. However, in the presence of multiple quasi-degenerate dot orbitals, it is in general not experimentally feasible to tune all couplings individually. This paper shows that such configurations could make it difficult to avoid a destructive multi-orbital interference effect that substantially reduces the read-out visibility. We analyze this effect using a Lindblad quantum master equation. This exposes how the experimentally relevant system parameters enhance or suppress the visibility when strong charging energy, measurement dissipation and, most importantly, multi-orbital interference is accounted for. In particular, we find that an intermediate-time readout could mitigate some of the interference-related visibility reductions affecting the stationary limit.