Biased-noise qubits are a promising candidate for realizing hardware efficient fault-tolerant quantum computing. One promising biased-noise qubit is the Kerr cat qubit, which has recently been demonstrated experimentally. Despite various unique advantages of Kerr cat qubits, we explain how the noise bias of Kerr cat qubits is severely limited by heating-induced leakage in their current implementations. Then, we show that by adding frequency-selective single-photon loss to Kerr cat qubits we can counteract the leakage and thus recover much of their noise bias. We refer to such Kerr cat qubits combined with frequency-selective single-photon loss as colored Kerr cat qubits as they are protected by a colored dissipation. In particular, we show how a suitably engineered lossy environment can suppress the leakage and bit-flip errors of a Kerr cat qubit while not introducing any additional phase-flip errors. Since our scheme only requires single-photon loss, it can be readily implemented by using passive and linear elements. Moreover, our frequency-selectivity technique can be generally applied to energy-gap protected qubits whose computational basis states are given by near degenerate ground states of a Hamiltonian with a non-zero energy gap between the ground and excited state manifolds.