The mitigation of climate change requires a fundamental transition of the energy system. Affordability, reliability and the reduction of greenhouse gas emissions constitute central but often conflicting targets for this energy transition. Against this context, we reveal limitations and counter-intuitive results in the model-based optimization of energy systems, which are often applied for policy advice. When system costs are minimized in the presence of a CO2 cap, efficiency gains free a part of the CO2 cap, allowing cheap technologies to replace expensive low-emission technologies. Even more striking results are observed in a setup where emissions are minimized in the presence of a budget constraint. Increasing CO2 prices can oust clean, but expensive technologies out of the system, and eventually lead to higher emissions. These effects robustly occur in models of different scope and complexity. Hence, extreme care is necessary in the application of energy system optimization models to avoid misleading policy advice.