We report on devices based on graphene charge detectors (CDs) capacitively coupled to graphene and carbon nanotube quantum dots (QDs). We focus on back action effects of the CD on the probed QD. A strong influence of the bias voltage applied to the CD on the current through the QD is observed. Depending on the charge state of the QD the current through the QD can either strongly increase or completely reverse as a response to the applied voltage on the CD. To describe the observed behavior we employ two simple models based on single electron transport in QDs with asymmetrically broadened energy distributions of the source and the drain leads. The models successfully explain the back action effects. The extracted distribution broadening shows a linear dependency on the bias voltage applied to the CD. We discuss possible mechanisms mediating the energy transfer between the CD and QD and give an explanation for the origin of the observed asymmetry.