The data from ground based gravitational-wave detectors such as Advanced LIGO and Virgo must be calibrated to convert the digital output of photodetectors into a relative displacement of the test masses in the detectors, producing the quantity of interest for inference of astrophysical gravitational wave sources. Both statistical uncertainties and systematic errors are associated with the calibration process, which would in turn affect the analysis of detected sources, if not accounted for. Currently, source characterization algorithms either entirely neglect the possibility of calibration uncertainties or account for them in a way that does not use knowledge of the calibration process itself. We present physiCal, a new approach to account for calibration errors during the source characterization step, which directly uses all the information available about the instrument calibration process. Rather than modeling the overall detectors response function, we consider the individual components that contribute to the response. We implement this method and apply it to the compact binaries detected by LIGO and Virgo during the second observation run, as well as to simulated binary neutron stars for which the sky position and distance are known exactly. We find that the physiCal model performs as well as the method currently used within the LIGO-Virgo collaboration, but additionally it enables improving the measurement of specific components of the instrument control through astrophysical calibration.