In recent years, there have been significant advances in multi-messenger astronomy due to the discovery of the first, and so far only confirmed, gravitational wave event with a simultaneous electromagnetic (EM) counterpart, as well as improvements in numerical simulations, gravitational wave (GW) detectors, and transient astronomy. This has led to the exciting possibility of performing joint analyses of the GW and EM data, providing additional constraints on fundamental properties of the binary progenitor and merger remnant. Here, we present a new Bayesian framework that allows inference of these properties, while taking into account the systematic modeling uncertainties that arise when mapping from GW binary progenitor properties to photometric light curves. We extend the relative binning method presented in Zackay et al. (2018) to include extrinsic GW parameters for fast analysis of the GW signal. The focus of our EM framework is on light curves arising from r-process nucleosynthesis in the ejected material during and after merger, the so called kilonova, and particularly on black hole - neutron star systems. As a case study, we examine the recent detection of GW190425, where the primary object is consistent with being either a black hole (BH) or a neutron star (NS). We show quantitatively how improved mapping between binary progenitor and outflow properties, and/or an increase in EM data quantity and quality are required in order to break degeneracies in the fundamental source parameters.