We present a detailed method to simulating sensor distortions using a photon and electron Monte Carlo method. We use three dimensional electrostatic simulations to parameterize the perturbed electric field profile for non-ideal sensor details. We follow the conversion of simulated photons, and the subsequent response of the converted electrons to the electric field pattern. These non-ideal sensor details can be implemented efficiently in a Monte Carlo approach. We demonstrate that the non-ideal sensor distortions have a variety of observable consequence including the modification of the astrometric pattern, the distortion of the electron diffusion size and shape, and the distortion of flats. We show analytic validation of the diffusion physics, reproduce two kinds of edge distortion, and show qualitative validation of field-free regions, lithography errors, and fringing. We also demonstrate that there are two related effects of doping variation having different observable consequences. We show that field distortions from accumulated electrons lead to intensity-dependent point-spread-functions and the sub-linear variance in flats. The method is implemented in the Photon Simulator (PhoSim) and the code is publically available.