Now that LIGO has revealed the existence of a large number of binary black holes, identifying their origin becomes an important challenge. They might originate in more isolated regions of the galaxy or alternatively they might reside in dense environments such as galactic centers or globular clusters. In the latter case, their center of mass motion as well as their orbital parameters should lead to observable changes in the waveforms, which would reflect their gravitational interactions with the surrounding matter. This would be reflected in the gravitational wave signal by a net phase change or even a time-dependent Doppler shift. We show that this time-dependence might be observable in future space gravitational wave detectors such as LISA which could provide direct information about the black hole binary environments and otherwise invisible ambient mass.