One of the pronounced characteristics of gravity, distinct from other interactions, is that there are no local observables which are independent of the choice of the spacetime coordinates. This property acquires crucial importance in the quantum domain in that the structure of the Hilbert space pertinent to different observers can be drastically different. Such intriguing phenomena as the Hawking radiation and the Unruh effect are all rooted in this feature. As in these examples, the quantum effect due to such observer-dependence is most conspicuous in the presence of an event horizon and there are still many questions to be clarified in such a situation. In this paper, we perform a comprehensive and explicit study of the observer dependence of the quantum Hilbert space of a massless scalar field in the vicinity of the horizon of the Schwarzschild black holes in four dimensions, both in the eternal (two-sided) case and in the physical (one-sided) case created by collapsing matter. Specifically, we compare and relate the Hilbert spaces of the three types of observers, namely (i) the freely falling observer, (ii) the observer who stays at a fixed proper distance outside of the horizon and (iii) the natural observer inside of the horizon analytically continued from outside. The concrete results we obtain have a number of important implications on black hole complementarity pertinent to the quantum equivalence principle and the related firewall phenomenon, on the number of degrees of freedom seen by each type of observer, and on the thermal-type spectrum of particles realized in a pure state.