The causal limit usually considered in cosmology is the particle horizon, delimiting the possibilities of causal connection in the expanding universe. However it is not a realistic indicator of the effective local limits of important interactions in spacetime. We consider here the matter horizon for the Solar System, that is,the comoving region which has contributed matter to our local physical environment. This lies inside the effective domain of dependence, which (assuming the universe is dominated by dark matter along with baryonic matter and vacuum-energy-like dark energy) consists of those regions that have had a significant active physical influence on this environment through effects such as matter accretion and acoustic waves. It is not determined by the velocity of light c, but by the flow of matter perturbations along their world lines and associated gravitational effects. We emphasize how small a region the perturbations which became our Galaxy occupied, relative to the observable universe -- even relative to the smallest-scale perturbations detectable in the cosmic microwave background radiation. Finally, looking to the future of our cosmic domain, we suggest simple dynamical criteria for determining the present domain of influence and the future matter horizon. The former is the radial distance at which our local region is just now separating from the cosmic expansion. The latter represents the limits of growth of the matter horizon in the far future.