In this paper we ground the asymmetry of causal relations in the internal physical states of a special kind of open dissipative physical system, a causal agent. A causal agent is an autonomous physical system, maintained far from equilibrium by a low entropy source of energy, with accurate sensors and actuators. It has a memory to record sensor measurements and actuator operations. It contains a learning system that can access the sensor and actuator records to learn and represent the causal relations. We claim that causal relations are relations between the internal sensor and actuator records and the causal concept inherent in these correlations is then inscribed in the physical dynamics of the internal learning machine. The existence of contingent internal memory states means each causal agent is in a different physical state. We argue that it is in this sense that causal relations are perspectival. From the outside, averaging over internal states, the causal agents are identical thermodynamic systems.