We present measurements of the thermal expansion coefficient $alpha$ of polycrystalline RFeAsO (R = La,Ce,Pr,Sm,Gd). Anomalies at the magnetic ordering transitions indicate a significant magneto-elastic coupling and a negative pressure dependence of $T_{rm N}$ . The structural transitions are associated by large anomalies in $alpha$. Rare earth magnetic ordering in CeFeAsO, PrFeAsO, and SmFeAsO yields large positive anomalies at low temperatures.