[Abridged] Protoplanetary disks have been studied extensively, both physically and chemically, to understand the environment in which planets form. However, the first steps of planet formation are likely to occur already when the protostar and disk are still embedded in their natal envelope. The initial conditions for planet formation may thus be provided by these young embedded disks, of which the physical and chemical structure is poorly characterized. We aim to constrain the midplane temperature structure, one of the critical unknowns, of the embedded disk around L1527. In particular, we set out to determine whether there is an extended cold outer region where CO is frozen out, as is the case for Class II disks. We use archival ALMA data to directly observe the midplane of the near edge-on L1527 disk. Optically thick $^{13}$CO ($J=2-1$) and C$^{18}$O ($J=2-1$) emission is observed throughout the disk and inner envelope, while N$_2$D$^+ (J=3-2$), which can only be abundant when CO is frozen out, is not detected. Both CO isotopologues have brightness temperatures $gtrsim$ 25 K along the midplane. Disk and envelope emission can be disentangled kinematically, because the largest velocities are reached in the disk. A power law radial temperature profile constructed using the highest midplane temperature at these velocities suggest that the temperature is above 20 K out to at least 75 AU, and possibly throughout the entire 125 AU disk. Radiative transfer models show that a model without CO freeze-out in the disk matches the C$^{18}$O observations better than a model with the CO snowline at $sim$70 AU. In addition, there is no evidence for a large (order of magnitude) depletion of CO. The disk around L1527 is likely to be warm enough to have CO present in the gas phase throughout the disk, suggesting that young embedded disks can indeed be warmer than the more evolved Class II disks.