The thermal conductivity $kappa$ of the cuprate superconductor La$_{1.6-x}$Nd$_{0.4}$Sr$_x$CuO$_4$ was measured down to 50 mK in seven crystals with doping from $p=0.12$ to $p=0.24$, both in the superconducting state and in the magnetic field-induced normal state. We obtain the electronic residual linear term $kappa_0/T$ as $T to 0$ across the pseudogap critical point $p^{star}= 0.23$. In the normal state, we observe an abrupt drop in $kappa_0/T$ upon crossing below $p^{star}$, consistent with a drop in carrier density $n$ from $1 + p$ to $p$, the signature of the pseudogap phase inferred from the Hall coefficient. A similar drop in $kappa_0/T$ is observed at $H=0$, showing that the pseudogap critical point and its signatures are unaffected by the magnetic field. In the normal state, the Wiedemann-Franz law, $kappa_0/T=L_0/rho(0)$, is obeyed at all dopings, including at the critical point where the electrical resistivity $rho(T)$ is $T$-linear down to $T to 0$. We conclude that the non-superconducting ground state of the pseudogap phase at $T=0$ is a metal whose fermionic excitations carry heat and charge as conventional electrons do.