In tunnel junctions with ferroelectric barriers, switching the polarization direction modifies the electrostatic potential profile and the associated average tunnel barrier height. This results in strong changes of the tunnel transmission and associated resistance. The information readout in ferroelectric tunnel junctions (FTJs) is thus resistive and non-destructive, which is an advantage compared to the case of conventional ferroelectric memories (FeRAMs). Initially, endurance limitation (i.e. fatigue) was the main factor hampering the industrialization of FeRAMs. Systematic investigations of switching dynamics for various ferroelectric and electrode materials have resolved this issue, with endurance now reaching $10^{14}$ cycles. Here we investigate data retention and endurance in fully patterned submicron Co/BiFeO$_3$/Ca$_{0.96}$Ce$_{0.04}$MnO$_3$ FTJs. We report good reproducibility with high resistance contrasts and extend the maximum reported endurance of FTJs by three orders of magnitude ($4times10^6$ cycles). Our results indicate that here fatigue is not limited by a decrease of the polarization or an increase of the leakage but rather by domain wall pinning. We propose directions to access extreme and intermediate resistance states more reliably and further strengthen the potential of FTJs for non-volatile memory applications.