We use direct $N$-body calculations to study the evolution of the unusually extended outer halo globular cluster Palomar 4 (Pal~4) over its entire lifetime in order to reproduce its observed mass, half-light radius, velocity dispersion and mass function slope at different radii. We find that models evolving on circular orbits, and starting from a non-mass segregated, canonical initial mass function (IMF) can reproduce neither Pal 4s overall mass function slope nor the observed amount of mass segregation. Including either primordial mass segregation or initially flattened IMFs does not reproduce the observed amount of mass segregation and mass function flattening simultaneously. Unresolved binaries cannot reconcile this discrepancy either. We find that only models with both a flattened IMF and primordial segregation are able to fit the observations. The initial (i.e. after gas expulsion) mass and half-mass radius of Pal~4 in this case are about 57000 M${odot}$ and 10 pc, respectively. This configuration is more extended than most globular clusters we observe, showing that the conditions under which Pal~4 formed must have been significantly different from that of the majority of globular clusters. We discuss possible scenarios for such an unusual configuration of Pal~4 in its early years.