We have performed a detailed study of thermal annealing of the moment configuration in artificial spin ice. Permalloy (Ni$_{80}$Fe$_{20}$) artificial spin ice samples were examined in the prototypical square ice geometry, studying annealing as a function of island thickness, island shape, and annealing temperature and duration. We also measured the Curie temperature as a function of film thickness, finding that thickness has a strong effect on the Curie temperature in regimes of relevance to many studies of the dynamics of artificial spin ice systems. Increasing the interaction energy between island moments and reducing the energy barrier to flipping the island moments allows the system to more closely approach the collective low energy state of the moments upon annealing, suggesting new channels for understanding the thermalization processes in these important model systems.