Animals form groups for many reasons but there are costs and benefit associated with group formation. One of the benefits is collective memory. In groups on the move, social interactions play a crucial role in the cohesion and the ability to make consensus decisions. When migrating from spawning to feeding areas fish schools need to retain a collective memory of the destination site over thousand of kilometers and changes in group formation or individual preference can produce sudden changes in migration pathways. We propose a modelling framework, based on stochastic adaptive networks, that can reproduce this collective behaviour. We assume that three factors control group formation and school migration behaviour: the intensity of social interaction, the relative number of informed individuals and the preference that each individual has for the particular migration area. We treat these factors independently and relate the individuals preferences to the experience and memory for certain migration sites. We demonstrate that removal of knowledgable individuals or alteration of individual preference can produce rapid changes in group formation and collective behavior. For example, intensive fishing targeting the migratory species and also their preferred prey can reduce both terms to a point at which migration to the destination sites is suddenly stopped. The conceptual approaches represented by our modelling framework may therefore be able to explain large-scale changes in fish migration and spatial distribution.