In this paper we address many of the fundamental open questions regarding the glassy behavior of the magnetic/electronic phase segregated state in rare earth perovskites. In particular, magnetic relaxation experiments support that the collective effects (memory, ageing, etc.) are due to interparticle interactions, rather than the double-exchange vs. superexchange competition. A careful study of the non-linear susceptibility in the critical region is performed, and the critical exponents contrasted with those of conventional spin-glasses and concentrated quenched ferrofluids. The phase segregated state constitutes a sort of self-generated assembly of magnetic particles in which magnetic interaction introduces collectivity among the clusters.