We consider a class of multi-matrix models with an action which is O(D) invariant, where D is the number of NxN Hermitian matrices X_mu, mu=1,...,D. The action is a function of all the elementary symmetric functions of the matrix $T_{mu u}=Tr(X_mu X_ u)/N$. We address the issue whether the O(D) symmetry is spontaneously broken when the size N of the matrices goes to infinity. The phase diagram in the space of the parameters of the model reveals the existence of a critical boundary where the O(D) symmetry is maximally broken.