We construct a class of matrix models, where supersymmetry (SUSY) is spontaneously broken at the matrix size $N$ infinite. The models are obtained by dimensional reduction of matrix-valued SUSY quantum mechanics. The potential of the models is slowly varying, and the large-$N$ limit is taken with the slowly varying limit. First, we explain our formalism, introducing an external field to detect spontaneous SUSY breaking, analogously to ordinary (bosonic) symmetry breaking. It is observed that SUSY is possibly broken even in systems in less than one-dimension, for example, discretized quantum mechanics with a finite number of discretized time steps. Then, we consider spontaneous SUSY breaking in the SUSY matrix models with slowly varying potential, where the external field is turned off after the large-$N$ and slowly varying limit, analogously to the thermodynamic limit in statistical systems. On the other hand, without taking the slowly varying limit, in the SUSY matrix model with a double-well potential whose SUSY is broken due to instantons for finite $N$, a number of supersymmetric behavior is explicitly seen at large $N$. It convinces us that the instanton effect disappears and the SUSY gets restored in the large-$N$ limit.