A N=4 supersymmetric matrix KP hierarchy is proposed and a wide class of its reductions which are characterized by a finite number of fields are described. This class includes the one-dimensional reduction of the two-dimensional N=(2|2) superconformal Toda lattice hierarchy possessing the N=4 supersymmetry -- the N=4 Toda chain hierarchy -- which may be relevant in the construction of supersymmetric matrix models. The Lax pair representations of the bosonic and fermionic flows, corresponding local and nonlocal Hamiltonians, finite and infinite discrete symmetries, the first two Hamiltonian structures and the recursion operator connecting all evolution equations and the Hamiltonian structures of the N=4 Toda chain hierarchy are constructed in explicit form. Its secondary reduction to the N=2 supersymmetric alpha=-2 KdV hierarchy is discussed.