We study the phenomenon of Hilbert space fragmentation in isolated Hamiltonian and Floquet quantum systems using the language of commutant algebras, the algebra of all operators that commute with each term of the Hamiltonian or each gate of the circuit. We provide a precise definition of Hilbert space fragmentation in this formalism as the case where the dimension of the commutant algebra grows exponentially with the system size. Fragmentation can hence be distinguished from systems with conventional symmetries such as $U(1)$ or $SU(2)$, where the dimension of the commutant algebra grows polynomially with the system size. Further, the commutant algebra language also helps distinguish between classical and quantum Hilbert space fragmentation, where the former refers to fragmentation in the product state basis. We explicitly construct the commutant algebra in several systems exhibiting classical fragmentation, including the $t-J_z$ model and the spin-1 dipole-conserving model, and we illustrate the connection to previously-studied Statistically Localized Integrals of Motion (SLIOMs). We also revisit the Temperley-Lieb spin chains, including the spin-1 biquadratic chain widely studied in the literature, and show that they exhibit quantum Hilbert space fragmentation. Finally, we study the contribution of the full commutant algebra to the Mazur bounds in various cases. In fragmented systems, we use expressions for the commutant to analytically obtain new or improved Mazur bounds for autocorrelation functions of local operators that agree with previous numerical results. In addition, we are able to rigorously show the localization of the on-site spin operator in the spin-1 dipole-conserving model.