We derive a family of matrix models which encode solutions to the Seiberg-Witten theory in 4 and 5 dimensions. Partition functions of these matrix models are equal to the corresponding Nekrasov partition functions, and their spectral curves are the Seiberg-Witten curves of the corresponding theories. In consequence of the geometric engineering, the 5-dimensional case provides a novel matrix model formulation of the topological string theory on a wide class of non-compact toric Calabi-Yau manifolds. This approach also unifies and generalizes other matrix models, such as the Eguchi-Yang matrix model, matrix models for bundles over $P^1$, and Chern-Simons matrix models for lens spaces, which arise as various limits of our general result.