We study singularity structure of Yang-Mills flow in dimensions $n geq 4$. First we obtain a description of the singular set in terms of concentration for a localized entropy quantity, which leads to an estimate of its Hausdorff dimension. We develop a theory of tangent measures for the flow, which leads to a stratification of the singular set. By a refined blowup analysis we obtain Yang-Mills connections or solitons as blowup limits at any point in the singular set.