We consider the topological category of $h$-cobordisms between manifolds with boundary and compare its homotopy type with the standard $h$-cobordism space of a compact smooth manifold.