The cutoff dependence of the Casimir energy and stress is studied using the Greens function method for a system that is piecewise-smoothly inhomogeneous along one dimension. The asymptotic cylinder kernel expansions of the energy and stress are obtained, with some extra cutoff terms that are induced by the inhomogeneity. Introducing interfaces to the system one by one shows how those cutoff terms emerge and illuminates their physical interpretations. Based on that, we propose a subtraction scheme to address the problem of the remaining cutoff dependence in the Casimir stress in an inhomogeneous medium, and show that the nontouching Casimir force between two separated bodies is cutoff independent. The cancellation of the electric and magnetic contributions to the surface divergence near a perfectly conducting wall is found to be incomplete in the case of inhomogeneity.