Notched components are commonly used in engineering structures, where stress concentration may easily lead to crack initiation and development. The main goal of this work is to develop a simple numerical method to predict the structural strength and crack-growth-path of U-notched specimens made of brittle materials. For this purpose, the Fragile Points Method (FPM), as previously proposed by the authors, has been augmented by an interface damage model at the interfaces of the FPM domains, to simulate crack initiation and development. The formulations of FPM are based on a discontinuous Galerkin weak form where point-based piece-wise-continuous polynomial test and trial functions are used instead of element-based basis functions. In this work, the numerical fluxes introduced across interior interfaces between subdomains are postulated as the tractions acting on the interface derived from an interface damage model. The interface damage is triggered when the numerical flux reaches the interface strength, and the process of crack-surface separation is governed by the fracture energy. In this way, arbitrary crack initiation and propagation can be naturally simulated without the need for knowing the fracture-patch before-hand. Additionally, a small penalty parameter is sufficient to enforce the weak-form continuity condition before damage initiation, without causing problems such as artificial compliance and numerical ill-conditioning. As validations, the proposed FPM method with the interface damage model is used to predict the structural strength and crack-development from U-notched structures made of brittle materials, which is useful but challenging in engineering structural design practices.