We seek to identify the primary agents causing Forbush decreases (FDs) observed at the Earth in high rigidity cosmic rays. In particular, we ask if such FDs are caused mainly by coronal mass ejections (CMEs) from the Sun that are directed towards the Earth, or by their associated shocks. We use the muon data at cutoff rigidities ranging from 14 to 24 GV from the GRAPES-3 tracking muon telescope to identify FD events. We select those FD events that have a reasonably clean profile, and can be reasonably well associated with an Earth-directed CME and its associated shock. We employ two models: one that considers the CME as the sole cause of the FD (the CME-only model) and one that considers the shock as the only agent causing the FD (the shock-only model). We use an extensive set of observationally determined parameters for both these models. The only free parameter in these models is the level of MHD turbulence in the sheath region, which mediates cosmic ray diffusion (into the CME, for the CME-only model and across the shock sheath, for the shock-only model). We find that good fits to the GRAPES-3 multi-rigidity data using the CME-only model require turbulence levels in the CME sheath region that are only slightly higher than those estimated for the quiet solar wind. On the other hand, reasonable model fits with the shock-only model require turbulence levels in the sheath region that are an order of magnitude higher than those in the quiet solar wind. This observation naturally leads to the conclusion that the Earth-directed CMEs are the primary contributors to FDs observed in high rigidity cosmic rays.