Graphene is a very attractive material for broadband photodetection in hyperspectral imaging and sensing systems. However, its potential use has been hindered by tradeoffs between the responsivity, bandwidth, and operation speed of existing graphene photodetectors. Here, we present engineered photoconductive nanostructures based on gold-patched graphene nanoribbons, which enable simultaneous broadband and ultrafast photodetection with high responsivity. These nanostructures merge the advantages of broadband optical absorption, ultrafast photocarrier transport, and carrier multiplication in graphene nanoribbons with the ultrafast transport of photocarriers to the gold patches before recombination. Through this approach, high-responsivity operation is achieved without the use of bandwidth- and speed-limiting quantum dots, defect states, or tunneling barriers. We demonstrate high-responsivity photodetection from the visible to the infrared regime (0.6 A/W at 0.8 {mu}m and 11.5 A/W at 20 {mu}m) with operation speeds exceeding 50 GHz. Our results demonstrate an improvement of the response times by more than seven orders of magnitude and an increase in bandwidths of one order of magnitude compared to those of higher-responsivity graphene photodetectors based on quantum dots and tunneling barriers.