A single nitrogen-vacancy (NV) center in diamond is a prime candidate for a solid-state quantum magnetometer capable of detecting single nuclear spins with prospective application to nuclear magnetic resonance (NMR) at the nanoscale. Nonetheless, an NV magnetometer is still less accessible to many chemists and biologists, as its experimental setup and operational principle are starkly different from those of conventional NMR. Here, we design, construct, and operate a compact tabletop-sized system for quantum sensing with a single NV center, built primarily from commercially available optical components and electronics. We show that our setup can implement state-of-the-art quantum sensing protocols that enable the detection of single $^{13}$C nuclear spins in diamond and the characterization of their interaction parameters, as well as the detection of a small ensemble of proton nuclear spins on the diamond surface. This article providing extensive discussions on the details of the setup and the experimental procedures, our system will be reproducible by those who have not worked on the NV centers previously.