We report on Chandra X-ray observations of four candidate low-mass black hole (<10^6Msun) active galactic nuclei (AGNs) that have the estimated Eddington ratios among the lowest (~10^(-2)) found for this class. The aims are to validate the nature of their AGNs and to confirm the low Eddington ratios that are derived from the broad H_alpha line, and to explore this poorly studied regime in the AGN parameter space. Among them, two objects with the lowest significance of the broad lines are also observed with Multi-Mirror Telescope, and the high-quality optical spectra taken confirm them as Seyfert 1 AGNs and as having small black hole masses. X-ray emission is detected from the nuclei of two of the galaxies, which is variable on timescales of 10^3s, whereas no significant (or only marginal at best) detection is found for the remaining two. The X-ray luminosities are on the order of 10^(41) ergs/s or even lower, on the order of 10^(40) ergs/s for non-detections, which are among the lowest regimes ever probed for Seyfert galaxies. The low X-ray luminosities, compared to their black hole masses derived from H_alpha, confirm their low accretion rates assuming typical bolometric corrections. Our results hint at the existence of a possibly large population of under-luminous low-mass black holes in the local universe. An off-nucleus ultra-luminous X-ray source (ULX) in one of the dwarf galaxies is detected serendipitously, with a luminosity (6-9)x10^(39) ergs/s in 2-10 keV.