Reliably identifying active galactic nuclei (AGNs) in dwarf galaxies is key to understanding black hole demographics at low masses and constraining models for black hole seed formation. Here we present Chandra X-ray Observatory observations of eleven dwarf galaxies that were chosen as AGN candidates using Wide-field Infrared Survey Explorer (WISE) mid-infrared (mid-IR) color-color selection. Hubble Space Telescope images are also presented for ten of the galaxies. Based on Sloan Digital Sky Survey spectroscopy, six galaxies in our sample have optical evidence for hosting AGNs and five are classified as star-forming. We detect X-ray point sources with luminosities above that expected from X-ray binaries in the nuclei of five of the six galaxies with optical evidence of AGNs. However, the X-ray emission from these AGNs is generally much lower than expected based on AGN scaling relations with infrared and optical tracers. We do not find compelling evidence for AGNs in the five optically-selected star-forming galaxies despite having red mid-IR colors. Only two are detected in X-rays and their properties are consistent with stellar-mass X-ray binaries. Based on this multiwavelength study, we conclude that two-color mid-IR AGN diagnostics at the resolution of WISE cannot be used to reliably select AGNs in optically-star-forming dwarf galaxies. Future observations in the infrared with the James Webb Space Telescope offer a promising path forward.