Recent advances in quantum technology have led to the development and manufacturing of experimental programmable quantum annealing optimizers that contain hundreds of quantum bits. These optimizers, named `D-Wave chips, promise to solve practical optimization problems potentially faster than conventional `classical computers. Attempts to quantify the quantum nature of these chips have been met with both excitement and skepticism but have also brought up numerous fundamental questions pertaining to the distinguishability of quantum annealers from their classical thermal counterparts. Here, we propose a general method aimed at answering these, and apply it to experimentally study the D-Wave chip. Inspired by spin-glass theory, we generate optimization problems with a wide spectrum of `classical hardness, which we also define. By investigating the chips response to classical hardness, we surprisingly find that the chips performance scales unfavorably as compared to several analogous classical algorithms. We detect, quantify and discuss purely classical effects that possibly mask the quantum behavior of the chip.