We show a dramatic deviation from ergodicity for the conductance fluctuations in graphene. In marked contrast to the ergodicity of dirty metals, fluctuations generated by varying magnetic field are shown to be much smaller than those obtained when sweeping Fermi energy. They also exhibit a strongly anisotropic response to the symmetry-breaking effects of a magnetic field, when applied perpendicular or parallel to the graphene plane. These results reveal a complex picture of quantum interference in graphene, whose description appears more challenging than for conventional mesoscopic systems.