We measured the optical properties of mixed valent vanadium oxide nanoscrolls and their metal exchanged derivatives in order to investigate the charge dynamics in these compounds. In contrast to the prediction of a metallic state for the metal exchanged derivatives within a rigid band model, we find that the injected charges in Mn$^{2+}$ exchanged vanadium oxide nanoscrolls are pinned. A low-energy electronic excitation associated with the pinned carriers appears in the far infrared and persists at low temperature, suggesting that the nanoscrolls are weak metals in their bulk form, dominated by inhomogeneous charge disproportionation and Madelung energy effects.