Access to and availability of MRI scanners is typically limited by their cost, siting and infrastructure requirements. This precludes MRI diagnostics, the reference standard for neurological assessment, in patients who cannot be transported to specialized scanner suites. This includes patients who are critically ill and unstable, and patients located in low-resource settings. The scanner design presented here aims to extend the reach of MRI by substantially reducing these limitations. Our goal is to shift the cost-benefit calculation for MRI toward more frequent and varied use, including improved accessibility worldwide and point of care operation. Here, we describe a portable brain MRI scanner using a compact, lightweight permanent magnet, with a built-in readout field gradient. Our low-field (80 mT) Halbach cylinder design of rare-earth permanent magnets results in a 122 kg magnet with minimal stray-field, requiring neither cryogenics nor external power. The built-in magnetic field gradient reduces reliance on high-power gradient drivers, which not only lowers overall system power and cooling requirements, but also reduces acoustic noise. Imperfections in the encoding fields are mitigated with a generalized iterative image reconstruction technique, that uses prior characterization of the field patterns. Our system was validated using T1, T2 and proton density weighted in vivo brain images with a spatial resolution of 2.2 x 1.3 x 6.8 mm$^3$.