Fiber-based optical microcavities exhibit high quality factor and low mode volume resonances that make them attractive for coupling light to individual atoms or other microscopic systems. Moreover, their low mass should lead to excellent mechanical response up to high frequencies, opening the possibility for high bandwidth stabilization of the cavity length. Here, we demonstrate a locking bandwidth of 44 kHz achieved using a simple, compact design that exploits these properties. Owing to the simplicity of fiber feedthroughs and lack of free-space alignment, this design is inherently compatible with vacuum and cryogenic environments. We measure the transfer function of the feedback circuit (closed-loop) and the cavity mount itself (open-loop), which, combined with simulations of the mechanical response of our device, provide insight into underlying limitations of the design as well as further improvements that can be made.