Open Fabry-Perot microcavities represent a promising route for achieving a quantum electrodynamics (cavity-QED) platform with diamond-based emitters. In particular, they offer the opportunity to introduce high purity, minimally fabricated material into a tunable, high quality factor optical resonator. Here, we demonstrate a fiber-based microcavity incorporating a thick (> 10 {mu}m) diamond membrane with a finesse of 17,000, corresponding to a quality factor Q ~ $10^6$. Such minimally fabricated, thick samples can contain optically stable emitters similar to those found in bulk diamond. We observe modified microcavity spectra in the presence of the membrane, and develop analytic and numerical models to describe the effect of the membrane on cavity modes, including loss and coupling to higher-order transverse modes. We estimate that a Purcell enhancement of approximately 20 should be possible for emitters within the diamond in this device, and provide evidence that better diamond surface treatments and mirror coatings could increase this value to 200 in a realistic system.