In the growing field of capacitive deionization (CDI), a number of performance metrics have emerged to describe the desalination process. Unfortunately, the separation conditions under which these metrics are measured are often not specified, resulting in optimal performance at minimal removal. Here we outline a system of performance metrics and reporting conditions that resolves this issue. Our proposed system is based on volumetric energy consumption (Wh/m$^3$) and throughput productivity (L/h/m$^2$) reported for a specific average concentration reduction, water recovery, and feed salinity. To facilitate and rationalize comparisons between devices, materials, and operation modes, we propose a nominal standard testing condition of removing 5 mM from a 20 mM NaCl feed solution at 50% water recovery for CDI research. Using this separation, we compare the desalination performance of a flow-through electrode (fte-CDI) cell and a flow between membrane (fb-MCDI) device, showing how significantly different systems can be compared in terms of generally desirable desalination characteristics. In general, we find that performance analysis must be considered carefully so to not allow for ambiguous separation conditions or the maximization of one metric at the expense of another. Additionally, for context we discuss a number of important underlying performance indicators and cell characteristics that are not performance measures in and of themselves but can be examined to better understand differences in performance.