We study notions of genericity in models of $mathsf{PA}$, inspired by lines of inquiry initiated by Chatzidakis and Pillay and continued by Dolich, Miller and Steinhorn in general model-theoretic contexts. These papers studied the theories obtained by adding a random predicate to a class of structures. Chatzidakis and Pillay axiomatized the theories obtained in this way. In this article, we look at the subsets of models of $mathsf{PA}$ which satisfy the axiomatization given by Chatzidakis and Pillay; we refer to these subsets in models of $mathsf{PA}$ as CP-generics. We study a more natural property, called strong CP-genericity, which implies CP-genericity. We use an arithmetic version of Cohen forcing to construct (strong) CP-generics with various properties, including ones in which every element of the model is definable in the expansion, and, on the other extreme, ones in which the definable closure relation is unchanged.