We characterize a natural class of modular categories of prime power Frobenius-Perron dimension as representation categories of twisted doubles of finite p-groups. We also show that a nilpotent braided fusion category C admits an analogue of the Sylow decomposition. If the simple objects of C have integral Frobenius-Perron dimensions then C is group-theoretical. As a consequence, we obtain that semisimple quasi-Hopf algebras of prime power dimension are group-theoretical. Our arguments are based on a reconstruction of twisted group doubles from Lagrangian subcategories of modular categories (this is reminiscent to the characterization of doubles of quasi-Lie bialgebras in terms of Manin pairs).