We review several parallel tempering schemes and examine their main ingredients for accuracy and efficiency. The present study covers two selection methods of temperatures and several choices for the exchange of replicas, including a recent novel all-pair exchange method. We compare the resulting schemes and measure specific heat errors and efficiency using the two-dimensional (2D) Ising model. Our tests suggest that, an earlier proposal for using numbers of local moves related to the canonical correlation times is one of the key ingredients for increasing efficiency, and protocols using cluster algorithms are found to be very effective. Some of the protocols are also tested for efficiency and ground state production in 3D spin glass models where we find that, a simple nearest-neighbor approach using a local n-fold way algorithm is the most effective. Finally, we present evidence that the asymptotic limits of the ground state energy for the isotropic case and that of an anisotropic case of the 3D spin-glass model are very close and may even coincide.