In the preparation of Cafe Latte, spectacular layer formation can occur between the expresso shot in a glass of milk and the milk itself. Xue et al. (Nat. Commun., vol. 8, 2017, pp. 1-6) showed that the injection velocity of expresso determines the depth of coffee-milk mixture. After a while when a stable stratification forms in the mixture, the layering process can be modelled as a double diffusive convection system with a stably-stratified coffee-milk mixture cooled from the side. More specifically, we perform (two-dimensional) direct numerical simulations of laterally cooled double diffusive convection for a wide parameter range, where the convective flow is driven by a lateral temperature gradient while stabilized by a vertical concentration gradient. When the thermal driving force dominates over the stabilizing force, the flow behaves like vertical convection in which a large-scale circulation develops. However, with increasing strength of the stabilizing force, a meta-stable layered regime emerges. Initially, several vertically-stacked convection rolls develop, and these well-mixed layers are separated by sharp interfaces with large concentration gradients. The initial thickness of these emerging layers can be estimated by balancing the work exerted by thermal driving and the required potential energy to bring fluid out of its equilibrium position in the stably stratified fluid. In the layered regime, we further observe successive layer merging, and eventually only a single convection roll remains. We elucidate the following merging mechanism: As weakened circulation leads to accumulation of hot fluid adjacent to the hot sidewall, larger buoyancy forces associated with hotter fluid eventually break the layer interface. Then two layers merge into a larger layer, and circulation establishes again within the merged structure.