The discovery of multiple coexisting magnetic phases in a crystallographically homogeneous compound Ca$_3$Co$_2$O$_6$ has stimulated an ongoing research activity. In recent years the main focus has been on the zero field state properties, where exceedingly long time scales have been established. In this study we report a detailed investigation of static and dynamic properties of Ca$_3$Co$_2$O$_6$ across the magnetic field induced transition around 3.5 T. This region has so far been practically neglected while we argue that in some aspects it represents a simpler version of the transition across the $B = 0$ state. Investigating the frequency dependence of the ac susceptibility we reveal that on the high field side ($B > 3.5$ T) the response corresponds to a relatively narrow distribution of magnetic clusters. The distribution appears very weakly dependent on magnetic field, with an associated energy barrier of around 200 K. Below 3.5 T a second contribution arises, with much smaller characteristic frequencies and a strong temperature and magnetic field dependence. We discuss these findings in the context of intra-chain and inter-chain clustering of magnetic moments.