The double-electron capture and the electron capture with positron emission in $^{168}$Yb have been investigated for the first time at the STELLA facility of the Gran Sasso underground laboratory (Italy) measuring 371 g of highly purified ytterbium oxide placed on the end-cap of a 465 cm$^3$ ultra-low-background high purity Germanium detector (HPGe). No gamma associated to double beta processes in $^{168}$Yb have been observed after 2074 h of data taking. This has allowed setting the half-life limits on the level of $lim T_{1/2}sim$ $10^{14}-10^{18}$ yr at 90% C.L. Particularly, a lower half-life limit on a possible resonant neutrinoless double-electron capture in $^{168}$Yb to the $(2)^-$ 1403.7 keV excited state of $^{168}$Er is set as $T_{1/2}geq1.9times 10^{18}$ yr at 90% C.L. Half-life limits $T_{1/2}^{2 u(0 u)}geq 4.5(4.3)times10^{16}$ yr were set on the $2 u(0 u)2beta^-$ decay of $^{176}$Yb to the $2^+$ 84.3 keV first excited level of $^{176}$Hf.