Identification, understanding, and manipulation of novel magnetic textures is essential for the discovery of new quantum materials for future spin-based electronic devices. In particular, materials that manifest a large response to external stimuli such as a magnetic field are subject to intense investigation. Here, we study the kagome-net magnet YMn$_{6}$Sn$_{6}$ by magnetometry, transport, and neutron diffraction measurements combined with first principles calculations. We identify a number of nontrivial magnetic phases, explain their microscopic nature, and demonstrate that one of them hosts a large topological Hall effect (THE). We propose a new nematic chirality mechanism, reminiscent of the nematicity in Fe-based superconductors, which leads to the THE at elevated temperatures. This interesting physics comes from parametrically frustrated interplanar exchange interactions that trigger strong magnetic fluctuations. Our results pave a path to new chiral spin textures, promising for novel spintronics.