We propose a probabilistic quantum protocol to realize a nonlinear transformation of qutrit states, which by iterative applications on ensembles can be used to distinguish two types of pure states. The protocol involves single-qutrit and two-qutrit u
nitary operations as well as post-selection according to the results obtained in intermediate measurements. We utilize the nonlinear transformation in an algorithm to identify a quantum state provided it belongs to an arbitrary known finite set. The algorithm is based on dividing the known set of states into two appropriately designed subsets which can be distinguished by the nonlinear protocol. In most cases this is accompanied by the application of some properly defined physical (unitary) operation on the unknown state. Then, by the application of the nonlinear protocol one can decide which of the two subsets the unknown state belongs to thus reducing the number of possible candidates. By iteratively continuing this procedure until a single possible candidate remains, one can identify the unknown state.
