Likelihood Ratio Gradient Estimation for Steady-State Parameters

We consider a discrete-time Markov chain $boldsymbol{Phi}$ on a general state-space ${sf X}$, whose transition probabilities are parameterized by a real-valued vector $boldsymbol{theta}$. Under the assumption that $boldsymbol{Phi}$ is geometrically ergodic with corresponding stationary distribution $pi(boldsymbol{theta})$, we are interested in estimating the gradient $ abla alpha(boldsymbol{theta})$ of the steady-state expectation $$alpha(boldsymbol{theta}) = pi( boldsymbol{theta}) f.$$ To this end, we first give sufficient conditions for the differentiability of $alpha(boldsymbol{theta})$ and for the calculation of its gradient via a sequence of finite horizon expectations. We then propose two different likelihood ratio estimators and analyze their limiting behavior.