People change their physical contacts as a preventive response to infectious disease propagations. Yet, only a few mathematical models consider the coupled dynamics of the disease propagation and the contact adaptation process. This paper presents a model where each agent has a default contact neighborhood set, and switches to a different contact set once she becomes alert about infection among her default contacts. Since each agent can adopt either of two possible neighborhood sets, the overall contact network switches among 2^N possible configurations. Notably, a two-layer network representation can fully model the underlying adaptive, state-dependent contact network. Contact adaptation influences the size of the disease prevalence and the epidemic threshold---a characteristic measure of a contact network robustness against epidemics---in a nonlinear fashion. Particularly, the epidemic threshold for the presented adaptive contact network belongs to the solution of a nonlinear Perron-Frobenius (NPF) problem, which does not depend on the contact adaptation rate monotonically. Furthermore, the network adaptation model predicts a counter-intuitive scenario where adaptively changing contacts may adversely lead to lower network robustness against epidemic spreading if the contact adaptation is not fast enough. An original result for a class of NPF problems facilitate the analytical developments in this paper.