The conjectures of Alday, Gaiotto and Tachikawa and its generalizations have been mathematically formulated as the existence of an action of a $W$-algebra on the cohomology or $K$-theory of the instanton moduli space, together with a Whitakker vector. However, the original conjectures also predict intertwining properties with the natural higher rank version of the $Ext^1$ operator which was previously studied by Okounkov and the author in [CO], a result which is now sometimes referred to as AGT in rank one [Alb,PSS]. Physically, this corresponds to incorporating matter in the Nekrasov partition functions, an obviously important feature in the physical theory. It is therefore of interest to study how the $Ext^1$ operator relates to the aforementioned structures on cohomology in higher rank, and if possible to find a formulation from which the AGT conjectures follow as a corollary. In this paper, we carry out something analogous using a modified Segal-Sugawara construction for the $hat{mathfrak{sl}}_2mathbb{C}$ structure that appears in Okounkov and Nekrasovs proof of Nekrasovs conjecture [NO] for rank two. This immediately implies the AGT identities when the central charge is one, a case which is of particular interest for string theorists, and because of the natural appearance of the Seiberg-Witten curve in this setup, see for instance Dijkgraaf and Vafa [DV], as well as [IKV].