V1280 Sco is one of the slowest dust-forming nova ever historically observed. We performed multi-epoch high-spatial resolution observations of the circumstellar dusty environment of V1280 Sco to investigate the level of asymmetry of the ejecta We observed V1280 Sco in 2009, 2010 and 2011 using unprecedented high angular resolution techniques. We used the NACO/VLT adaptive optics system in the J, H and K bands, together with contemporaneous VISIR/VLT mid-IR imaging that resolved the dust envelope of V1280 Sco, and SINFONI/VLT observations secured in 2011. We report the discovery of a dusty hourglass-shaped bipolar nebula. The apparent size of the nebula increased from 0.30 x 0.17 in July 2009 to 0.64 x 0.42 in July 2011. The aspect ratio suggests that the source is seen at high inclination. The central source shines efficiently in the K band and represents more than 56+/-5% of the total flux in 2009, and 87+/-6% in 2011. A mean expansion rate of 0.39+/-0.03 mas per day is inferred from the VISIR observations in the direction of the major axis, which represents a projected upper limit. Assuming that the dust shell expands in that direction as fast as the low-excitation slow ejecta detected in spectroscopy, this yields a lower limit distance to V1280 Sco of 1kpc; however, the systematic errors remain large due to the complex shape and velocity field of the dusty ejecta. The dust seems to reside essentially in the polar caps and no infrared flux is detected in the equatorial regions in the latest dataset. This may imply that the mass-loss was dominantly polar.