We present ~800 days of photometric monitoring of Boyajians Star (KIC 8462852) from the All-Sky Automated Survey for Supernovae (ASAS-SN) and ~4000 days of monitoring from the All Sky Automated Survey (ASAS). We show that from 2015 to the present the brightness of Boyajians Star has steadily decreased at a rate of 6.3 +/- 1.4 mmag yr^-1, such that the star is now 1.5% fainter than it was in February 2015. Moreover, the longer time baseline afforded by ASAS suggests that Boyajians Star has also undergone two brightening episodes in the past 11 years, rather than only exhibiting a monotonic decline. We analyze a sample of ~1000 comparison stars of similar brightness located in the same ASAS-SN field and demonstrate that the recent fading is significant at >99.4% confidence. The 2015-2017 dimming rate is consistent with that measured with Kepler data for the time period from 2009 to 2013. This long-term variability is difficult to explain with any of the physical models for the stars behavior proposed to date.