Signal occlusion by building blockages is a double-edged sword for the performance of millimeter-wave (mmW) communication networks. Buildings may dominantly attenuate the useful signals, especially when mmW base stations (BSs) are sparsely deployed compared to the building density. In the opposite BS deployment, buildings can block the undesired interference. To enjoy only the benefit, we propose a building-aware association scheme that adjusts the directional BS association bias of the user equipments (UEs), based on a given building density and the concentration of UE locations around the buildings. The association of each BS can thereby be biased: (i) toward the UEs located against buildings for avoiding interference to other UEs; or (ii) toward the UEs providing their maximum reference signal received powers (RSRPs). The proposed association scheme is optimized to maximize the downlink average data rate derived by stochastic geometry. Its effectiveness is validated by simulation using real building statistics.