The surface potential and the efficiency of interfacial charge transfer are extremely important for designing future semiconductor devices based on the emerging two-dimensional (2D) phosphorene. Here, we directly measured the strongly layer-dependent surface potential of mono- and few-layer phosphorene on gold, which confirms with the reported theoretical prediction. At the same time, we used an optical way - photoluminescence (PL) spectroscopy to probe the charge transfer in phosphorene-gold hybrid system. We firstly observed highly anisotropic and layer-dependent PL quenching in the phosphorene-gold hybrid system, which is attributed to the highly anisotropic/layer-dependent interfacial charge transfer.