Layered perovskites $A_3M_2$O$_7$ are known to exhibit the so-called hybrid improper ferroelectricity. Despite experimentally confirmed cases (e.g. nonmagnetic $M$=Ti and Sn), the ferroelectricity in magnetic Ca$_3$Mn$_2$O$_7$ remains a puzzle. Here, the structural, ferroelectric, magnetoelectric, and optical properties of Ca$_3$Mn$_2$O$_7$ are systematically investigated. Switchable polarization is directly measured, demonstrating its ferroelectricity. In addition, magnetoelectric response is also evidenced, implying the coupling between magnetism and ferroelectricity. Furthermore, strong visible light absorption is observed, which can be understood from its electronic structure. Its direct and appropriate band gap, as well as wide conducting bands, makes Ca$_3$Mn$_2$O$_7$ a potential candidate for ferroelectric photoelectric applications.