Unmanned aerial vehicles (UAVs) can be utilized as aerial base stations to provide communication service for remote mobile users due to their high mobility and flexible deployment. However, the line-of-sight (LoS) wireless links are vulnerable to be intercepted by the eavesdropper (Eve), which presents a major challenge for UAV-aided communications. In this paper, we propose a latency-minimized transmission scheme for satisfying legitimate users (LUs) content requests securely against Eve. By leveraging physical-layer security (PLS) techniques, we formulate a transmission latency minimization problem by jointly optimizing the UAV trajectory and user association. The resulting problem is a mixed-integer nonlinear program (MINLP), which is known to be NP hard. Furthermore, the dimension of optimization variables is indeterminate, which again makes our problem very challenging. To efficiently address this, we utilize bisection to search for the minimum transmission delay and introduce a variational penalty method to address the associated subproblem via an inexact block coordinate descent approach. Moreover, we present a characterization for the optimal solution. Simulation results are provided to demonstrate the superior performance of the proposed design.