An optical equivalent of the field-programmable gate array (FPGA) is of great interest to large-scale photonic integrated circuits. Previous programmable photonic devices relying on the weak, volatile thermo-optic or electro-optic effect usually suffer from a large footprint and high energy consumption. Phase change materials (PCMs) offer a promising solution due to the large non-volatile change in the refractive index upon phase transition. However, the large optical loss in PCMs poses a serious problem. Here, by exploiting an asymmetric directional coupler design, we demonstrate PCM-clad silicon photonic 1 times 2 and 2 times 2 switches with a low insertion loss of ~1 dB and a compact coupling length of ~30 {mu}m while maintaining a small crosstalk less than ~10 dB over a bandwidth of 30 nm. The reported optical switches will function as the building blocks of the meshes in the optical FPGAs for applications such as optical interconnects, neuromorphic computing, quantum computing, and microwave photonics.