We present Simphony, a free and open-source software toolbox for abstracting and simulating photonic integrated circuits, implemented in Python. The toolbox is both fast and easily extensible; plugins can be written to provide compatibility with existing layout tools, and device libraries can be easily created without a deep knowledge of programming. We include several examples of photonic circuit simulations with novel features and demonstrate a speedup of more than 20x over a leading commercially available software tool.