Under a steady DC electric field of sufficient strength, a weakly conducting dielectric sphere in a dielectric solvent with higher conductivity can undergo spontaneous spinning (Quincke rotation) through a pitchfork bifurcation. We design an object composed of a dielectric sphere and an elastic filament. By solving an elasto-electro-hydrodynamic (EEH) problem numerically, we uncover an EEH instability exhibiting diverse dynamic responses. Varying the bending stiffness of the filament, the composite object displays three behaviours: a stationary state, undulatory swimming and steady spinning, where the swimming results from a self-oscillatory instability through a Hopf bifurcation. By conducting a linear stability analysis incorporating an elastohydrodynamic model, we theoretically predict the growth rates and critical conditions, which agree well with the numerical counterparts. We also propose a reduced model system consisting of a minimal elastic structure which reproduces the EEH instability. The elasto-viscous response of the composite structure is able to transform the pitchfork bifurcation into a Hopf bifurcation, leading to self-oscillation. Our results imply a new way of harnessing elastic media to engineer self-oscillations, and more generally, to manipulate and diversify the bifurcations and the corresponding instabilities. These ideas will be useful in designing soft, environmentally adaptive machines.