Corner-shared ABX$_3$ perovskites have long featured prominently in solid-state chemistry and condensed matter physics. Still, the joint understanding of their two main subgroups-halides and oxides-has not been fully developed. Indeed, unlike the case that compounds having a single repeated motif (monomorphous), certain cubic perovskites can manifest a non-thermal distribution of local motifs (polymorphous networks). Such intrinsic deformations can include positional degrees of freedom. Unlike thermal motion, such intrinsic distortions do not time-average to zero. The present study compares electronic structure features of oxide and halide perovskites starting from the intrinsic polymorphous network described by DFT minimization of the internal energy, continuing to finite temperature thermal disorder using AIMD. We find that (i) different oxide vs. halide ABX$_3$ compounds adopt different energy-lowering distortion modes. The DFT calculated pair distribution function (PDF) of SrTiO$_3$ agrees with the recently measured PDF. (ii) In both oxides and halides, such intrinsic distortions lead to bandgap blueshifts with respect to monomorphous structure. (iii) For oxide perovskites, high-temperature AIMD simulations initiated from the polymorphous structures reveal that the thermally-induced distortions can lead to a bandgap redshift. (iv) In contrast, for cubic CsPbI$_3$, both the intrinsic distortions and the thermal distortions contribute in tandem to bandgap blueshift, the former, intrinsic effect being dominant. (v) In the oxide SrTiO$_3$ and CaTiO$_3$ (but not in halide), octahedral tilting leads to the emergence of a distinct $Gamma$-$Gamma$ direct bandgap component as a secondary valley minimum to the well-known indirect R-$Gamma$ gap. Understanding such intrinsic vs. thermal effects on oxide vs. halide perovskites holds the potential for designing target electronic properties.