The classification of topological phases of matter in the presence of interactions is an area of intense interest. One possible means of classification is via studying the partition function under modular transforms, as the presence of an anomalous phase arising in the edge theory of a D-dimensional system under modular transformation, or modular anomaly, signals the presence of a (D+1)-D non-trivial bulk. In this work, we discuss the modular transformations of conformal field theories along a (2+1)-D and a (3+1)-D edge. Using both analytical and numerical methods, we show that chiral complex free fermions in (2+1)-D and (3+1)-D are modular invariant. However, we show in (3+1)-D that when the edge theory is coupled to a background U(1) gauge field this results in the presence of a modular anomaly that is the manifestation of a quantum Hall effect in a (4+1)-D bulk. Using the modular anomaly, we find that the edge theory of (4+1)-D insulator with spacetime inversion symmetry(P*T) and fermion number parity symmetry for each spin becomes modular invariant when 8 copies of the edges exist.