Effective field theories describing gravity coupled to matter are investigated, allowing for operators of arbitrary mass dimension. Terms violating local Lorentz and diffeomorphism invariance while preserving internal gauge symmetries are included. The theoretical framework for violations of local Lorentz and diffeomorphism invariance and associated conceptual issues are discussed, including transformations in curved and approximately flat spacetimes, the treatment of various types of backgrounds, the implications of symmetry breaking, and the no-go constraints for explicit violation in Riemann geometry. Techniques are presented for the construction of effective operators, and the possible terms in the gravity, gauge, fermion, and scalar sectors are classified and enumerated. Explicit expressions are obtained for terms containing operators of mass dimension six or less in the effective action for General Relativity coupled to the Standard Model of particle physics. Special cases considered include Einstein-Maxwell effective field theories and the limit with only scalar coupling constants.