Let $d$ be a positive integer, and let $mu$ be a finite measure on $br^d$. In this paper we ask when it is possible to find a subset $Lambda$ in $br^d$ such that the corresponding complex exponential functions $e_lambda$ indexed by $Lambda$ are orthogonal and total in $L^2(mu)$. If this happens, we say that $(mu, Lambda)$ is a spectral pair. This is a Fourier duality, and the $x$-variable for the $L^2(mu)$-functions is one side in the duality, while the points in $Lambda$ is the other. Stated this way, the framework is too wide, and we shall restrict attention to measures $mu$ which come with an intrinsic scaling symmetry built in and specified by a finite and prescribed system of contractive affine mappings in $br^d$; an affine iterated function system (IFS). This setting allows us to generate candidates for spectral pairs in such a way that the sets on both sides of the Fourier duality are generated by suitably chosen affine IFSs. For a given affine setup, we spell out the appropriate duality conditions that the two dual IFS-systems must have. Our condition is stated in terms of certain complex Hadamard matrices. Our main results give two ways of building higher dimensional spectral pairs from combinatorial algebra and spectral theory applied to lower dimensional systems.