This paper focuses on the fast evaluation of the matvec $g=Kf$ for $Kin mathbb{C}^{Ntimes N}$, which is the discretization of a multidimensional oscillatory integral transform $g(x) = int K(x,xi) f(xi)dxi$ with a kernel function $K(x,xi)=e^{2pii Phi(x,xi)}$, where $Phi(x,xi)$ is a piecewise smooth phase function with $x$ and $xi$ in $mathbb{R}^d$ for $d=2$ or $3$. A new framework is introduced to compute $Kf$ with $O(Nlog N)$ time and memory complexity in the case that only indirect access to the phase function $Phi$ is available. This framework consists of two main steps: 1) an $O(Nlog N)$ algorithm for recovering the multidimensional phase function $Phi$ from indirect access is proposed; 2) a multidimensional interpolative decomposition butterfly factorization (MIDBF) is designed to evaluate the matvec $Kf$ with an $O(Nlog N)$ complexity once $Phi$ is available. Numerical results are provided to demonstrate the effectiveness of the proposed framework.