Linear optical computing (LOC) with thermal light has recently gained attention because the problem is connected to the permanent of a Hermitian positive semidefinite matrix (HPSM), which is of importance in the computational complexity theory. Despite the several theoretical analyses on the computational structure of an HPSM in connection to LOC, the experimental demonstration and the computational complexity analysis via the linear optical system have not been performed yet. We present, herein, experimental LOC for estimating the permanent of an HPSM. From the linear optical experiments and theoretical analysis, we find that the LOC efficiency for a multiplicative error is dependent on the value of the permanent and that the lower bound of the computation time scales exponentially. Furthermore, our results are generalized and applied to LOC of permanents of unitary matrices, which can be implemented with a multi-port quantum interferometer involving single-photons at the input ports. We find that LOC with single-photons, for the permanent estimation, is on average less efficient than the most efficient classical algorithm known to date, even in ideal conditions.