We propose a prime factorizer operated in a framework of quantum annealing (QA). The idea is inverse operation of a multiplier implemented with QA-based Boolean logic circuits. We designed the QA machine on an application-specific-annealing-computing architecture which efficiently increases available hardware budgets at the cost of restricted functionality. The invertible operation of QA logic gates consisting of superconducting flux qubits was confirmed by circuit simulation with classical noise sources. The circuits were implemented and fabricated by using superconducting integrated circuit technologies with Nb/AlOx/Nb Josephson junctions. We also propose a 2.5Dimensional packaging scheme of a qubit-chip/interpose /package-substrate structure for realizing practically large-scale QA systems.