Levitated oscillators with millimeter or sub-millimeter size are particularly attractive due to their potential role in studying various fundamental problems and practical applications. One of the crucial issues towards these goals is to achieve efficient measurements of oscillator motion, while this remains a challenge. Here we theoretically propose a lens-free optical detection scheme, which can be used to detect the motion of a millimeter or sub-millimeter levitated oscillator with a measurement efficiency close to the standard quantum limit with a modest optical power. We demonstrate experimentally this scheme on a 0.5 mm diameter micro-sphere that is diamagnetically levitated under high vacuum and room temperature, and the thermal motion is detected with high precision. Based on this system, an estimated acceleration sensitivity of $9.7 times 10^{-10}rm g/sqrt{Hz}$ is achieved, which is more than one order improvement over the best value reported by the levitated mechanical system. Due to the stability of the system, the minimum resolved acceleration of $3.5times 10^{-12}rm g$ is reached with measurement times of $10^5$ s. This result is expected to have potential applications in the study of exotic interactions in the millimeter or sub-millimeter range and the realization of compact gravimeter and accelerometer.