Ghost imaging incorporating deep learning technology has recently attracted much attention in the optical imaging field. However, deterministic illumination and multiple exposure are still essential in most scenarios. Here we propose a ghost imaging scheme based on a novel conjugate-decoding deep learning framework (Y-net), which works well under both deterministic and indeterministic illumination. Benefited from the end-to-end characteristic of our network, the image of a sample can be achieved directly from a pair of correlated speckles collected by the detectors, and the sample is illuminated only once in the experiment. The spatial distribution of the speckles encoding the sample in the experiment can be completely different from that of the simulation speckles for training, as long as the statistical characteristics of the speckles remain unchanged. This approach is particularly important to high-resolution x-ray ghost imaging applications due to its potential for improving image quality and reducing radiation damage. And the idea of conjugate-decoding network may also be applied to other learning-based imaging