This work evaluates the viability of polyimide and parylene-C for passivation of lithium-drifted silicon (Si(Li)) detectors. The passivated Si(Li) detectors will form the particle tracker and X-ray detector of the General Antiparticle Spectrometer (GAPS) experiment, a balloon-borne experiment optimized to detect cosmic antideuterons produced in dark matter annihilations or decays. Successful passivation coatings were achieved by thermally curing polyimides, and the optimized coatings form an excellent barrier against humidity and organic contamination. The passivated Si(Li) detectors deliver $lesssim,4$ keV energy resolution (FWHM) for 20$-$100 keV X-rays while operating at temperatures of $-$35 to $-45,^{circ}$C. This is the first reported successful passivation of Si(Li)-based X-ray detectors operated above cryogenic temperatures.