Purpose: Developing photon-counting CT detectors requires understanding the impact of parameters such as converter material, absorption length and pixel size. We apply a novel linear-systems framework, incorporating spatial and energy resolution, to study realistic silicon (Si) and cadmium telluride (CdTe) detectors at low count rate. Approach: We compared CdTe detector designs with $0.5times0.5; mathrm{mm}^2$ and $0.225times0.225; mathrm{mm}^2$ pixels and Si detector designs with $0.5times0.5; mathrm{mm}^2$ pixels of 30 and 60 mm active absorption length, with and without tungsten scatter blockers. Monte-Carlo simulations of photon transport were used together with Gaussian charge sharing models fitted to published data. Results: For detection in a 300 mm thick object at 120 kVp, the 0.5 mm and 0.225 mm pixel CdTe systems have 28-41 $%$ and 5-29 $%$ higher DQE, respectively, than the 60 mm Si system with tungsten, whereas the corresponding numbers for two-material decomposition are 2 $%$ lower to 11 $%$ higher DQE and 31-54 $%$ lower DQE compared to Si. We also show that combining these detectors with dual-spectrum acquisition is beneficial. Conclusions: In the low-count-rate regime, CdTe detector systems outperform the Si systems for detection tasks, while silicon outperforms one or both of the CdTe systems for material decomposition.