Quantum key distribution (QKD) promises provably secure cryptography, even to attacks from an all-powerful adversary. However, with quantum computing development lagging behind QKD, the assumption that there exists an adversary equipped with a universal fault-tolerant quantum computer is unrealistic for at least the near future. Here, we explore the effect of restricting the eavesdroppers computational capabilities on the security of QKD, and find that improved secret key rates are possible. Specifically, we show that for a large class of discrete variable protocols higher key rates are possible if the eavesdropper is restricted to a unitary operation from the Clifford group. Further, we consider Clifford-random channels consisting of mixtures of Clifford gates. We numerically calculate a secret key rate lower bound for BB84 with this restriction, and show that in contrast to the case of a single restricted unitary attack, the mixture of Clifford based unitary attacks does not result in an improved key rate.