Let V be a symplectic vector space and let $mu$ be the oscillator representation of Sp(V). It is natural to ask how the tensor power representation $mu^{otimes t}$ decomposes. If V is a real vector space, then Howe-Kashiwara-Vergne (HKV) duality asserts that there is a one-one correspondence between the irreducible subrepresentations of Sp(V) and the irreps of an orthogonal group O(t). It is well-known that this duality fails over finite fields. Addressing this situation, Gurevich and Howe have recently assigned a notion of rank to each Sp(V) representation. They show that a variant of HKV duality continues to hold over finite fields, if one restricts attention to subrepresentations of maximal rank. The nature of the rank-deficient components was left open. Here, we show that all rank-deficient Sp(V)-subrepresentations arise from embeddings of lower-order tensor products of $mu$ and $barmu$ into $mu^{otimes t}$. The embeddings live on spaces that have been studied in quantum information theory as tensor powers of self-orthogonal Calderbank-Shor-Steane (CSS) quantum codes. We then find that the irreducible Sp(V) subrepresentations of $mu^{otimes t}$ are labelled by the irreps of orthogonal groups O(r) acting on certain r-dimensional spaces for r <= t. The results hold in odd charachteristic and the stable range t <= 1/2 dim V. Our work has implications for the representation theory of the Clifford group. It can be thought of as a generalization of the known characterization of the invariants of the Clifford group in terms of self-dual codes.