We present a measurement of the ratio of top quark branching fractions R = B(t -> Wb)/B(t -> Wq), where q can be a d, s or b quark, in the lepton+jets and dilepton ttbar final states. The measurement uses data from 5.4 fb-1 of ppbar collisions collected with the D0 detector at the Fermilab Tevatron Collider. We measure R = 0.90 +/- 0.04, and we extract the CKM matrix element |Vtb| as |Vtb| = 0.95 +/- 0.02, assuming unitarity of the 3x3 CKM matrix.