Most biological fluids are viscoelastic, meaning that they have elastic properties in addition to the dissipative properties found in Newtonian fluids. Computational models can help us understand viscoelastic flow, but are often limited in how they deal with complex flow geometries and suspended particles. Here, we present a lattice Boltzmann solver for Oldroyd-B fluids that can handle arbitrarily-shaped fixed and moving boundary conditions, which makes it ideally suited for the simulation of confined colloidal suspensions. We validate our method using several standard rheological setups, and additionally study a single sedimenting colloid, also finding good agreement with literature. Our approach can readily be extended to constitutive equations other than Oldroyd-B. This flexibility and the handling of complex boundaries holds promise for the study of microswimmers in viscoelastic fluids.