It is well known that numerical errors grow exponentially in $N$-body simulations of gravitational bound stellar systems, but it is not well understood how the accuracy parameters of algorithms affect the physical evolution in simulations. By using the hybrid $N$-body code, PeTar, we investigate how escapers and the structure evolution of collisional stellar systems (e.g., star clusters) depend on the accuracy of long-range and short-range interactions. We study a group of simulations of ideal low-mass star clusters in which the accuracy parameters are varied. We find that although the number of escapers is different in individual simulations, its distribution from all simulations can be described by Poisson statistics. The density profile also has a similar feature. By using a self-consistent set-up of the accuracy parameters for long- and short-range interactions, such that orbits are resolved well enough, the physical evolution of the models is identical. But when the short-range accuracy is too low, a nonphysical dynamical evolution can easily occur; this is not the case for long-range interactions. This strengthens the need to include a proper algorithm (e.g. regularization methods) in the realistic modelling of collisional stellar systems. We also demonstrate that energy-conservation is not a good indicator to monitor the quality of the simulations. The energy error of the system is controlled by the hardest binary, and thus, it may not reflect the ensemble error of the global system.