The PHENIX experiment has studied nuclear effects in $p$$+$Al and $p$$+$Au collisions at $sqrt{s_{_{NN}}}=200$ GeV on charged hadron production at forward rapidity ($1.4<eta<2.4$, $p$-going direction) and backward rapidity ($-2.2<eta<-1.2$, $A$-going direction). Such effects are quantified by measuring nuclear modification factors as a function of transverse momentum and pseudorapidity in various collision multiplicity selections. In central $p$$+$Al and $p$$+$Au collisions, a suppression (enhancement) is observed at forward (backward) rapidity compared to the binary scaled yields in $p$+$p$ collisions. The magnitude of enhancement at backward rapidity is larger in $p$$+$Au collisions than in $p$$+$Al collisions, which have a smaller number of participating nucleons. However, the results at forward rapidity show a similar suppression within uncertainties. The results in the integrated centrality are compared with calculations using nuclear parton distribution functions, which show a reasonable agreement at the forward rapidity but fail to describe the backward rapidity enhancement.