The motor system adapts to changes in the body and the environment by learning from sensory prediction errors, the difference between predicted and actual visual feedback. This process is highly automatic and implicit, with the response to the visual feedback occurring even when the resultant change from the sensory prediction error worsens task performance. Given that informative and relevant feedback in the real world is often intertwined with distracting and irrelevant feedback, we asked how the relevancy of visual feedback impacts implicit adaptation. To tackle this question, we presented multiple cursors as visual feedback in a center-out reaching task and varied the task relevance of these cursors. In other words, participants were instructed to hit a target with a specific task-relevant cursor, while ignoring the other cursors. In Experiment 1, we found that reach aftereffects were attenuated by the mere presence of distracting cursors, compared to reach aftereffects in response to a single task-relevant cursor. The degree of attenuation did not depend on the position of the distracting cursors. In Experiment 2, we examined the interaction between task relevance and attention. Participants were asked to adapt to a task-relevant cursor/target pair, while ignoring the task-irrelevant cursor/target pair. Critically, we jittered the location of the relevant and irrelevant target in an uncorrelated manner, allowing us to index attention via how well participants tracked the position of target. As expected, participants who were better at tracking the task-relevant target/cursor pair showed greater aftereffects, and interestingly, the same correlation applied to the task-irrelevant target/cursor pair. Together, these results highlight the novel role of task relevancy on modulating implicit adaptation, perhaps by giving greater attention to informative sources of feedback, increasing the saliency of the sensory prediction error.