No, that’s not immature. It is sensible to avoid situations where hurting someone’s feelings or just having an unenjoyable awkward moment is a reasonably likely outcome, particularly when there is no foreseeable good likely to come out of these shared moments.
As for the abuser, just stay away. There is no statute of limitations that requires employment of retired robbers by a bank. Forgiveness does not preclude keeping a distance between yourself and someone who has abused your trust, no matter how long it has been. Let him close to him only when it seems sensible to do so, and then only if you actually want to. He’s had his chance at that position with you to which no one has a right.
Conversely, I would not avoid situations likely to be enjoyable because of a mere possibility that a few minutes of the outing might be awkward. Still, if anxiety is going to ruin the whole event for you, there are probably plenty of other ways to entertain yourself without that stress. Go, or don’t go, but it isn’t anything to get too worked up about.