not to mention that when he was referring to his followers, he called them his sheep, and Christ said ‘you’ as in Peter and the apostles, to tend to his ‘sheep’ (us). So basically, the idea of ‘you’ when Christ is giving his apostles the power to retain and forgive sins, he meant those particular people, because he then (or before, I am very bad on scripture, sorry) asks Peter to tend to his sheep, i.e. his followers.
In summary (for loose in heaven etc. part):
You = individual
Sheep = followers.
It is indeed about context, and also to see what other words there may be to mean the followers and the apostles.