Let’s look at the scriptural justification again: John 20:21-23 (Jesus) said to them again, “Peace be with you. As the Father has sent me, so I send you.” And when he had said this, he breathed on them and said to them, “Receive the holy Spirit. Whose sins you forgive are forgiven them, and whose sins you retain are retained.”
Now, when was the last time God “breathed” upon Man? The creation of Adam, a pretty significant event. He doesn’t do that action lightly, it clearly signify’s a LOT.
Now, Christ has established a Church, headed by apostles who are in turn headed by one man (Peter). By the actions of the apostles in the NT, it is clear they pass on the authority (though NOT the title) of the apostles to chosen sucessors, via “the laying on of hands”. NOT to the Church members at large.
Now, look at the second part. “Whose sins you forgive are forgiven them, and whose sins you retain are retained.” This is NOT a call to blanket forgivness, especially to a mass of people. Jesus is COMMANDING the Apostles to make judgements dicernments if you will, about sins and sinners. How can they do that? By listening to sins, and dicerning if the person is truly SINCERE about realizing their offense and wanting forgiveness. He didn’t make the Apostles or their sucessors mind-readers. Granted, undergoing this off-time unpleasent activity is often (not always) proof enough that the sinner IS sincere.
I hope this helps