I feel your pain, I’ve had similar statements said to me.
I will first quote Tertullian (197-220 A.D.)
“Did not certain of the disciples turn back from the Lord Himself, (John 6:66) when they were offended? Yet the rest did not therefore think that they must turn away from following Him, but because they knew that He was the Word of Life, and had come from God, they continued in His company to the very last, after He had gently inquired of them whether they also would go away. (John 6:67) It is a comparatively small thing, that certain men, like Phygellus, and Hermogenes, and Philetus, and Hymenæus, deserted His apostle: the betrayer of Christ was himself one of the apostles. We are surprised at seeing His churches forsaken by some men, although the things which we suffer after the example of Christ Himself, show us to be Christians. They went out from us, says (St. John,) but they were not of us. If they had been of us, they would no doubt have continued with us.”
Tertullian, “The Prescription Against Heretics,” 197-220 A.D… [Online]. Available:
earlychristianwritings.com/text/tertullian11.html.
I like to think of it this way, if one teacher molests a child, does that make the school/institution wrong? Does that make all the other teachers paedophiles? Now, what if every teacher except one molests children, doest that make that one good teacher a paedophile? Does that make the institution wrong? The answer to all these questions should be a resounding NO. We cannot therefore assume that simply because it may appear to us that all Catholics are immoral that they do not have the fullness of Truth right within their grasp and that all Catholics are bad. If he thinks that all Catholics are bad, this must mean that he has never turned on a t.v and heard of mother Theresa. Even one of those closest to Jesus, Judas, turned his back on God himself and betrayed him for money. Even Solomon, David and Saul, though they were close to God chose to ignore our Lords supremely wise counsel.