Marriage is natural, that doesn’t make celibacy a ‘charism’. It’s a sacrifice and indeed a gift (because it is a sacrifice!), but please find me a list by the church where celibacy is deemed a charism. If you can, I think it’ll be easier to accept your interpretation of St. Paul.
Sure, there’re special graces in holy orders to enable priests live out their faithfulness, just as there is in marriage for married people. But ‘gift’ and ‘charism’ are not synonymous. If they were, we would see celibacy in the church’s lists of charism.
(1) There are natural talents/'gift’s/potentials we are all born with, (2) there are natural talents/'gift’s/potentials we are born with but differ in (3) there are graces we all find in prayer, sacraments, and daily life, and (4) there are special ‘powers’ or gifts that come to us, also through grace, but in our callings/vocations.
Now, “I wish that all men were [unmarried] as I am. But each man has his own gift from God; one has this gift, another has that,” St. Paul says. And from this you extrapolate that he means the same things we label ‘charisms’ of the Holy Spirit.
But he’s clearly speaking of what we now call ‘vocations’, which come with accompanying ‘gifts’/‘talents’/‘strengths’ etc.
Do we all have a vocation to live celibately, whether in holy orders, the religious life, or through some lay ministry? Of course not. MOST of us are called to marriage as we’ve always known and the church has always taught–forever. Does it mean we can’t live celibately? Nope!
So does it make celibacy a ‘charism’, like miraculous healing or stigmata or prophecy? Or even wisdom, teaching, and counsel? Absolutely not. Celibacy is something every healthy person can do, however difficult or unpleasant it may be sometimes. The rest are not.
To put it differently, you seem to be arguing or suggesting that it’s an ability you’re born with or not, like your natural speed or some other athletic ability. But we are all designed for marriage and able to be celibate if our lives turn out that way. The special gifts the Lord, the apostles, and the church speak of re-celibacy are not to be found in our nature but in grace: which means none of us are excluded should we need to lead a celibate life, for whatever reason, like:
(a) Holy orders or Religious vocation
(b) A spouse who cannot/will not perform the marital act
(c )The lack of a spouse/singleness.
Everyone is able to refrain from sex for moral reasons, regardless of their situation. It is not the special ‘gift’ of some only.