A remnant is not necessarily small.
One example of a remnant in the Bible is Jesus and his Apostles following certain scandals or scandalous affairs.
Judas betraying Jesus created a scandal for the Apostles. The persecution and death of Jesus scandalized the Apostles, until his ressurection. Some abandoned him (perhaps too strong of a word?) while he slept in the grave. The “remnant”, thanks to our Lord, was ultimately very significant: all except Judas were recovered. How many of his disciples left him during this same time, and how many were recovered at Pentecost, is a matter of debate.
God is not feeble. The Catholic Church, by definition as the Mystical Body of Christ, true, and faithful to Our Lord’s teachings, is always “the remnant.” Following each sweep of heresy or schism those who persisted and remained Catholic are the remnant.
Traditional Protestant arguments regarding the remnant would have made a Jew in the 1st century question whether or not he should remain loyal to the Temple worship in Jerusalem or join the schismatical/heretical sect of Samaritans who worshipped God on a hill. This doubting Jew may have wondered if the corruption of the Sadducees, Jewish leaders or perhaps misconduct or scandals of Levitical Priests, the High Priest, etc., meant God had abondoned the Jews or the Jews in Judaea could not be the true People of God. Maybe our 1st century Jew would have concluded the Samaritan’s were the true “remnant.” Thankfully, Our Lord reassured us (and them) that the Jerusalem Temple was the proper form and obedience of God’s People. The Jews, who greatly outnumbered the Samaritans, were the “remnant,” though this did not exclude others (such as the Samaritans) from God’s plan of salvation.
Anyways, that’s how I feel