The “small remnant” referred to in Isaiah (and others) has always been a warning to Israel and the Jewish people:
When Joseph was unfairly treated by his brothers and sold into slavery, he later became second in power next to Pharaoh and said “God sent me before you to preserve for you a remnant on earth, and to keep alive for you many survivors” (Gen 45:7).
Also, “When Joshua and the sons of Israel had finished striking (an enemy nation) with a great blow until they were wiped out, and when the remnant that remained of them had entered into the fortified cities” (Joshua 10:20) this was not good. Joshua told the nation of Israel “Be very careful, therefore, to love the Lord your God. For if you turn back and cling to the remnant of these nations remaining among you and make marriages with them, so that you associate with them and they with you, know for certain that the Lord your God will no longer drive out these nations before you, but they shall be a snare and a trap for you, a whip on your sides and thorns in your eyes, until you perish from off this good ground that the Lord your God has given you” (Joshua 23:11-13) so here, a remnant of a foreign nation left in the land would come back to haunt them and in fact, it did.