I don’t know if there is a doctrinal teaching on this, but I will tell you what Fr. Butler says, and then what seems right to me.
Fr. Butler addresses a whole chapter to the “lost vocation”. He examines three (I think) accounts by apparently devout religious who after many years abandoned their orders, and he tries to demonstrate that by their own records, they did not enter religious life with the right intention. One grew continually more lax in prayer over a period of something like twenty years until he stopped praying and finally stopped having faith altogether – that is clearly a moral failing of someone who may indeed have been called to the religious life.
For my part, I think the suggestion that a religious “vocation” is a perpetual discernment even after final vows is inaccurate. Someone leading a holy life who, with the right intention, makes these vows, can and should keep them: he is, with all people, invited to live the evangelical counsels, he has the ability to do so, and he commits himself to it. If he could never be sure that he had a “vocation,” there would be no point in making final vows (as there would be no point in making marriage vows if marriage were a perpetual discernment). How can one be a religious while lacking the certitude that he is a religious? Moreover, God will not refuse a sincere soul the grace to keep these vows. But just as nuptial vows are broken, the vows of religious can be broken (contrary to marriage, however, they can also be licitly lifted). But I’d be strongly tempted to say that anyone who has left the religious life did not avail himself of the graces offered by God.