The answer is yes and no.
If you aren’t a priest or a religious because you didn’t expect yourself to pursue holiness wholeheartedly and so never even stopped to listen to what kind of vocation God had in mind for you, then yes, you’re probably less holy than if you had been willing to do whatever God asked of you. If you simply will not give yourself to God that much, then of course you may be a lot less holy than you could have been, even if you have no vocation to the priesthood or religious life. (I speak in theory, only, and perhaps to provoke self-examination: Some resist a religious vocation early in life because the grace to be open was not given to them so soon as to others, and others resist because a different vocation was clearly calling them…we must not sit in judgment of anyone in particular, and it is not my intention to arouse scruples in anyone!)
If you aren’t a priest or religious because you were open to whatever God wanted you to do and the answer you got when you opened yourself to the priesthood or religious life was, “No, I have something else in mind for you”, then that is better than trying to plunge ahead into a vocation for which you have no true call. The priesthood and religious states of life are only holier states in life for those who are actually called to that.