Thank God that he has blessed you with the grace to take this important step, and that you have the courage to cooperate with that grace!
I assume that you are still living together during the annulment process, and this may be the problem. It can cause scandal, and living together is also a tremendous temptation for you both. He may fear you are not stong enough to succeed, and he does not want you to approach the Eucharist unworthily.
Discuss this again with your confessor. Find out if these are his reasons. Perhaps you can live apart for awhile, or if scandal is not an issue, perhaps by making separate sleeping arrangements, your confessor will be more convinced of your resolve to live chastely until you can be validly married.
But if you must wait to receive, then offer up your hunger for the Eucharist in reparation for your sins of the past. But in any case, remain committed to chastity and pray daily for the grace to live out your commitment.
May God keep you strong during this time.