I think one thing you need to consider is your wants over your calling. Perhaps what the priest is trying to get at is the fact that you already intend to put off your calling for four years to do something, beit noble, but still much the same.
As you grow older time moves faster and this could easily be one more thing ad naseum.
As you know your vocation BEFORE you go into the military and this is your option I think it will be very difficult. God dosn’t like to wait. Just think if you would do this to a woman you loved and wanted to have an intimate relationship with. Waiting four years would sound ludicroius.