I believe it was physical, but also spiritual. Consider what the numbers mean.
Jesus feeds five thousand with five loaves and two fish in all four Gospels (Matthew 14:13-21, Mark 6:31-44, Luke 9:10-17 and John 6:5-15).
and he ordered the crowds to sit down on the grass. Taking the five loaves and the two fish, and looking up to heaven, he said the blessing, broke the loaves, and gave them to the disciples, who in turn gave them to the crowds. They all ate and were satisfied, and they picked up the fragments left over—twelve wicker baskets full. Those who ate were about five thousand men, not counting women and children. (Matthew 14:19-21)
This was done in Israel and all present were Jews. Five represents the first five books in the Hebrew Bible or the Jewish Law of Moses. Two represents the two pillars of Jewish teaching - the law and the prophets. Jesus is feeding the Jews with the fullness of revelation from both the law and the prophets. Here Jesus has twelve baskets left over, the same number of tribes in Israel. Remember that the ten northern tribes had never returned from the Babylonian exile and so the twelve baskets left over represents the return of the ten tribes and the reunification of all twelve tribes under their new king.
Jesus then moves to Galille which is not Jewish land and feeds four thousand non-Jews with seven loaves and has seven baskets left over (Mark 8:1-9 and Matthew 15:32-39).
Jesus said to them, “How many loaves do you have?” “Seven,” they replied, “and a few fish.” He ordered the crowd to sit down on the ground. Then he took the seven loaves and the fish, gave thanks, broke the loaves, and gave them to the disciples, who in turn gave them to the crowds. They all ate and were satisfied. They picked up the fragments left over—seven baskets full. Those who ate were four thousand men, not counting women and children. (Matthew 15:34-38)
Four represents the four corners of the earth and so four thousand Gentiles represents the fullness of the Gentile nations. Seven represents perfection and so Jesus feeding four thousand non-Jews with seven loaves and having seven baskets left over represents his brining the fullness of all Gentiles to perfection.
Again, I believed it actually happened, but there is a deep spiritual and theological message behind the text and the numbers tell us a lot.
-Tim-