According to my pastor, Ordinary here does not mean “normal or usual” or “not very impressive” (these are among the definitions in the dictionary). Rather it means ordered as in Ordinal Numbers.
Cardinal numbers count the quantity; one, two, three, and so on.
Ordinal numbers count the order: first, second, third, and so on.
I noticed that the Sundays of Ordinary time are numbered in this way. Last Sunday was the Thirtieth Sunday in Ordinary Time.