It does matter when the Mass is not centered on Christ, but rather centered on people celebrating together be it as individuals or as a group. It is true that there are great values when people come together and celebrate. But with Christ not being in the center of worship, it is not a Mass. It is simply at best a great gathering of people.
Christ is of course everywhere. We pray daily to Him from anywhere while facing any direction. That is not the issue. In Ad Orientem, when the priest faces and leads his flock to Christ from the liturgical East, it is a clear, faithful, communal act leading by the priest focused completely on Christ.