As Catholics, I’m not sure we should honour all the modern ways of describing people. Maybe, saying “a man with Down’s syndrome” shows you’re not labelling him, but the number one thing you’re still saying is that he has Down’s syndrome. But why say Down’s syndrome at all? Perhaps we shouldn’t describe people. The OP would have done better to not tell us he had a chromosonal variation, and focus on the irritating behaviour.
You will agree it’s awkward to say people of height (tall people), people of substantial weight and rotund shape (fat people), people of the feminine gender (females), people of Catholic belief. So why say, “people with Down’s syndrome?”
What is more important is that we respect everybody.