Sometimes distractions are more or less involuntary, and sometimes there are practical issues that prevent full direct participation. If you can’t participate directly or fully, you can engage in other acts that a) aren’t distracting to anyone else and b) are your best option for attending to the Mass as fully you are able to do. It is possible, then, that reciting the rosary might be your best option for keeping your heart and attention where it belongs.
Pretending you’re “all there” when you’re not or refusing to join in the responses of the people when you’re needed and able but don’t care to do so is at least a spiritual fault.
There is no need to concern yourself about someone else is doing, however, unless they are under your spiritual authority or come to you for spiritual advice. It is a virtue and great habit in favor of peace to confine yourself to the duties of your own state in life, too, after all.