I also find holding hands during the Lord’s Prayer to be distracting. Even in my Episcopal church as a child, we never held hands, nor did anyone try to. I found it peaceful to look at the cross (we did not have a crucifix at my childhood Episcopal church) and pray straight to God adding in my own thoughts any transgressions I wished to repent and pondering the kingdom of God in reverence. I wish this did not occur in my current Catholic church that I attend with husband (now of five years) and two children. Fortunately I have the excuse not to hold hands because I am often holding a toddler. I would like the priest or a layperson after the mass to make the statement that no one should feel obliged to hold hands. This is a personal thing to me, a time of communion with God before Eucharist to set my heart right to recieve the sacrament (or in my case since I have not been confirmed, but am finally taking RCIA classes to do so, to accept the blessing of God from the priest). The previous Catholic parish I attended did not hold hands either, so this is only a relatively recent distraction for me.
~Take care and God Bless