Your pastor was incorrect in that there is nothing in the GIRM as to how people are to hold their hands - with one another, solo, palms together fingers pointing up or out; palms together, fingers interlaces, hands in pockets, hands holding a baby…
There are people who wish to hold hands and are offended when the person next to them does not. They take it as a personal rejection.
There are people who do not want to hold hands and are offended when the person next to them wants to. They take it as an intrusion of their personal space.
It is easy to condemn either of the two people noted above if one is on the “other side” of the issue; both are somewhat natural responses. And while holding hands during the Our Father is not the following, it is well known, at least among sociologists that people of a Northern European heritage tend to be more reserved and less prone to any showing which might be considered “emotionally laden”; and people from Southern Europe tend to be more openly expressive physically.
The source of the holding hands appears to have spread out from the early Charismatic Movement in the mid to late 1960’s. Rome has been aware of it from early on, and has apparently chosen, through several editions/revisions of the GIRM to make no statement whatsoever on the matter. Or as I have said before, the silence has been deafening.
Recently my Archbishop came out about the matter, saying he was not in favor of it. Considering that he has no authority to regulate the matter (as any opportunity to regulate a matter is where there is clear indication of alternatives), it was unfortunate, as some will take that as a command.
Archbishop Chaput, when he was in Colorado finally sent a public letter on the matter and said, in so many words, that charity needed to be the rule; people were neither required nor forbidden to hold hands; but they needed to be charitable to the person sitting next to them in the pew.
And personally, I don’t find Archbishop Chaput to be someone I would ever call a progressive. Wise beyond many, however.