Whenever there are blessings bestowed in the context of a mass, they are not specifically
liturgical rites, but are permitted for those occasions such as you mentioned, OP. As such, there are no rubrics forbidding the faithful from adding their prayer to the priest’s, and I think it is a beautiful gesture to the ones who are being prayed for. Last Sunday, we had a couple renewing their vows on their 50th wedding anniversary and Father gave them a special blessing. True, we did not extend hands, but I have seen this practice on other similar occasions.
Mark 16:17, “and believers shall lay hands upon the sick and they will get will get well.” There is a tranfer of spiritual power/anointing/blessing that is not present by simply saying the words without the laying on of hands, or extending them. If so, Jesus would not have advocated using physical contact.
I would not be so quick to dismiss the request of your pastor, but remember - God sees the heart, If you make the gesture with ill will, there will be no blessing. Maybe it’s best to adjust your thoughts.