I think that people who speak of the Sign of Peace as a ‘silly minute or distraction’ are speaking of the current 50 year old Sign of Peace as done in the OF in the US (mostly) which in their experience has been abused to the point that it has (for them, in their churches, at some point in those 50 years) become ‘silly’ and ‘distracting’.
In 50 years a rather sizable anecdotal cadre of Catholics have experienced, often over decades, a sign of peace which involves much noise, laughter, movement, and even giddiness among many —they have seen Father and his servers come down from the altar where Father has just consecrated the Eucharist, has presided over us while we said the Our Father, and then come down, often going throughout the whole church, reaching across people, laughing, joking, talking, hugging, kissing, and then after often more than 5 minutes of this, while people are positively bullied into ‘shaking’ and pointedly snubbed if they do not—Father and the servers go back up onto the altar and immediately start saying, “Lamb of God’, and then getting ready for communion.
While people are still talking through the Lamb of God, still making hurried comments to finish up the important things they were discussing at the sign of peace, like how little Harvey and Harriet are doing, or whether so-and-so is going to do such and such.
I do not think that the SOP should be done away with. But I do believe that the placement in the Mass, while it was well-intentioned, is not and has not worked to provide the average person in the pew with the message of what the sign of peace IS.
Whereas if the sign of peace took place at another point: Perhaps at the very start of Mass, just before the people all say a penitential rite and where the symbolism of making peace with our neighbor would be quite clear; perhaps after the homily and just prior to the Creed, where again, the symbolism of our offering peace before we all stand together to state what we believe—surely these points at Mass would be far better suited to our offering peace to our neighbors.