You should be troubled by “chit-chat” among people who should be in preparation for the Mass. They are disturbing you, and me, and themselves by their noise. Your pastor should be concerned about this, because his responsibility for our souls, and their souls, is grave. He is supposed to be their father - and your father - and my father - and he should work to make his offering of the Mass (and our offering with him of the Mass) effective for us all, to help us all toward our common vocation to holiness.
I have tried to talk with pastors about this, and some grieve about this more than we do. One holy priest put up signs in the narthex - concerning what is righteous to do with the time in preparation for Holy Mass, and proper dress for it as well - and I’m told he got so much push-back and grief because of it, with complaints going even to the Bishop, that the priest caved. Very, very sad.
Pray. Pray! Pray for our Church, so much lukewarmness, so much worldliness, so much irreverence! God help us.