I am unaware of any Church document that specifically proscribes clapping, but the Church’s liturgical documents are usually ordered toward directing what is to be done, not what is not to be done. In the absence of specific documentation, prudential judgment should be used to determine if clapping is appropriate.
Traditional church etiquette, at least in Western societies, frowns on clapping as a form of appreciation for the efforts of musicians, homilists, etc., because general social norms have been that all presentations in church are offered up to God as worship, not to the congregation for their personal critique. Generally speaking, then, social protocol encourages people not to applaud.
Your friend, however, is confusing applause with rhythmic clapping, which is usually done as a supplement to the music and not as audience appreciation for a performance. If a particular liturgical hymn can be appropriately supplemented with rhythmic clapping, prudential judgment would not caution against that. Whether such a hymn is appropriate for Mass is something that the pastor should discuss with the music director.
To my knowledge, there is only one time when applause – as distinguished from rhythmic clapping – is indicated by the liturgical rubrics. During an ordination ceremony, the bishop will usually ask the congregation to applaud as a sign of their support for the newly ordained priests.
