A ‘sign’ doesn’t have to be physical matter - a sign is simply anything that stands for something else. And any words - whether spoken or written - always stand for (or are a ‘sign’ of) the reality they signify.
And no, not all the sacraments need use physical matter. The sacrament of marriage, for example, is effected by the couple exchanging vows. The traditional exchange of rings is not necessary, and there are no other physical signs used.
So the priest’s WORDS of absolution are not the reality itself, but rather they ARE the sign. Imagine, for example, that you deliberately or knowingly lie in the confessional, deliberately omit a mortal sin or have absolutely no genuine wish to turn away from your sins, you won’t receive the reality of the forgiveness regardless of the words being spoken.