No trinkets. Get a wall plaque he can hang in his office, one that recognizes his tenure and service to the church community he is leaving. If you look him up 15 years from now, you will probably find that plaque hanging near his desk.
One pastor of ours, upon being reassigned, told the congregation that yes, he was taking Candles with him. This upset one sweet lady tremendously - she asked how he could consider taking the candles that adorn the altar??? Then Father explained to her what most of the other parishioners already knew: Candles had four feet, a waggy tail, barked, and spent most of his time in the fenced-in yard behind the parsonage.