If you’re Caucasian, and you’re in a room full of Caucasian people, you will be able to find ways to describe someone in that room without mentioning race. If everyone is of the same race in the room, of course it would be pretty pointless to mention race, but my point is that you can find a way to describe someone’s other features. Just because it’s more convenient for you to use racial descriptors, it doesn’t make it right. The easiest way is not always the best way.
As for your comment about feeling offended, you need to carefully consider a person’s reasons for being offended. I will agree with you that certain things should not offend someone. For example, if I went into a restaurant and became offended that I didn’t get served before others who had come before me simply because I felt myself to be superior, then that would be a silly reason to be offended for something, and most would dismiss my case. But when it comes to racial tension, I don’t think it’s our call (as the traditionally non-marginalized race) to say when minorities should and should not be offended. At best it comes off as ignorant and at worst as racist.
Try to put yourself in the shoes of those who have been treated poorly by a particular race. It would certainly seem frustrating to feel like your race is being seen before your other attributes and achievements. Imagine if Caucasians had been the typically marginalized race and you worked very hard at something–say you worked to become a great musician–it would get tiring to hear people say things like, “Well, he’s a good musician…for a white person.” It would be better to say: He’s a good musician. And so I think this ties back to the comment about racial descriptors. Why use terms that could be potentially harmful when you can find another way? Don’t look at what’s easiest for you, but rather what will create the best situation for others.