Well I guess everyone has their opinion. My name is Sarah, and I always wanted a nickname. I like the versatility of sort of changing it based on the situation - more formal or less formal. So all my kids have nicknames that they are generally referred by unless it is a more formal situation when they are called their full name.
However, with that being said I would never refer to someone as Jim if he introduced himself as James. You’re right - that is rude. It is up to the individual to decide how they want to be referred to, and I think it’s nice to have a different variation.
And maybe Seb has to grow on you a bit before you like it? I was really unsure about Sebastian when he was born. We actually almost changed it to Xavier on the second day before we had filled out the birth papers. However, he is now 2 and it has become one of my favorite names. He is called Sebastian a lot, but goes by Seb most of the time. He is also affectionately called Sebbie by close family.