Sexual arousal in itself is morally neutral - it is a natural gift and we can’t always avoid being aroused by things if they occur naturally, or despite ourselves.
To be perfectly honest, if I had a job that required me to wear a uniform and if that uniform made me feel ‘sexy’, then provided I didn’t actually DO anything immoral, then I would simply take it as an added bonus that the job I had allowed me to wear something that made me feel good about myself and made me feel attractive.
Of course if the uniform you are required to wear presents you with intolerable temptations no matter where you are and is actually causing you distress, then you should probably either seek another job that relieves you of the need to wear that uniform, or you should seek counselling about managing unwanted desires.
To my mind though, I would suggest you’re probably just being a little too scrupulous and too hard on yourself. Don’t overthink it. I’d venture to suggest that after some time has gone by and the job has got boring or repetitive, the uniform will cease to hold much in the way of attraction as it’ll become connected less with the ‘fetish’ and more with mundanity. That will take a lot of the frisson away.