If someone is having a bad effect on you - being abusive, consistently making you feel bad, or if they’re just a bad influence on you, then I would say that’s the time to walk away. You don’t need to do it in an unkind way, of course - but actually sometimes by rejecting someone you can help them to question and change their behaviour (as it stops getting them the results they want).
Like the previous poster, I find it difficult to advise any better than this on such little info! You don’t mention if it’s a friend, family member or someone you’re romantically involved with or any of the specifics. But I hope what I’ve said may be helpful.