It is fairly simple. Their house, their rules. You have lived with them long enough that if they are not saying “No, you may not go” but are pressuring you with the “We’ll be hurt if you choose to ignore our request” then you are bing manipulated to some degree. As long as you live there, this is going to be the issue, so go back and read my second sentence.
Is it unfair? Probably. That is the cost of living there.