Do you invite them to come with you? It would be a bit harder for them to think you’re looking for an excuse to get away from them if you were always inviting them to come along. For instance, if you say, “Would anyone like to come to Mass with me this weekend? We could go at 5:30 on Saturday night, at 8:30 or 10:30 on Sunday morning, or at 7 pm on Sunday night.” If they decline the invitation, then you ask if one of those times would work better for them as the time that you go by yourself, freeing up the rest of the time to spend with them. (It would be an act of charity to make it a point to attend Mass at the time that is best for the rest of your family, even if they are not going with you.)