Logically, there is no alternative other than to simply state that you do not wish to go. To say anything else, or to try and manufacture or imply a reason for not going that is not true is deceptive and therefore sinful. To say that you have another commitment or that the timing is inconvenient are both lies if the true reason you do not wish to go is, say, that you dislike the company of the individual you would be going with.
The correct response to an invitation you would rather not observe will always be: “I will not attend” (it is pointless and deceptive to add that you are sorry since, in preferring not to share that individual’s company, you would not be sorry at all).
If pressed for a reason why you will not be going and you do not feel comfortable in cataloguing the many reasons why you dislike the individual’s company, the appropriate response will be “it is none of your business,” since no one has the absolute right to your impressions of them.