Your pastor seems to have said it “could be” a mortal sin, not that it is always definitively so. Some elections are so clear-cut good versus evil that to abstain would be a serious matter. Others involve a collection of candidates, one worse than the next, who are all unsuitable for elective office. I don’t see how it could be sinful to abstain in this case.
In my state, we have basically one party rule. People registered with the minority party vote for candidates in the primary elections who will have zero chance of being elected. In the general election, we can vote for the minority candidates, and we do, but until things change, it still makes no difference. So is it sinful to omit such a meaningless gesture?
I know, I know, if we all don’t vote, nothing will ever change, and that’s why we do vote. But if something came up that made voting a hardship, I wouldn’t be trying too hard to overcome it unless and until my vote is meaningful.
Betsy