This is the way I look at it.
I would not do it simply because he is a coworker (and a fairly new one, it sounds like) who wishes to befriend you. That relationship could lead to feelings of obligation to attend more than once or a deeper friendship than it turns out you want (sometimes a friendly coworker can become your buddy at work when you really don’t want that). There may or may not be some faith based issues that are talked about at work between the two of you that cause friction. Also, I think you have some good insticts about attending a service without your husband, since this seems to be a budding friendship. Not that the man has alterior motives, but it is a situation best not to get into.
My advice if you want to be friendly with your coworker would be to keep it low key and take it slow, simply because you are at work to do your job and, while it is fine and good to keep a friendly attitude at work, you are just getting to know your coworkers and your job so making friends with one outside of the workplace complicates things. That is especially true if the person is a preacher.
In the past I was invited by a couple of different coworkers to attend their church. I never did, mostly because I just did not want to or see the point. I was friends with one of the ladies and the other used to give me rides home from time to time. I felt funny about being asked to come and see what their churches were like. I guess I felt fairly open to one, but the other, being Pentecostal, I knew would not suit me. I turned them down politely, though. They did keep asking and I would assume your coworker may. I finally just said that I was really happy with “my church” and one of them told me, “Well, you know you have to go where you get fed.” Meaning where my spirit gets fed. I assumed she meant her church, but since she was such a sweet person I decided to overlook that and think of it as I was doing what she suggested by sticking with the Catholic Church.
I think people mean well when they make these offers to coworkers to attend their church, but to take up a coworker on such any offer is crossing a boundry that likely should not be crossed unless you have worked with them for years and are truly friends (including families are friendly).