Most people won’t agree but I think you need to do what you feel is right for you and your family and what you feel god wants for both of you as for you are one in marriage. I think a lot of people on here get so caught up in what they feel is an important sin they forget another. You are the only one that has to live inside your walls and no one should judge you for what you decide.
Read Matthew 19:3-11 and take that into consideration as well.
They asked, “Is it lawful to divorce a wife for any cause?” 4 He answered, “Have you not read that from the beginning the Creator made them male and female, 5 and said, ‘For this reason a man will leave his father and mother and will be united with his wife, and the two will become one flesh’? 6 So they are no longer two, but one flesh. Therefore what God has joined together, let no one separate.” 7 They said to him, “Why then did Moses command us to give a certificate of dismissal and to divorce her?” 8 Jesus said to them, “Moses permitted you to divorce your wives because of your hard hearts, but from the beginning it was not this way. 9 Now I say to you that whoever divorces his wife, except for immorality, and marries another commits adultery.” 10 The disciples said to him, “If this is the case of a husband with a wife, it is better not to marry!” 11 He said to them, “Not everyone can accept this statement, except those to whom it has been given.