I am not really sure what kinds of responses you expect to hear from people in your Church community (or here on Catholic Answers, for that matter) after you tell them this. If you’d rather not hear that what you are doing is wrong, then you can simply stop broadcasting it to people.
If you have chosen to be open about your infertility with people at church, coworkers, neighbors, and even random cashiers, you are opening yourself up to comments of all kinds. This is true of anything private you choose to share with people.