That’s definitely sometimes the case. Unfortunately your example is really poor. It’s sometimes helpful to distinguish between ἀνθρωπος (= man in the old fashioned sense, i.e. human being/s, people) and ἀνηρ (= man/husband in the sex-specific sense, i.e. a male). For the same reason, one often ought to distinguish clearly between אָדָם (Adam = human) in Hebrew and אִישׁ (iysh = man, husband). “Fishers of men” sounds good, but there are good scholarly reasons to render it “fishers of people/fishers of human beings”.
I’d be tempted to compromise and have “fishers of mankind”, but that avoids gender-neutral language at the expense of strict grammatical accuracy; you lose the sense of a group of individual human beings implied by the genitive plural.