The people leading the charge appear to be conservatives:
In an interview with NBC News’ Chris Jansing on Meet The Press Sunday, Carson blamed the “secular progressive movement” for many of the stories reporting on claims made in Gifted Hands. However, the most prominent articles calling his stories into question have come from Politico and the Wall Street Journal, both of which are owned by conservatives: Robert Allbritton and Rupert Murdoch.
Carson said the news stories are coming out now because he is considered a threat to “the progressive, the secular progressive movement in this country.”
“I’m a very big threat because you know they can look at the polling data, they can tell that I’m the candidate who’s most likely to beat Hillary Clinton,” Carson told Jansing.
thinkprogress.org/politics/2015/11/08/3720350/carson-blames-secular-progressive-movement/
Edit. Even O’Reilly weighs in;
mediamatters.org/video/2015/11/06/oreilly-slams-ben-carsons-dumb-statements/206694