Most scientists 'can't replicate studies by their peers'

  • Thread starter Thread starter ucfengr
  • Start date Start date
Status
Not open for further replies.
“It’s about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about.”

She says it’s about the funding bodies that want to secure the biggest bang for their bucks, the peer review journals that vie to publish the most exciting breakthroughs, the institutes and universities that measure success in grants won and papers published and the ambition of the researchers themselves."

Bullseye.
 
It sounds scary and sensational, but this is science doing what science is supposed to do. Checking and re-checking experiments, engaging in healthy self-criticism, and ultimately increasing knowledge.

All good.
 
I remember reading there used to be a medical journal that took pride is not having studies peer reviewed. With all the problems that system can create I’m not surprised. As some have said, having peer review of work is similar to letting Ford and Toyota decide if GM can develop a new car device.

I remember reading a few former Medical journal editors being greatly concerned over the poor quality and honesty of many studies being produced today. Sadly over the years little has been done to improve the situation.

The former editors had this to say ~

The Augean stables

drmalcolmkendrick.org/2015/08/27/the-augean-stables/

excerpt:

…Well I shall now quote Dr Marcia Angell, Dr Richard Horton and Dr Richard Smith. Editors of, respectively, the New England Journal of Medicine, the Lancet and the British Medical Journal. The three highest impact factor journals in medical research.

‘It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgement of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as editor of the New England Journal of Medicine.’ Marcia Angell.

‘The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.’ Richard Horton

‘The poor quality of medical research is widely acknowledged, yet disturbingly the leaders of the medical profession seen only minimally concerned about the problems and make no apparent efforts to find a solution.’ Richard Smith

Who, in a position of power, will finally wake up and realise that the vast database of medical research stinks of bias and manipulation. Who can we call upon to take up the gigantic and painful task of clearing out the Augean stables?..
 
The basic premise of the method is to question and test. When politics gets in the fold you are told not to question or test!

Science is not black and white. People who make it thier god try to shape it into a final authority. In reality, it is messy, and hard to understand.
It’s hardly settled. In any field.
 
Tim_D said:
“It’s about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about.”

She says it’s about the funding bodies that want to secure the biggest bang for their bucks, the peer review journals that vie to publish the most exciting breakthroughs, the institutes and universities that measure success in grants won and papers published and the ambition of the researchers themselves."

Bullseye.
Click to expand...
I think that sounds right. Perhaps it’s human nature or at least our competitive culture that scientists would like to the first with their theory, method, and/or evidence to prove something new.

It is so laughable when people criticize climate scientists as being in cahoots or some large conspiracy, when (as they explain it) they are highly competitive, vying with each other, honed in on any mistakes they can point out or ideas they can dispute in other scientists’ studies.

So the upshot is there are now many studies supporting CC from different & new angles. And that makes the body of their works “robust.”
 
forums.catholic-questions.org/showthread.php?t=1039277

There was a thread started not too long ago about this very subject, and that a foundation, The Arnold Foundation, was started in order to shed light on these questionable scientific studies that are not peer reviewed and cannot be replicated because the methodology is not published in the study.
 
lynnvinc said:
It is so laughable when people criticize climate scientists as being in cahoots or some large conspiracy, when (as they explain it) they are highly competitive, vying with each other, honed in on any mistakes they can point out or ideas they can dispute in other scientists’ studies.
Click to expand...
Your post stands in stark contrast to what was presented in the article, that "“It’s about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about.”

It is all too apparent that the cult of climate change largely depends on “impact over substance.” Leonardo di Caprio is so certain of the “proven” science of climate change and the supposed disastrous effects that it will have that he spends his time flying on his personal jet all over the world to warn us.
 
After reading the article, it also occurred to me that some science is skewed and even fraudulent – like finding some medicine is more effective and with less bad side effects than is actually the case.

In those cases one would not be able to replicate the study…unless they used the same devious methods (like in one case I read about switching the control group for the experimental group stats). If I find the source (it was over 20 yrs ago I think) I’ll let you know.
 
Roselle said:
I remember reading a few former Medical journal editors being greatly concerned over the poor quality and honesty of many studies being produced today. Sadly over the years little has been done to improve the situation.
Click to expand...
ncbi.nlm.nih.gov/pubmed/12775614

Pharmaceutical industry sponsorship and research outcome and quality: systematic review.

Abstract

OBJECTIVE:
To investigate whether funding of drug studies by the pharmaceutical industry is associated with outcomes that are favourable to the funder and whether the methods of trials funded by pharmaceutical companies differ from the methods in trials with other sources of support.
METHODS:
Medline (January 1966 to December 2002) and Embase (January 1980 to December 2002) searches were supplemented with material identified in the references and in the authors’ personal files. Data were independently abstracted by three of the authors and disagreements were resolved by consensus.
RESULTS:
30 studies were included. Research funded by drug companies was less likely to be published than research funded by other sources. Studies sponsored by pharmaceutical companies were more likely to have outcomes favouring the sponsor than were studies with other sponsors (odds ratio 4.05; 95% confidence interval 2.98 to 5.51; 18 comparisons). None of the 13 studies that analysed methods reported that studies funded by industry was of poorer quality.
CONCLUSION:
Systematic bias favours products which are made by the company funding the research. Explanations include the selection of an inappropriate comparator to the product being investigated and publication bias.
 
Roselle said:
I remember reading a few former Medical journal editors being greatly concerned over the poor quality and honesty of many studies being produced today. Sadly over the years little has been done to improve the situation.
Click to expand...
ncbi.nlm.nih.gov/pubmed/16924299

Ethical dilemmas in scientific publication: pitfalls and solutions for editors.

Abstract

Editors of scientific journals need to be conversant with the mechanisms by which scientific misconduct is amplified by publication practices. This paper provides definitions, ways to document the extent of the problem, and examples of editorial attempts to counter fraud. Fabrication, falsification, duplication, ghost authorship, gift authorship, lack of ethics approval, non-disclosure, ‘salami’ publication, conflicts of interest, auto-citation, duplicate submission, duplicate publications, and plagiarism are common problems. Editorial misconduct includes failure to observe due process, undue delay in reaching decisions and communicating these to authors, inappropriate review procedures, and confounding a journal’s content with its advertising or promotional potential. Editors also can be admonished by their peers for failure to investigate suspected misconduct, failure to retract when indicated, and failure to abide voluntarily by the six main sources of relevant international guidelines on research, its reporting and editorial practice. Editors are in a good position to promulgate reasonable standards of practice, and can start by using consensus guidelines on publication ethics to state explicitly how their journals function. Reviewers, editors, authors and readers all then have a better chance to understand, and abide by, the rules of publishing.
 
Here’s something that addresses the issues raised in the OP:

realclimate.org/index.php/archives/2017/02/someone-c-a-r-e-s/#more-20143
Someone C.A.R.E.S.
Click to expand...
Filed under: Climate Science Scientific practice — gavin @ 25 February 2017
Click to expand...
Do we need a new venue for post-publication comments and replications?
Click to expand...
Social media is full of commentary (of varying degrees of seriousness) on the supposed replication crisis in science. Whether this is really a crisis, or just what is to be expected at the cutting edge is unclear (and may well depend on the topic and field). But one thing that is clear from all the discussion is that it’s much too hard to publish replications, or even non-replications, in the literature. Often these efforts have to be part of a new paper that has to make its own independent claim to novelty before it can get in the door and that means that most attempted replications don’t get published at all…
Click to expand...
I have thought for a long time that an independent journal venue for comments would be a good idea, but a tweet by Katharine Hayhoe last weekend made me realize that the replication issue might be well served by a similar approach. So, here’s a proposal for a new journal.
Click to expand...
Commentary And Replication in Earth Science (C.A.R.E.S.)…
Click to expand...
Read on for what he proposes.
 
This is a coffee table mag article giving the author’s summary of a survey. The author of this article claims 70% of scientists said their results were not reproducible.

““Science is facing a “reproducibility crisis” where more than two-thirds of researchers have tried and failed to reproduce another scientist’s experiments, research suggests.””

Realoy Tom, back it up
So what exactly do we have here?
  1. A survey making a conclusion about reproducibility
  2. link to original study is where?
  3. Where is the reference for any fact or figure?
  4. How many scientists were surveyed?
  5. What area were they in?
  6. What era?
  7. Why did they believe their results werent reproducible?
Fake science news.

So its fine to just publish this without evidence to back it up, or details?
““According to a survey published in the journal Nature last summer, more than 70% of researchers have tried and failed to reproduce another scientist’s experiments.””

I am starting to agree with the Trump about fake news
 
ucfengr said:
This should cause some reflection among “the science is settled” crowd, but I predict it won’t.
Click to expand...
The “reproducibility crisis” that has been getting press exists, but as people involved understand it’s in the social sciences and human physiology/pharmacology realms that the problem mostly exists. Science as a whole hasn’t been experiencing a crisis but certain areas that rely on self-reported data or papers based on p-hacking are where the studies become impossible to reproduce.

If, for example, your “science is settled” remark refers to anthropogenic climate change it’s misinformed. The studies and models that confirm and predict ACC are valid and eminently reproducible. If it refers to something like the constantly changing diet advice we get from the mainstream media it’s accurate.
 
Status
Not open for further replies.
Back
Top