The M. Jackson Group Update – August 2015 –  Current Challenges to the Scientific Enterprise

This month’s article is again from Ken Pope’s listserv. Science, including the science of Psychology is facing a number of challenges.  These include key results that cannot be reproduced (probability is like that, particularly if you only publish supportive findings), findings “shaped” (okay, made up) by the need to “publish or perish” and the associated financial incentives, and the lack of an appropriate place in the theatre of publication for the disappointing but nonetheless informative results when hypotheses are not supported. The following article is relevant to these issues:

The *Chronicle of Higher Education* includes an article: “Amid a Sea of False Findings, the NIH Tries Reform; Science needs to get its house in order, says Francis Collins, director of the NIH” by Paul Voosen.

Here are some excerpts:

[begin excerpts]

How do you change an entire scientific culture?

It may sound grandiose, but that is the loaded question now facing the National Institutes of Health, the federal agency that oversees and finances U.S. biomedical research.

While the public remains relatively unaware of the problem, it is now a truism in the scientific establishment that many preclinical biomedical studies, when subjected to additional scrutiny, turn out to be false.

Many researchers believe that if scientists set out to reproduce preclinical work published over the past decade, a majority would fail.


The NIH, if it was at first reluctant to consider the problem, is now taking it seriously. Just over a year ago, the agency’s director, Francis S. Collins, and his chief deputy, Lawrence A. Tabak, announced actions the agency would take to improve the research it finances.

Science needs to get its house in order, Dr. Collins said in a recent interview with The Chronicle.

“We can’t afford to waste resources and produce nonreproducible conclusions if there’s a better way,” he said.

The problem may be most visible in preclinical biomedicine and fields like psychology, but it’s by no means limited to them, Dr. Collins added.

“No matter what your science is, there’s a risk here. If experimental methods are not well detailed and well planned, then stuff can happen.”

Science today is riven with perverse incentives:

*Researchers judge one another not by the quality of their science — who has time to read all that? — but by the pedigree of their journal publications.

*High-profile journals pursue flashy results, many of which won’t pan out on further scrutiny.

*Universities reward researchers on those publication records.

*Financing agencies, reliant on peer review, direct their grant money back toward those same winners.

*Graduate students, dependent on their advisers and neglected by their universities, receive minimal, ad hoc training on proper experimental design, believing the system of rewards is how it always has been and how it always will be.

But many senior researchers will oppose reform, said Arturo Casadevall, a professor at the Albert Einstein College of Medicine at Yeshiva University.

“Scientists themselves are playing this game because once they succeed, the rewards are so great they basically force everyone to do it.”

The tight financing of the past decade has only made the problem worse, said Ferric C. Fang, a microbiology professor at the University of Washington who, like Dr. Casadevall, has called for changes in the system.

“We’re ending up with an ecosystem that’s increasingly populated by predators,” he said.

Some researchers fear that “skeptics” who dispute valid scientific work on a host of hot-button issues will seize on the reproducibility crisis as a political weapon.

“This is a very serious problem,” said C. Glenn Begley, chief scientific officer of Tetra-Logic Pharmaceuticals, who, several years ago, published a prominent paper on how his former company, Amgen, had found it impossible to replicate more than 90 percent of potential cancer drugs.

“It really threatens the general feeling among the community in terms of how important science is,” he said.

The NIH’s response is wide- ranging.

Its institutes are revising how they review grants, requiring far more data on experimental design, including validation of past findings that studies purport to build upon.

The agency is financing new programs to teach first-year graduate students how to design experiments.

It is pressing journals to raise their review standards.


Those are all worthy steps, Dr. Casadevall said, and he applauds how quickly the NIH, once it understood the scale of the problem, has responded.


“My concern is that the problem here is cultural,” he said, “and the fixes are procedural.”


If the NIH could be said to have an internal whistle-blower on the problems with preclinical research, Shai D. Silberberg would be a candidate for the role.

A decade ago, Mr. Silberberg, a biophysicist, became a program director at the National Institute of Neurological Disorders and Stroke, and he was shocked to see what was considered high-quality science….

Many experiments supporting proposed clinical investigations used low sample sizes and were prone to bias.

As it happened, neurology researchers were ahead of the curve in pointing out those problems.

Scientists in Australia and Scotland had begun documenting flaws in animal studies that had seemed to support promising, but failed, stroke treatments. A similar effort found parallel problems in spinal-cord research.

And ALS researchers, seeking a cure for Lou Gehrig’s disease, went back and reproduced studies on more than 70 promising drugs.

They found no real effects.

“Zero of those were replicable,” Dr. Collins said.


Armed with such data, Mr. Silberberg made a case for change.

He won over the institute’s director, Story C. Landis, who put her grant reviewers on notice that higher standards were expected.

But it was clear to Ms. Landis that the problem wasn’t limited to neurology; all of NIH needed to change. T

hat culminated in a 2012 meeting with editors from journals like Nature, Science, and Cell; financing agencies; and prominent reviewers.

The first day, everyone blamed each other.

The second day, they started talking solutions.

Though replication has now become an NIH-wide concern, Mr. Silberberg remains enmeshed in the agency’s efforts to improve transparency in detailing how studies are conducted.

He gave 24 talks about the effort last year.

“I view this like being a Johnny Appleseed,” Mr. Silberberg said.

“You’ve got to plant the seeds and let them grow.”


“As long as universities think that the way for investigators to get money is to publish in Nature and Science, then that’s what they’re going to look for in investigators,” Mr. Silberberg said.

“They want that money.”

Dr. Begley would like to see the NIH go further, requiring universities to ensure a minimum research standard among their employees.

Even if that step cut in half the amount of research universities produced, if it increased the science’s quality by a commensurate amount, Dr. Begley would take it.

The NIH has also pushed scientific journals to raise their review standards for life-science studies.

More than 100 journals have agreed to elevated principles, many of which stem from the neurology institute’s recommendations.

Most of the journals are instituting checklists that will require submissions to run down and justify the decisions made in designing the study.

Checklists can prevent all kinds of human-scale problems, Dr. Casadevall said.

“The next time you take a flight, look left as you’re boarding the plane. The pilots are doing a checklist.”


But perhaps the agency’s most radical step is buried in its education proposals. Not only does it want training on experimental methods; it wants lessons on the sociology of science. Why do promotion and tenure work like they do? Are there alternatives? Seeds must be planted. Students need to know that it doesn’t have to be this way.

“We want them questioning, wanting to fix what’s broken in the system,” Mr. Lorsch said. “If they don’t hear what the problems are, and don’t hear their professors reflecting on them, it’s going to be much harder to change things.”

[end excerpts]

The article is online at:

Ken Pope


“If you want to truly understand something, try to change it.”
–Social psychologist Kurt Lewin (1890-1947)

Take care,



John Pullyblank, Ph.D., R.Psych.


Managing Partner