Meet the Skeptics: Why Some Doubt Biomedical Models - and What it Takes to Win Them Over
Disentangling the different types of skeptics and what modelers can learn from each.
What are the telltale signs of a modeling talk at a biology conference? Just look for the sighs, shifting, and eye-rolling in the audience, says Donald C. Bolser, PhD, professor of physiological sciences at the University of Florida College of Veterinary Medicine. Bolser, once a skeptic himself, says, “I would see presentations at meetings and wonder: ‘Why would you want to do that?’ I’m an in vivo person, so I never really saw the value of it.” Bolser has since become an ardent fan of modeling, but many of his colleagues remain suspicious.
Though few biologists or physicians will admit to skepticism (we couldn’t get any card-carrying skeptics to go on record for this story), modelers claim that skepticism is near-universal—popping up in grant evaluations, paper reviews, and interactions with experimentalists. “I have encountered a tremendous amount of skepticism for modeling,” says Grace Peng, PhD, a program director at the National Institute of Biomedical Imaging and Bioengineering.
Senior-level people at the NIH may not openly oppose modeling, but they don’t seem to appreciate its true power to change biomedical research, Peng says. Peng chairs IMAG (the Interagency Modeling and Analysis Group), which brings together scientists from 10 governmental agencies who manage programs in biomedical, biological, and behavioral modeling. In 2009, members convened for a two-day conference—the IMAG Futures meeting—that explored reasons for and solutions to skepticism. Threads from that meeting, as well as from interviews with modelers and biologists, form the basis for this story.
Modelers may assume that the problem of skepticism rests solely with experimentalists. But, in fact, modelers play an enabling role—in the way they treat non-modelers, present their results, and even build their models. Thus, overcoming skepticism is as much about changing the culture of modeling as it is about changing the minds of biomedical researchers.
It also turns out that skepticism is heterogeneous. The degree of skepticism varies greatly across different fields of biology and medicine; and skeptics themselves come in many different flavors. Different kinds of skepticism have diverse origins and may present unique obstacles for modelers. This article disentangles the different types of skeptics and suggests what modelers can learn from each.
The Old Guard
Some biologists are not so much skeptical of modeling as dismissive, says Peter Sorger, PhD, professor of systems biology at Harvard Medical School. These are mostly older biologists, who achieved success without modeling, and are stuck in their way of doing things. When you’ve been doing something one way for a long time, change is hard, says Timothy Mitchison, PhD, also a professor of systems biology at Harvard Medical School. “That’s just human nature.”
Unfortunately, there’s little modelers can do to combat this kind of skepticism. “It only goes away when people die or retire; no one ever changes their mind. It’s like why people vote Democrat or Republican; these things go deep,” Mitchison says.
The good news is that the newer generation of biologists is much more open-minded, Sorger says. At classic biology conferences, he says, “I’m barely old enough to get a plenary talk, because everyone is in their seventies”; but at the DREAM modeling competitions, “I’m just this creaky old guy, because the mean age is about 30.” So, the solution to this kind of skepticism is simply to “fill the place with young people,” he says.
We also need to incorporate modeling into the curriculum of future biomedical scientists, Peng says. “I think if modeling is introduced earlier in the pipeline, even starting at the K through 12 stage, modeling will be accepted as standard practice,” she says.
The Math Phobes
Some biologists are open to modeling in principle, but avoid it because they are too intimidated by the math. Modelers don’t help the situation because they tend to be dismissive of people who aren’t quantitatively trained. Modelers have been known to call biologists “dumb,” “idiots,” and “the students who weren’t smart enough to go into math or physics.” With this attitude, it’s not hard to understand why biologists would feel intimidated and shut out.
Some modelers need a dose of humility. They also need to put more time and effort into explaining their models as simply as possible to potential users, Peng says. “The experimentalists say, ‘It would be nice if I could just sit down with the modeler in front of a computer and go through the model,’” she says.
Mitchison (who counts himself among the math-phobic) recalls a math PhD student who did a stint in his lab and was able to make the math comprehensible: “Having someone who can just think that way with their eyes closed, and can explain it to you… once you have that experience of really working with someone, it makes a huge difference.”
Modelers also have to be more willing to make user-friendly tools that don’t require a degree in mathematics, Peng says. “The models need to be as easy to use as TurboTax,” she says.
“If you think about the large number of biologists out there, the notion that everything is going to be done by making people aware of how to do all the underlying mathematics … I think is nonsense, at least in the next generation of individuals and the existing biological investigator pool,” Ron Germain, MD, PhD, told IMAG Futures attendees; Germain is chief of the Laboratory of Systems Biology at the Center for Human Immunology and Inflammation at the NIH. Modelers worry that biologists will abuse models if they don’t understand their inner workings, but Germain points out that people can accurately resize photos in Photoshop without understanding the complex math behind this operation.
Programmers in Germain’s lab have developed software (called Simmune) that allows immunologists to build complex models with all the mathematics handled behind-the-scenes. “People with no computer training can do this with no assistance,” Germain says. “And the response I’ve gotten talking to biologists—instead of the glazed over eyes, ‘oh you’re talking about modeling’—there’s a great deal of enthusiasm.” If modeling is too difficult for biologists to implement, they won’t adopt it even if they think it’s useful, Germain says.
Yoram Vodovotz, PhD, professor of surgery and of immunology at the University of Pittsburgh, encouraged IMAG Futures attendees to explore agent-based models—which are more intuitive than equation-based models—as a way to draw biologists into modeling. “There’s an entire class of simulation platforms that already exist and that is already usable to people in high school without differential equations.”
The “Modelers Are From Mars” Skeptics
Some biologists are skeptical because they feel that modelers are out of touch with the biology. Germain told attendees at IMAG Futures that he has biology colleagues who are initially excited to read a paper in the Journal of Theoretical Biology (or similar journals). But “the first thing they read is ‘for reasons of computational complexity we decided to make the following assumptions…’ And they basically throw out the three or four most important things to the biologist before they go on. At which point the biologist will stop reading.”
Sorger agrees. “There was this notion, maybe 5 to 10 years ago, that basically one was going to take a series of tools that had been developed in another discipline—either computer science, chemical engineering, or control theory—and those things would whole-hog be applied to biology and that would solve the problem,” Sorger says. This was not only an arrogant point of view; it was simply wrong, he says. It also “created a whole series of straw men for the people who are skeptical of modeling to hang onto.”
The solution to this kind of skepticism is for modelers to become immersed in the biology. “I’m very much of the opinion you can come from either direction [biology or modeling] and become an effective computational biologist, but that ultimately you have to be a biologist,” Sorger says. “I don’t think biomedicine is going to be taken over by physicists and computer scientists working half-time.”
Modelers need to develop a deep understanding of the biological data, agrees William J. Heetderks, MD, PhD, director of extramural science programs at the National Institute of Biomedical Imaging and Bioengineering. “If you just take data that was published in the literature and plug it into your model and don’t understand the domain that it was acquired in, you can be badly misled,” Heetderks says.
For modelers to be successful and advance the field, “they must either understand the biology themselves or be joined at the hip with a biologist,” he says.
The “I’m Not Ready” Skeptics
Some biologists think modeling is fine for others, but it’s premature for their biological niche. They may think that their biological problem is too complicated to pin down with a model or that they don’t have enough data yet to build an accurate model.
This type of skepticism stems from misconceptions about the role of modeling, Peng says. “People still think that models are just a tool to maybe fit the data at the end of the experiment,” she says. But models are actually a platform for designing experiments. Models can help biologists organize and archive the data they do have; systematically figure out what new data are needed; and design more efficient and more informative experiments, Peng says. To turn experimentalists around, modelers should interface with them during the planning stages of grants, she says.
For example, Vodovotz told IMAG Futures attendees how models could be used to design better clinical trials. His group ran simulated trials of anti-TNF drugs for treating sepsis and predicted that the compounds would help certain types of people and harm others (with no net benefit). Had pharmaceutical companies used these simulations, they could have targeted the correct group for treatment and avoided treating those who might be harmed.
Models also don’t have to be perfect to be useful. Modelers can drive home this point by highlighting the shortcomings of the alternatives to modeling, David M. Eddy, MD, PhD, told IMAG Futures attendees; Eddy is founder and medical director of Archimedes, a healthcare modeling company located in San Francisco.
“The only alternative in the clinical field is to use clinical judgment or what we’d call the art of medicine. And when you think about all the factors that are involved in clinical decision making, it’s out of the question,” Eddy says. “That’s where models come in—because they’re better than the alternative. They’re not perfect; they’re not as good as clinical trials. But we would argue that they’re better than the alternative.”
The “Show Me the Beef” Skeptics
Some biologists see modeling as esoteric because they can’t point to a concrete example of how biomedical modeling has impacted human health.
Combating this kind of skepticism requires better salesmanship, Peng says. “People who develop models are so into the details of their models that they forget the bigger picture of why their model is so useful and what the model is helping them to do that they couldn’t do otherwise,” Peng says.
It’s often difficult to read a computational paper and figure out what the breakthrough was, Mitchison says. Biologists come from a discovery-oriented tradition; if you’ve been looking for the receptor to a particular hormone, and now you found it, the impact is easy to see, he says. But “often successful modelers are not people who come from this tradition of telling a story. They come from different traditions, and they may not see the value of that,” he says. Modelers need to do a better job of communicating what they learned from a piece of work and what it lets them do that they couldn’t do before, he says.
Modelers can also help convince skeptics by pointing to specific success stories. “What we as program people are always looking for is a killer app or a success story that we can tell our higher-ups: ‘Look, they couldn’t have made this discovery without this model,’” Peng says.
One of the goals of the IMAG Futures meeting was to compile some of these success stories. Marco Viceconti, PhD, professor of biomechanics at the University of Sheffield, in the United Kingdom, presented several examples of how modeling is already being used in patient care. For example, aneurIST (http://www.aneurist.org/) is a program that predicts the rupture of incidentally detected cerebral aneurysms using patient-specific information. And companies such as Phillips and Siemens are integrating similar simulations into their imaging tools for aneurysms, Viceconti says. Modeling also has a practical utility for many companies because the FDA is now allowing in silico simulations as part of the approval process for certain devices.
Many biologists who once were skeptical now count themselves as enthusiastic converts to modeling, including Bolser and Mitchison. Their stories offer lessons for modelers.
Bolser studies the neurological circuits that control airway behaviors such as coughing and swallowing; few investigators in this field use computational modeling. Before his conversion, he thought that models were just about fitting data after an experiment, he says.
But he began working more closely with modelers as part of submitting a grant application. At one point, he and his modeling collaborator ran some simulations and generated a baffling result. Bolser realized that no one had ever done that particular experiment in vivo, so he did it. At first, he didn’t believe the result and suspected a technical error.
But then came the ‘aha’ moment. He realized that the simulation had predicted the outcome exactly. “And it just blew me away. I was just stunned by that. I mean modelers know this, they live by this; but for me it was a long road to go before I understood internally what a model could do. What you’re really aiming at with a computational model is prediction,” Bolser says. “For me, once I got it, I was totally sold on it.”
He’s also shifted from thinking that the airway problem was too complex to model to realizing that the behavior is so complex that “there is no way we could understand it without computational modeling.”
You can’t sell experimentalists on modeling simply by talking about it and showing it at seminars and conferences, Bolser advises. They need hands-on experience, such as a tutorial or workshop, or directly sitting down with a modeler and working on their biological system.
Mitchison says he is also a convert, but it was a more gradual transition. He was a skeptic of both the “I’m not ready” and “math phobe” varieties described above. But he’s slowly come around, helped by people like the mathematician in his lab who was able to explain the math to him clearly. He’s seen a similar gradual transition in his field. For example, the American Society of Cell Biology annual meetings have gone from no one even thinking about modeling 20 years ago to having several lively, well-attended modeling sessions today. “And cell biology is a pretty traditional, conservative field of biology,” he says.
Modeling has even become state of the art for some niche areas of cell biology. In 2008, a couple of modeling papers on yeast cell polarity were published in high-profile journals. Many people disputed the papers (including a new PhD student in his department who went on to do work in the area), Mitchison says. In fact, it angered them: “Here’s a high-profile paper published in Nature on theory and it’s all wrong and I’m annoyed by that, so I’m going to do better.” This stimulated a number of groups to work on the problem and publish better models. It’s been a healthy progression and has resulted in many hard-core biologists adopting theory collaborators for the first time, Mitchison says. So getting theory papers on hot biological topics in high-profile journals can spark interest in modeling, he says.
The “Healthy” Skeptics
Some biomedical researchers are proponents of modeling in theory, but are skeptical about specific tools and approaches. This is a legitimate and healthy form of skepticism, Sorger says. “The skeptics rightly point out that even in the hands of people being fairly careful, the promise has run way ahead of the actual tools and knowledge needed to apply them correctly. Therefore, there are probably quite a lot of errors out there.” (See “Error: What Biomedical Computing Can Learn from Its Mistakes,” in Biomedical Computation Review online.)
The solution in this case is not to challenge the skeptics, but to fix the problems that drive their skepticism. This will require more investment into the methodology of modeling. “This general assumption that the methodology is good enough I think is really wrong,” Sorger says.
We need better ways of validating models, Eddy told IMAG Futures attendees. Validation means different things to different people and some of what is passed off as validation is garbage. “I can certainly understand why potential users of models are frustrated and confused. And they don’t know what they can believe and can’t believe,” Eddy says. Modelers need to devise a recognized standard so that users know that they can trust a model when it has the seal of validation for a particular application, he says.
Competitions and benchmark problems can demonstrate the reproducibility of models (or reveal errors in need of fixing). Standards in the reporting of models and simulations (such as MIRIAM and MIASE, Minimal Information Required In the Annotation of Models and Minimum Information About a Simulation Experiment, respectively) also help improve the reproducibility and testability of models, participants at IMAG Futures noted.
To win over healthy skeptics, modelers also need to be more humble in how they present models. In molecular and cell biology, researchers typically draw cartoon models with arrows and boxes; and it’s understood that these are just working hypotheses, not to be taken too seriously, Mitchison says. Computational models are also just provisional, but they often aren’t presented this way. “There’s a way that computational stuff is written up that sort of implies a rigor and absolute truth that experimentalists who don’t use quantitative methods have deliberately shied away from,” Mitchison says. This kind of overconfidence can drive skepticism, because experimentalists know that biology always involves hidden assumptions.
Modelers need to be more upfront about the limitations and potential pitfalls of their models and to make these issues more understandable for non-mathematicians, says Brian K. Kobilka, MD, professor of molecular and cellular physiology and medicine at Stanford University.
The “Insider” Skeptics
Some of biomedical modeling’s biggest skeptics are actually modeling insiders. They may challenge specific models and applications, or even whole paradigms of how modeling is done.
When it comes to reviewing grants with a modeling component, engineers are often the worst critics, Peng says. Engineers tend to have a critical mindset; and they can actually do a disservice to modeling by being too nitpicky in their reviews, says Peng (who is herself an engineer). We also face the “grumpy Russian mathematician problem,” Sorger says. He says pure mathematicians tend to give his papers unfavorable reviews because it’s “1950s, engineering mathematics” rather than cutting-edge modern math.
Some modeling insiders go even further, saying that almost all biomedical modeling is done incorrectly. Awareness of these skeptics’ point of view is important, because their critiques may ultimately explain some of the intuitive discomfort that biologists feel toward models.
For example, James Bower, PhD, professor of computational neurobiology at the University of Texas Health Science Center in San Antonio, says, “I would argue that, at present, the majority of mathematically based models in biology are not in fact useful in advancing the field.” Most biomedical modelers are building models simply to explain or convince others of what they already believe; but the purpose of modeling should be to discover new features of a system. “If you don’t know anything more about the system after you build the model than you did before, it is of little use,” Bower says. “We are just endlessly misled by what are basically Ptolemaic or religious models that are designed to enforce a particular doctrine.”
He advocates the use of anatomically and structurally realistic models that don’t have built-in assumptions about function, such as embodied by the GENESIS simulation toolkit for neuronal modeling (http://www.genesis-sim.org/GENESIS/). He also promotes the notion of community models where everyone uses and freely shares the same base models. “There is a way forward and it’s slowly starting to happen,” Bower says.
John C. Criscione, MD, PhD, associate professor of biomedical engineering at Texas A&M, is another advocate of sweeping change in biomedical modeling. In his field of multiscale tissue modeling, he says he has shown that the current framework of modeling yields an infinity of solutions. Modelers are essentially trying to solve for three variables with two equations, he says. “If a freshman algebra student did this, I would flunk them,” he says. “Everybody is going, ‘but it fits the data.’ Well, yeah, the sun moving around the earth fits the observational data too,” he says. He says we need to get back to basics and figure out models that solve the simplest problems—like a perfectly homogenous elastic cylinder.
“We absolutely need modeling. I’m not saying we shouldn’t do multiscale modeling. I love it. It’s great stuff,” Criscione says. “What we don’t need is to spend money doing modeling where we’ll never get a right solution.”
Interestingly, both Bower and Criscione are increasingly pessimistic about convincing their colleagues. Both are therefore independently focused on exposing the next generation of engineers to modeling technology. Bower’s efforts are based in Whyville.net, which he founded as the first simulation-based educational virtual world 13 years ago, which now has more than 7.2 million subscribers worldwide.
The Demise of Skepticism
Skepticism may be pervasive, but it’s also on the decline. If researchers talked about modeling and mathematics at biology meetings 30 or 40 years ago, “we were going to be lynched almost,” Eddy told IMAG Futures attendees. Acceptance of modeling will continue to grow, because modeling in biomedicine is inevitable. In the future, everyone will use models and appreciate the use of models, Peng says. But what’s at stake is how quickly this transition will occur, Peng says. “How fast we get there is what I’m trying to address.”