There have been several really nice essays published by José Ricón (Nintil) and Alexey Guzey recently looking at ‘meta-science’ and different ways that science is done well or not so well from a birds eye perspective. José publilshed a series of essays titled “Fund People not Projects” and just published the third installment in what appears to be at least a 4 part series (essay 1, essay 2, essay 3). Alexey published an essay titled “How Life Sciences Work: Findings of a Year-Long Investigation”. There are a ton of interesting ideas here, ranging from the Matthew effect, the impact of HHMI funding versus NIH R01s, the fact that almost all biologists are solo-founders, and the Ortega Hypothesis.
As a young scientist just about to begin exploration into different fields and careers within sciences, these essays have sparked an interest into where exactly in the scientific pipeline someone like me can make the most impact. I’m interested in translational technologies; diagnostics and therapeutics, and to some extent also technologies that enable them, such as sequencing, CRISPR, microscopy, nanotechnology, etc. My personal mission aligns with that of the Chan-Zuckerberg initiative: to cure, prevent, or manage all diseases by the end of the century. The upshot of this is that I want to do things fast and with optimal productivity (as defined as progress towards the above goal) and unfortunately (or fortunately), I think that there is a lot of room for improvement within the current biomedical ecosystem. Here are a couple thoughts about how I would improve the biomedical innovation system.
There is a biennial protein-structure prediction challenge called CASP which was founded in 1994. I’m sure everyone has heard the news by now, but a DeepMind program called AlphaFold performed so well in the competition that protein structure prediction is considered by some to be a solved problem. This is a problem that scientists have been working on for decades, but DeepMind only began participation in the competition in 2018. How is it that a team was able not only to win the competition, but also solve the core biomedical problem? I think a good deal of it has to do with the context. Competitions breed excellence. They provide incentive structures that simply are not there in a traditional academic setting.
On the other hand, it could be argued that this is a very special type of problem amenable to competition. Computational areas have relatively well defined inputs and target outputs, and a good deal of work in fields like machine learning are about optimization and improvements in accuracy. Patents can be awarded and papers can be published on a 2% improvement of the current standard. However, in a more experimental and lab based setting, there are several issues that make this difficult. There are no reference datasets, and there is significant operator variability that can make small improvements undetectable or unreproducible. A bigger problem is the defined target output. For prediction type tasks, accuracy is a strong and easily verifiable metric. In the development of therapeutics, even animal models are expensive and highly variable which makes comparisons difficult. Agreeing on the correct metric (overall survival, presence of a biomarker, quality of life, etc) can be difficult if drugs are meant to do different things or have different safety profiles.
In the end, we actually do run competitions in the biomedical life sciences, but in an incredibly contrived and unoptimized way. One could argue that the grant review process is in some ways a private competition, where peer reviewers decide what ‘wins’ and will have the greatest impact. The obvious problem here is that the evaluation criteria for grants is not the same as those who are evaluating technology based on ability to translate to clinic. As a result, increased emphasis is put on novelty, and lesser emphasis on scalability, and raw increases in accuracy over a previous method. Clinical trials are sometimes competitions, where a drug will be compared against the previous standard of care. However, as Vinay Prasad points out, a lot of these trials aren’t fair comparisons and they need to be continuously updated and done again as new therapies come out.
What got me thinking about this is all the different types of drug or gene delivery vectors that exist. While its true that each vector might have its own advantages or use cases, surely not all will be used clinically. Retroviruses, lentiviruses, adenoviruses, polymeric nanoparticles, lipid nanoparticles, each with their own different variations all exist. How, as a drug developer, am I supposed to know which to use if there is no way to compare them effectively? In the end, I think that we should be having more direct comparisons (or competitions), early in the development pipeline (to keep costs low), which should be enabled by greater materials and methods sharing (so that competing researchers can learn how to use competing methods and not work on the same thing), which will in the end provide important answers to translational researchers who are looking to bring innovations to market.
Another issue is incentive structures. A direct comparison between cutting edge methods on a single metric is dangerous for an early career researcher who doesn’t want to ruffle a lot of feathers. Another issue is that researchers are awarded for novel contributions, and not necessarily on solving a specific problem in the optimal way. In other words, 0 to 1 innovations may be encouraged more than 1 to 100 innovations (see the awardees of the CRISPR Nobel prize). There is good reason for this, as 0 to 1 innovations really represent the spirit of research, which is to create and push the boundaries of human knowledge. Still, this isn’t the same goal as solving problems. Sometimes, there is a distinction between originality versus being good, or useful. It is a difficult thing to do, but I think it is important for researchers to be honest about where their work fits into the scientific landscape and what problems or questions their work can be applied towards. We don’t need 100 ways of solving a problem that are all mediocre. Rather, wouldn’t it make more sense to devote the manpower to solve a single problem really well?
Sometimes a technology will be developed, and a significant portion of that lab’s work will be built upon using that technology. If it is the case that another technology is better, what incentives are in place to give up and try something new? How can we develop incentives for people to work together to solve problems instead of each solving the problem in their own unique way? I have a couple ideas if you keep reading :)
I think some of the problem comes down to the differences between basic scientists and engineers. Growing up, the distinction was always that engineers were committed to solving real world problems, and that basic scientists were committed to finding out new things about the world. But recently, I have noticed a blurring of the lines; starting companies is becoming fashionable and keeping oneself siloed to academia is now unattractive to some, while before it was seen as the pure and noble thing to do. Especially in startups, there is this narrative of cross disciplinary teams being optimal, and I don’t disagree with this, but I can’t help but think that some things are falling through the cracks. In an age now where in companies, screening is king, who is doing the mechanistic work to figure out what is exactly going on? From my understanding, in the past it was the case that academics would figure out underlying biology and license away knowledge or consult for companies that developed target driven therapies. Now, academics want to be involved in developing the therapy and in many cases will co-found the company themselves instead of licensing away the technology.
I think that on balance, this is a very good thing. Where I worry is when the spirit of novelty gets forced on to problem solving or vice versa. There are cases where creativity is necessary for problem solving, but also many cases where it isn’t. Problem solving requires a focus on inputs and outputs, and is not necessarily dependent on novelty or creativity. The way that NIH grants are funded, there seems to be a lot of emphasis on developing novel methods or finding out new things, but also there is an emphasis on solving problems. Grant funding projects need to be ‘significant’. Is there any worry that by forcing researchers into this box of needing to be both novel and useful, you end up with work that ends up not really fulfilling either criteria? Some of the most important discoveries are based entirely on curiosity or happenstance, and not because a funding agency thought it would be a good experiment to do. On the other hand, companies and other translational researchers don’t want more work forced onto a problem and sold into a story if there are better ways of solving the problem. Suddenly, no one knows the best way to solve the problem and we have to rely on hype stories and experiments that only the lab who invented them has the expertise to carry out.
Science is difficult and there are so many challenges that need to be overcome for discoveries to be made useful. My belief is that we should optimize for quality over quantity. Researchers should not be pressured into doing everything (finding out mechanisms, translating the technology, discovering entirely new things). It should be okay for biologists and engineers to be distinct groups, each focusing on their own problems.
I think that discussion about productivity in science shouldn’t necessarily be about h-index, citations, or papers published. Rather, what is most interesting to me is companies founded, patents filed, students trained, discoveries made, patients treated, etc. The current metrics that are used are in essence just vanity or surrogate metrics. They don’t actually mean anything and are more used for convenience than anything else. Unfortunately, the only way to really understand and gauge impact is to talk to peer researchers honestly and maybe even anonymously. Still, the goals of basic scientists and translational scientists are different. This is another reason why vanity metrics don’t work well (different fields, goals, circumstances all affect them). People who sell out for citations or papers or who applaud others for these metrics are doing science a disservice.
For translational scientists, I think it really should be about patients dosed, or QALYs added. Companies formed could be another one, because even if they fail it is a good public message that the strategy didn’t work. For basic scientists, I think patents filed or discoveries made are most important. Unfortunately, the only ground truth we have are textbooks, as papers get overturned all the time. If the goal of basic science research is to make new discoveries, then shouldn’t the bar be to rewrite textbooks, not just publish papers or accumulate citations? When we take stock of what we know (like when textbooks are written), we do a rigorous and hopefully unbiased review of the literature. Unvalidated one-off discoveries or ‘me too’ papers don’t get included and rightfully so, because they often aren’t really changing the scientific landscape. Patents are similarly good in that an extensive review occurs until they are granted. Basic scientists can be sure that if they are awarded a patent, they did novel work that definitively has not been done before.
Another problem in sciences in line with point number 1 is the idea that because of the lack of focus on 1 to 100 innovations and the greater focus on novelty, we neglect to think about effect sizes. One of the things I realized right when I started learning about therapeutics was that just because there was a drug for something, didn’t mean it was a cure. In fact, most people that get a drug have to cycle to something else and might still not respond. How could it be that people would claim to have developed a cancer cure but it didn’t cure everyone? I think it is dishonest and actively bad to hype medicines like this, especially when people are desperate and will do anything for it to work. It leads to distrust of the scientific community and for some reason a hatred towards an industry whose sole purpose is to save people’s lives (biopharma). Pop science and discussion on Twitter need to be measured and describe what the scientists actually did. Did they dose in humans or just in mice? What percent of people responded to therapy? What was the duration of response? How was the side effect profile? How does the therapy compare to the standard of care?
One of the things I learned when working at Alix Ventures was that translation can’t keep up with academia. The supply of ‘promising’ companies far exceeds the supply of capital. This means we either need to fund science more, or be more selective about what we classify as promising. No one has the time or expertise to verify every published therapy, and so the onus must be on the original scientists to be truthful about the potential upsides and limitations of any given approach. An important part of that communication has to be discussion about effect size, not just that it might work. To what extent will it work? Everything is a spectrum.
I think that if there is one thing that we’ve learned from the tech industry, it is that hubs of talent can be extremely powerful. Silicon valley spurred a generation of extremely impactful companies on a scale that may never be matched. There has been a lot of talk recently about decentralization due to Covid or crypto, etc. I’m skeptical. I think the ‘rich get richer’ effect, if we saw it in tech, we will see it especially in biotech.
Why is this? In a discipline so rooted in experimentation, where experience is essential and involved mentorship is key, hubs benefit from economies of scale. Research equipment is prohibitively expensive, and having nearby dedicated cores for sequencing or flow cytometry or microscopy can save a dying project. Especially as equipment gets updated and needs to be maintained by service representatives, having them all in one place and consistently fixed and replaced is essential to keeping a research engine humming. The alternative is sending samples via mail where they could get lost or damaged, and where entire weeks can be wasted.
In fact, equipment turns out to be a relatively minor argument. The real kicker is having the expertise of thousands of scientists in one place, where ideas can cross populate and problems can easily be fixed. Modern science is a network, relying on work from hundreds of other labs each for different techniques or methods. Bumping into experts to chat and share research is the whole point of conferences and symposiums isn’t it? How nice would it be to have one of those every single day in house. Students get access to the very best professors and research opportunities. Professors get access to the most curious, brightest, and ambitious young scientists to train. The Boston life sciences ecosystem needs a better name, but there is no where better to do high impact work at light speed.
One last point: in the biomedical sciences, for better or for worse, there is significant aversion to risk not only from regulatory agencies, but also at times, investigators. Being in a hub with money and support from peers and a safety net of smart people to figure out what to do, there is a sense that you can do anything. You don’t need to ask for permission or validation in order to do something. Things move quickly, some things will fail, but you learn and sometimes come up with huge discoveries.