Great post and some valid points raised. However, I think several of the limitations you identify actually bias the RS estimate toward the null, not away from it.
Broad indication labels, for instance, make it harder to match genetic traits to drug indications at the 0.8 similarity threshold, so they are more likely to be classified as unsupported. Similarly, the lack of directionality of effect means that some T-I pairs classified as genetically supported actually have genetics arguing against the drug's mechanism. These pairs would be expected to fail, diluting the RS among the supported group. The true RS for directionally concordant genetic support is therefore likely even higher than 2.6x.
On L2G:
1) Minikel showed that RS increases with L2G share threshold (Figure 1c), which is what you would predict if the signal is real.
2) comparing L2G to nearest-gene at a single threshold is misleading because L2G is a continuous score and outperforms on precision at higher thresholds, which is what matters when a false positive costs millions.
4) 23andMe could reproduce the results, and even showed a dose response between gene mapping confidence and clinical success, reaching up to 5x RS at the highest tier.
I think a lot of people have misunderstood what the paper claims. I don’t think Minikel never argue that genetic support should be a gate for drug development. They argue it is an enrichment signal, and a probabilistic tool for portfolio prioritization. Individual targets succeeding without genetic support does not invalidate that observation.
Thanks. I agree that some limitations would bias the RS estimate toward the null, but it's not so easy to generalize this as a uniform pattern. I gave examples, where the similarity score >0.8 represented false positive matches, in which case they are labeled as "genetically supported"; if drugs for the same T-I pairs were approved, this could inflate the RS. Similarly, the tendency for duplicate entries among approved drugs could shift the RS to both directions, but in my feeling they appeared to be more common among genetically supported drugs.
On L2G: there is a trend toward higher RS with increasing L2G threshold, but the incresae is very modest (I don't even know if it is significant; I didn't see a statistic about it in the paper) and not throughout the spectrum. The 23andMe results indeed look more consistent in this regard and make intuitively sense. But in that dataset there is additional noise due to self-reported disease labeling, the degree of which varies across indications.
Overall, I'm on the same boat. I also believe that genetic data offer extremely valuable insights into target discovery and clinical development strategy — most likely way more than any other preclinical approach. The enrichment signal in the paper is also most probably real. My main criticism is that the widespread imprecisions in the underlying datasets make it extremely hard (if not impossible) to compute an accurate global RS estimate across the entire drug development landscape.
Fair points, and I think we mostly agree. The exact number is hard to nail down given the data issues you raised, and your review does a nice job showing where the noise comes from. For me, what matters is that the signal holds up across different datasets and groups, even if the precise estimate is fuzzy. Thanks for the write-up, looking forward to the next post!
Whenever a paper's methodology is surprising, I get suspicious. There might be good reasons. But often, it suggests that the obvious and intuitive analysis didn't yield the results they were looking for.
yeah, people just take these things on face value and lots of nuance here. For example: going by this logic, you would never pursue PDCD1 (famously called PD-1). When people started working in GLP1s, I don’t believe they had the necessary GWAS/ genetics support.
Great post and some valid points raised. However, I think several of the limitations you identify actually bias the RS estimate toward the null, not away from it.
Broad indication labels, for instance, make it harder to match genetic traits to drug indications at the 0.8 similarity threshold, so they are more likely to be classified as unsupported. Similarly, the lack of directionality of effect means that some T-I pairs classified as genetically supported actually have genetics arguing against the drug's mechanism. These pairs would be expected to fail, diluting the RS among the supported group. The true RS for directionally concordant genetic support is therefore likely even higher than 2.6x.
On L2G:
1) Minikel showed that RS increases with L2G share threshold (Figure 1c), which is what you would predict if the signal is real.
2) comparing L2G to nearest-gene at a single threshold is misleading because L2G is a continuous score and outperforms on precision at higher thresholds, which is what matters when a false positive costs millions.
3) imperfect gene mapping adds noise that, again, dilutes the RS estimate.
4) 23andMe could reproduce the results, and even showed a dose response between gene mapping confidence and clinical success, reaching up to 5x RS at the highest tier.
I think a lot of people have misunderstood what the paper claims. I don’t think Minikel never argue that genetic support should be a gate for drug development. They argue it is an enrichment signal, and a probabilistic tool for portfolio prioritization. Individual targets succeeding without genetic support does not invalidate that observation.
Thanks. I agree that some limitations would bias the RS estimate toward the null, but it's not so easy to generalize this as a uniform pattern. I gave examples, where the similarity score >0.8 represented false positive matches, in which case they are labeled as "genetically supported"; if drugs for the same T-I pairs were approved, this could inflate the RS. Similarly, the tendency for duplicate entries among approved drugs could shift the RS to both directions, but in my feeling they appeared to be more common among genetically supported drugs.
On L2G: there is a trend toward higher RS with increasing L2G threshold, but the incresae is very modest (I don't even know if it is significant; I didn't see a statistic about it in the paper) and not throughout the spectrum. The 23andMe results indeed look more consistent in this regard and make intuitively sense. But in that dataset there is additional noise due to self-reported disease labeling, the degree of which varies across indications.
Overall, I'm on the same boat. I also believe that genetic data offer extremely valuable insights into target discovery and clinical development strategy — most likely way more than any other preclinical approach. The enrichment signal in the paper is also most probably real. My main criticism is that the widespread imprecisions in the underlying datasets make it extremely hard (if not impossible) to compute an accurate global RS estimate across the entire drug development landscape.
Fair points, and I think we mostly agree. The exact number is hard to nail down given the data issues you raised, and your review does a nice job showing where the noise comes from. For me, what matters is that the signal holds up across different datasets and groups, even if the precise estimate is fuzzy. Thanks for the write-up, looking forward to the next post!
Whenever a paper's methodology is surprising, I get suspicious. There might be good reasons. But often, it suggests that the obvious and intuitive analysis didn't yield the results they were looking for.
yeah, people just take these things on face value and lots of nuance here. For example: going by this logic, you would never pursue PDCD1 (famously called PD-1). When people started working in GLP1s, I don’t believe they had the necessary GWAS/ genetics support.
Great post!
Thanks Marios, this was a very interesting and thoughtful review.
Neat breakdown, Marios. Thanks!