8 more tips for understanding trial results in biotech investing
Part 2: Presentation tricks, population biases and how to tell if a drug actually works in the real world.
In Part 1 I gave you the read on endpoints, significance, and how to spot the real data not the story the company is selling.
Part 2 focuses on how the framing of the results seeks to tell a story, the biases we see that can affect real world performance and the mistakes that happen in trial design that can mean even positive results aren’t reflected in patients once it goes to market.
Presentation and context tricks
Spin in abstracts and press releases.
Presenting results in a particular way to tell a story of success, particularly when primary endpoints are not significant. Techniques include ignoring the primary outcome, focusing on secondary results and using language that implies efficacy without claiming it (”the drug demonstrated encouraging activity”). Systematic reviews of randomised trials with non-significant primary outcomes find spin in a substantial majority of abstracts and conclusions.12
How to check: Fundamentally the check is asking, is the language used supported by the statistics? Does the press release (PR) focus on the direction of effect rather than the significance? e.g. the odds of survival were increased but the statistical test said this was not significant means the results could just be due to chance.
If the primary outcome was not statistically significant but the abstract/PR emphasises secondary outcomes, subgroups or within-group changes adopt a sceptical approach. Were these pre-planned or ‘post-hoc’ comparisons? Look for language that implies benefit without stating it directly: “promising,” “encouraging,” “reduction”, “trend towards.” Check whether the conclusion could reasonably be drawn from the results section, or whether it requires the reader to look away from what the trial actually found.
No comparator, or the wrong comparator.
The clinically relevant question for patients at the end of the day and what makes a drug a success is whether it beats the existing standard of care. That isn’t always a placebo. Many trials are designed against placebo precisely because that is easier to win, after which the press release implies clinical superiority over current treatment when the trial does not show that at all. Though if there is no existing treatment then placebo should be used. In theory this should always followed but in practice there are examples where it isn’t. Some trials have 3 arms or more when a treatment is compared to placebo and to the standard treatment or with different doses.
In oncology, the comparator question is now widely scrutinised. In other areas, less so. Always check what the control arm received and whether anyone in routine practice would recognise it as the standard of care.
How to check: Find the control arm in the methods section or trial registration. Ask whether the comparator reflects current standard of care for this indication and setting. (A quick google will usually tell you). If the trial is placebo-controlled and an established treatment already exists, look for any active-controlled or head-to-head data. Where none exists, treat any claim of clinical benefit as relative to placebo, not relative to what the patient would otherwise receive.
The wrong populations
Testing with the wrong people
A trial can be technically rigorous while studying a population that does not represent the disease in question.
Example
The PACE trial is a teaching example. Participants were recruited under the Oxford criteria for chronic fatigue syndrome, which require only six months of unexplained disabling fatigue. They did not require post-exertional malaise, the defining diagnostic feature of ME/CFS.34 A trial that recruits “chronic fatigue” and reports findings about ME/CFS is answering a different question from the one being asked.
How to check: Identify the diagnostic criteria stated in the inclusion criteria, in the methods section or trial registration. Cross-reference against the current consensus definition for the condition. If they diverge, ask whether the trial population plausibly represents the patients being treated in clinical practice. In some diseases patients present late - are they only treating patients who presented early?
Where a trial studies a broader, narrower, or otherwise different population from the disease in question, treat any clinical inference as provisional and ask what evidence exists in the representative group.
Enriched trial populations that do not reflect real patients.
Trials routinely exclude the elderly, people with comorbidities, and anyone on complex medication regimens, exactly the populations who will receive the drug in practice. One of the most egregious and still ongoing issues in trial design has been the exclusion of female patients from trials (this requires a post in it’s own right I might try sometime in the future).
Of course trials do need to define a tight population to make sure they are giving the medication to people who have the potential to benefit whilst weighing up potential harms from an unproven drug so exclusion criteria should be in there, but they should be about protecting people and defining a population who could realistically benefit.
The gap between trial and real-world response is well-documented across cardiology, oncology, and psychiatry.5
How to check: Read the inclusion and exclusion criteria in the methods section. Note what was excluded: age ranges, comorbidities, concomitant medications, prior treatment history. Consider the typical patient receiving this drug in practice and how far they depart from the trial population. The wider the gap between trial eligibility and clinical reality, the less the efficacy estimate generalises.
The wrong lense
Short follow-up hiding late harms or effect decay.
A 12-week trial showing weight loss says nothing about what happens at two years. A trial reporting tumour response says nothing about duration of response or overall survival. Short follow-up also systematically underestimates adverse events that develop over time.
Case study
Rofecoxib (Vioxx) was a NSAID approved in 1999 for pain and arthritis. Short-term gastrointestinal trials showed the safety profile Merck wanted but the increased risk of heart attacks and other cardiovascular events that ultimately led to a voluntary withdrawal of the drug from the market in 2004 took several years and a different trial design to surface.67
How to check: Find the primary endpoint and its timepoint. Ask whether that timepoint is long enough for the outcome to be clinically meaningful and whether harms with delayed onset could plausibly have been missed.
Where the primary outcome is a surrogate, look for evidence that this surrogate reliably predicts the long-term outcome you actually care about. Check the trial registration for any planned long-term extension study, and note whether it has been completed and its results posted.
Missing data
Publication bias and the missing negative trials.
Outcomes with positive results are more likely to be reported, and the effect distorts the evidence base systematically.8 If the published literature is the only input, the drug looks better than it is, and meta-analyses built on that literature inherits the bias.
Case study
Reboxetine is one of the cleanest case studies. When the German health technology body compelled disclosure of unpublished trials, the apparent efficacy of the drug largely disappeared.9 Mandatory trial registration and required posting of results to ClinicalTrials.gov has narrowed the gap, but enforcement is uneven and selective publication of registered trials persists.
How to check: Search the trial registry (ClinicalTrials.gov/WHO International Clinical Trials Registry Platform) for registered studies of this drug or intervention that do not appear in the published literature. Check completion dates against results posting. For older drugs or well-established indications, look for systematic reviews that have attempted to incorporate unpublished data, and check whether those reviews assessed risk of publication bias and what they found. An absence of negative trials in the published record is not evidence that negative trials do not exist.
Selective harm reporting and downplaying adverse events.
The press release tells you the drug worked. It does not dwell on who stopped taking it or why. Adverse event reporting in trial publications is frequently incomplete, inconsistent, or buried in supplementary appendices. Serious harms may be split across multiple narrow categories so that no single line item looks alarming.
The overall discontinuation rate due to adverse events, which is often the most honest single summary of tolerability, may not appear in the abstract at all.
How to check: Look at the adverse event tables. Check the overall discontinuation rate in each arm and the discontinuation rate due to adverse events specifically. Compare against placebo or comparator. Look for whether serious adverse events are reported individually or grouped. For drugs already on the market, check the FDA’s adverse event reporting system (FAERS) or the EU/UK equivalents for commercially available drugs.
Conflicts
Conflicts of interest and funding source bias.
Industry-funded trials are more likely to report favourable outcomes than independently funded trials of the same interventions. This is not a conspiracy theory; it is a repeatedly documented empirical finding across multiple systematic reviews.1011
None of this means industry-funded research is wrong or false. It means the funding source is a relevant piece of context when assessing how much weight to place on a finding, particularly when independent replication is absent or the modality is brand new. It’s another reason why well run double blinded trials are stronger than open label trials.
How to check: This applies to almost every biotech trial. If you’re not sure or it sounds independent, read the conflict of interest disclosures and funding statement and identify the sponsor.
Summary
None of this requires deep statistical expertise; it requires the willingness to spend twenty minutes with the source documents before forming a view.
The pharmaceutical industry produces life-saving useful drugs. The pipeline of medicines that have transformed survival in HIV, hepatitis C, several cancers, and a growing list of rare diseases is real, and nothing above changes that. The point of this piece is narrower. The document that reaches investors, prescribers, journalists, and patients is a product of marketing as well as science, a representation of the trial that has been optimised for purposes other than the reader’s understanding of the underlying data.
I do not have all the answers, and there are reasonable defences of many of the practices above in specific contexts.
Closing that gap of what is presented and what is real starts with having a sceptical view and making up your own mind about the data you’re presented with. To find the holes, start with one question: “From the data they’ve given how can I prove them wrong?”
Useful sources of information
FDA Drugs@FDA (review documents and labels)
SEC filings (for commercial context and risk disclosures)
WHO International Clinical Trials Registry Platform.
I have a chronic illness that limits my capacity to write; AI is sometimes used in drafting text, creating images and checking this work which may introduce errors (from AI or my brain).
Nothing in this newsletter constitutes investment, financial, or medical advice. I am a biotech investor and may hold positions in companies, funds, or sectors discussed here. Where a material conflict of interest exists, I will disclose it in the relevant piece. This writing is primarily my opinion and facts are referenced to allow checking. Opinions are my own and subject to change as evidence develops.
Hypothesis Next explores the intersection of science, health, and the forces that shape social and financial capital. If this added value, please consider donating to the Open Medicine Foundation, the world’s leading nonprofit funding research into ME/CFS and Long Covid. A cause close to my heart. Visit OMF.ngo to donate or learn more.
References
Boutron I, Dutton S, Ravaud P, Altman DG (2010). Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA 303(20): 2058–2064.
Yavchitz A, Ravaud P, Altman DG, et al. (2016). A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity. J Clin Epidemiol 75: 56–65.
Carruthers BM, Jain AK, De Meirleir KL, et al. (2003). Myalgic encephalomyelitis/chronic fatigue syndrome: clinical working case definition (Canadian Consensus Criteria). Journal of Chronic Fatigue Syndrome 11(1): 7–115.
Institute of Medicine (2015). Beyond Myalgic Encephalomyelitis/Chronic Fatigue Syndrome: Redefining an Illness. Washington DC: National Academies Press.
Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J (2015). A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials 16: 495.
Bresalier RS, Sandler RS, Quan H, et al., for the Adenomatous Polyp Prevention on Vioxx (APPROVe) Trial Investigators (2005). Cardiovascular events associated with rofecoxib in a colorectal adenoma chemoprevention trial. NEJM 352: 1092–1102.
Krumholz HM, Ross JS, Presler AH, Egilman DS (2007). What have we learnt from Vioxx? BMJ 334: 120–123.
Dwan K, Gamble C, Williamson PR, Kirkham JJ (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias: an updated review. PLoS One 8(7): e66844.
Eyding D, Lelgemann M, Grouven U, et al. (2010). Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 341: c4737.
Lexchin J, Bero LA, Djulbegovic B, Clark O (2003). Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 326: 1167–1170.
Lundh A, Lexchin J, Mintzes B, Schroll JB, Bero L (2017). Industry sponsorship and research outcome. Cochrane Database Syst Rev 2: MR000033.