Introduction
This paper arises from a dubious decision of the editor (myself) to publish in this Journal my rebuttal1 of an article supporting hydroxychloroquine (HCQ) for treatment of SARS-CoV-2 (Covid-19 or Covid), written by Emeritus Prof Robert Clancy and published in the Australian magazine Quadrant (March 2024 edition).2 The magazine initially indicated support for my article but subsequently reversed that decision. In response to my TMJ article,1 Clancy submitted his own response.3 I felt I had an obligation to publish it without external review or editorial comment. Our publication fee was waived. We also received further comment from two groups in the USA and these were also published under the same conditions.4,5 Clancy has since published a further article in Quadrant6 in which he again criticises my TMJ rebuttal, stating that my article reminded him “…of schoolboy debates where the truth never seemed to be particularly important: the point was to win”. Actually, my sole purpose was and remains to determine in retrospect (if possible) whether HCQ is effective against Covid-19. Paradoxically, he went on to thank me graciously for having the integrity to facilitate a debate in a peer-reviewed medical journal. I agree that debate over such matters is important. It is also important to add that none of the publications in the above history were peer-reviewed, and this was clearly stated. However, this article has been peer reviewed.
The broader background was the chaotic and disruptive international pandemic most likely caused by an escaped experimental virus that has killed over 7 million people internationally since 2020.7 The likely SARS-2-CoV history is well described by journalist Sharri Markson in her 2021 book “What Really Happened in Wuhan” (Harper Collins; ISBN 9781460714027). This book is worth reading.
The outcome of the above sequence of events has been the interesting debate that, regardless of whether HCQ is effective in SARS-CoV-2 infection, legitimate and still outstanding questions arise over how a drug’s potential effectiveness should be judged, including the question of the standing of randomised controlled clinical trials (RCT) as the gold standard of medical epistemology. This matter arose because in my rebuttal1 I focused on a sub-section of the website referred to by Clancy2 (https://c19hcq.org) which claims to summarise the total published evidence for the effectiveness of HCQ in patients who succumb to SARS-2-CoV. I initially studied a section of the c19hcq.org data that included trials categorised as randomised controlled trials (RCT) in which HCQ was given ‘early’ (11 studies). This arose from Clancy’s claim that HCQ is only effective if given early in the course of COVID infection. Having concluded that these trials were inconclusive (see added detail below) I disregarded the remaining trials under the assumption that they would be of lesser quality. This became the basis for the issue of the standing of RCTs raised in the other replies, particularly by the authors of reference 5.
The website https://c19hcq.org
Clancy’s arguments in support of HCQ rested on separate studies or papers by Dr Vladimir Zelenko8,9 and Dr Harvey Risch10 in the US, and Dr Didier Raoult’s group in Marseilles, France.11,12 He also referred to the studies listed at the above website. The site represents a prodigious amount of work and is an important resource for persons researching the efficacy of HCQ and other drugs. Regrettably, several matters compromise automatic acceptance of its analyses, as follows:
1. The theoretical basis of a potential effect of HCQ is that it has several antiviral actions at the time point when the virus enters cells and multiplies, with a presumed inhibitory action on disease progression, including development of viral pneumonia, organ failure and death.2,16 Thus emphasis is given to its early deployment. Indeed, Clancy’s claim2 is that late administration has no hope of benefit. However the forest plots for late treatment also show efficacy in 184 of 380 non-significant studies where the point estimate is to the left of the null line but the 95% CI crosses it, and in 89 that are statistically significant (i.e. the 95% CI is wholly to the left of the null indicator). This inconsistency is unexplained. Part of the problem here lies in defining precisely what is meant by ‘early’ and ‘late’ Covid infection. The above mechanism of action of HCQ would support administration at the point of infection – if only one could diagnose it during this pre-symptomatic phase which is associated with false negatives on PCR testing. The website classifies early treatment as “…if most patients are not already at a severe stage at the time of treatment (for example based on oxygen status or lung involvement), and treatment started within 5 days of the onset of symptoms.” The validity of this definition is unproven, and it may be too late. Such thoughts led me to suggest1 that if it were proven to be effective, HCQ should be given prophylactically in a pandemic, though this would have its own potential for harm from diversion of supplies away from treatment of other diseases in which HCQ has an established role, or direct toxicity.
2. The site contains a section explaining how RCTs “… have many potential biases” whereas the subsequent paragraph on observational studies begins “Observational studies have been found to be reliable”. The analysis may well be correct but the wording indicates a potential site bias, as it implies its editors may be inclined to believe positive results from observational studies and disbelieve negative results from RCTs. Most medical researchers, I suggest, would believe the reverse. Nevertheless, it is legitimate to question RCT methodology and seek alternatives (see below).
3. The web pages associated with each study display a box claiming an HCQ risk reduction for mortality, hospitalisation, progression, recovery, cases and viral clearance with a reported P < 0.00000000001. The method used to calculate this unlikely probability is not stated but it clearly combines trials with multiple end-points.
4. The website is anonymous. An effort to determine its owners, operators and their credentials or financing has failed.17 The anonymity does not by itself generate suspicion but it is a curious position for authors to take, given that one might assume they are interested in openness in scientific research and publishing. Another website grades the site anonymously as biased and “Questionable” because it allegedly promotes pseudoscience, lacks transparency, contains failed facts, and frequently misrepresents scientific studies.18 The absence of examples means that these accusations lack evidential support. [Note: c19hcq.com when entered into a browser appears as c19hcq.org].
I examined studies from three sub-lists. These were the 11 studies of ‘early’ treatment described as HCQ COVID-19 early treatment and prophylaxis RCTs,19-30 the statistically significant (defined as above) studies in the list All 39 early treatment studies (eighteen studies11,31-47 were identified) and eight studies reported for post-exposure prophylaxis (PEP).48-55 None of the first or third group and three of the second were ‘excluded’. The results in these three groups are presented below.
Sources of information relied on by Clancy in his original Quadrant article
In addition to promoting the efficacy of HCQ, Clancy claimed that the negative conclusions as to its efficacy by medical experts and regulatory authorities worldwide was a result of an interaction between the World Health Organisation (WHO) and the global pharmaceutical industry, designed to cancel the independence of medical practitioners in the treatment of individual patients. I have previously quoted his exact words.2 No evidence for these claims was given, but the topic has been expanded in Clancy’s more recent Quadrant article.6 The implication that co-operation between WHO and the global pharmaceutical industry to suppress the repurposing of existing cheap drugs in order to both centralise medical practice and to protect the high profits likely from the vaccines lacked credibility. I do not believe this theory (though stellar profits were indeed made), but mention it as a context for the claims regarding HCQ.
Gautret et al (Raoult).11 This study at the The Méditerranée Infection University Hospital Institute in Marseille is said to be the first trial to have reported treatment with HCQ (alone or with azithromycin). This centre has an outpatient facility but the paper does not specify whether the patients were inpatients, outpatients or both. Symptomatic patients were recruited within 5 days of symptom onset. Twenty treated patients and 16 ‘controls’ (untreated patients from other French centres, and Marseilles patients who declined entry) were studied, indicating lack of randomisation. The primary end-point was reduction in viral carriage or absence of virus at 6 days after enrolment. The treated group was older (51.2 vs 37.3 y), had more male patients, and fewer were asymptomatic (10% vs 25%). These differences would have biased the results away from the null. At day 6, 14 of 20 treated and 2 of 16 controls had Covid-negative nasopharyngeal swabs. All patients given both HCQ and azithromycin (AZ; n=6) were PCR negative by day five. These results were published by the authors in spite of the small sample size and opportunities for bias in the design, because the results appeared clear and the authors were conscious of an “urgent need for an effective drug against SARS-CoV-2 in the current pandemic context”. This was reasonable. The study demanded attention at least as a hypothesis-generating event to be tested under an unbiased trial protocol. Also, the published work itself does not generate the strong suspicion of malfeasance later directed at Dr Raoult, but it may nevertheless represent a hurried attempt with cut corners, to identify a potential effective treatment in the face of the COVID debacle. Ironically, c19hcq.org previously excluded this study on the grounds of “excessive unadjusted differences between groups; results only for PCR status which may be significantly different to symptoms.” In a further irony, the paper has now been retracted (17 December 2024) by the Journal’s publisher Elsevier, for several listed reasons,15 and by c19hcq.org.
Zelenko first publication.8 This was a retrospective and hence non-randomised study of patients with proven Covid infection living in Monroe, New York, USA being treated in Zelenko’s general practice (ie not hospitalised). Data were gathered from electronic health records. Patients were stratified according to risk according to age, symptoms, co-morbidities and lung disease. The trial treatment was HCQ, AZ and zinc sulphate, given to 104 patients and also to 37 ‘additional patients’ diagnosed with IgG tests. Patients (n = 377) not meeting the defined risk stratification criteria (but who were at either high or low but undefined risk), acted as a control group. Patients were selected if they had a confirmed PCR or antibody test of for SARS-CoV-2, were aged > 60 (group A), aged > 60 with dyspnoea (B), and > 60 y with symptoms and specified co-morbidities (C) or were pregnant (1 patient). Trial end points were hospitalisation or death within 28 days of follow-up. Four of 141 treated and 58/377 control patients were hospitalised (P < 0.0001 by my calculation in a chi-squared test but reported as P < 0.001). Corresponding results for death were 1 and 13 (P < 0.08). The P values given in the paper (0.12 for death) may have been incorrectly analysed from results expressed as percentages instead of absolute numbers. The authors note that “when we compared the outcome of all risk-stratified patients…the P value for death was 0.0154.” They explained that this result was not formally reported because the “clinical information available for control patients allowing for risk adjustment was incomplete.” The results of this study prompted Zelenko to address a letter to then US president Trump, who subsequently championed the use of the drug.
The capacity to interpret this study lies in the standing of the ‘control’ group. The differences between patients deemed to be controls and the treated participants include:
- The treated group were seen by study doctors (directly or by video link) for the purposes of entry. This does not appear to have been the case for controls.
- The paper mentions a ‘defined time point’ of the analysis but does not in fact define it. Presumably it was the date of the consultation or enrolment, which in turn defined the 28 day period of follow up for treated subjects. How this interval if any was met for control subjects is not described.
- The qualification above, given by the authors regarding deaths, may apply to all other potential comparisons. The paper is silent on the age, sex distribution, co-morbidities, and clinical severity of the control group.
I conclude that this publication is so deficient in its description of the control group characteristics that it is unacceptable as evidence HCQ efficacy.
Zelenko 2. The Quadrant article2 states that Dr Zelenko’s experience was “…later published as a case series of 2200 patients.” My understanding is that there was no publication in any journal. An uploaded file may have existed temporarily on the internet. The study is referred to by Gkioulekis56 as being in his supplementary data analysis and also contained in a separate paper by Risch,57 but neither publication contains what might be regarded as even a draft manuscript by Zelenko. Zelenko sent a two-page letter to Dr. Moshe Bar Siman Tov, the then Director-General of the Israeli Ministry of Health. Other recipients are likely, as the Tov letter claims that the author was in contact also with White House staff and the governments of Ukraine (where Zelenko was born), Brazil, Honduras, Peru and Chile. He also made several YouTube videos and incidentally was a controversial critic of Covid vaccines. The letter, provided to me in confidence, simply states that ‘approximately’ 2200 PCR-positive cases were seen, ‘approximately’ 1400 of these who were < 60 and mild cases were given supportive care only, and ‘approximately’ 800 were treated with what was then termed the Zelenko protocol of HCQ, AZ and ZnSO4. The outcome data are that none had serious side effects or cardiac arrhythmias, two died, and 12 were admitted to hospital (and subsequently discharged). Dr Zelenko suggested that it was “…imperative to initiate aggressive and early treatment of Covid-19 in the primary care setting“.
The above death rate (0.25%) is impressive, but is consistent with patients less than 60 years of age according to Chinese and Italian Covid mortality data given by Dr Gkioulekis.58 The absence of detail is also impressive. Obvious questions are whether the protocol was identical to the earlier paper, whether the patients were followed for 28 days as before, whether randomisation to treatment or control was deployed and what the results were in the untreated group. The lack of detail is ruinous for the evidentiary quality required to move governments, even in an emergency. Sadly, Dr Zelenko died in 2022 aged 48 before being able to ensure availability of his work to the wider scientific community.
Paper by Dr Harvey Risch. Dr Risch’s review paper10 is an erudite and well-written, though in some eyes controversial, review in defence of HCQ in the treatment of Covid. It should be read by anyone interested in the questions under discussion here. The publishing Journal (American Journal of Epidemiology) was aware of the controversy and included a disclaimer with the paper. My sole comment relates to Risch’s positive description of the paper by Esper et al,31 which was severely criticised (see below) after Risch’s paper was published.
Examination of selected trials at c19hcq.org 1: HCQ COVID-19 early treatment and prophylaxis RCTs
Eleven trials are displayed.19-30 Only one indicates statistical significance with a borderline P value (P < 0.02), and this is consistent with the conclusions of the group’s authors, several of whom advise against the use of HCQ. No trials were excluded by c19hcq.org. The meta-analysis gives a relative risk for all studies of 0.66 and therefore ascribes a 34% lower risk associated with HCQ (95% CI 0.44-1.01) across several end-points. One trial24 was not a study of the efficacy of HCQ. Rather, it studied whether doxycycline had an effect against SARS-2-CoV compared to HCQ. The authors state “We showed the non-inferiority of a seven-day course of doxycycline to a five-day course of hydroxychloroquine-azithromycin in terms of clinical recovery rate, virological suppression, and safety profile.” C19hcq.org states “…this trial compares (HCQ) with another treatment – results may be better when compared to placebo.” This misrepresents the comparison being made and raises an hypothesis that is impossible to verify. In my opinion, this study should be removed. Another study19 had a target sample size of 2000, but only 20 patients were randomised, of whom 16 were able to be reported (9 given HCQ+AZ, 7 placebo). One patient in the placebo group and none in the HCQ+AZ group were hospitalised under the composite end-point (hospitalisation or death within 24 weeks; there were no deaths). This trial is reported as showing a treatment benefit of 64%! This result relies on the mathematical treatment for zero events. One’s instinct is to ignore this unpublished trial and the suggestion that it shows substantial benefit of HCQ.
The study reporting a significant effect of HCQ29 (the PROTECT Study) divided the trial participants into 7 actively treated groups receiving HCQ alone, oseltamivir (OS), AZ, HC+OS, HC+AZ, OS+AZ, and HC+AZ+OS. A control group consisted of patients who declined to be randomised but agreed to be observed, but it appears that this group’s data were not used in any statistical analysis. This was not a true RCT. The primary end-point was PCR negativity or clinical improvement at 7 days, and the secondary end-point was the same at 14 days. Both patients or their physicians were unblinded to treatment. Recruitment was closed after 471 patients were enrolled from a target of 520, on the basis of futility. A P value <0.02 reported for PCR negativity at 14 days was obtained only after imputing results for missing data but the method of imputation is unclear, as is the comparison is being made. Indeed, one of the paper’s two references relating to imputation describes how it can introduce bias.14 The website c19hcq.org takes the reported data for deaths at 7 days (4/248 in the HCQ groups and 10/178 in the non-HCQ groups (which gives P < 0.022 by chi-squared test), and claims a significant RR of 0.29 (71% improvement). The publication itself does not contain this analysis. It may be legitimate to accept this result at face value but the P value is bordeline and it depends on data imputation and the other likely sources of bias. The conclusion of the authors was “Among patients with mild COVID-19, there was no statistically significant difference in the effectiveness of oral antimalarial, antiviral, or antibiotic treatments.”
In summary, of 11 RCTs deployed to test the benefit of HCQ in ‘early’ Covid infection, one was invalid and of the remainder only one was positive but contained several potential sources of bias.
Examination of selected trials at c19hcq.org 2: All statistically significant of 39 early treatment studies This sub-category returns 18 studies11,31-47 reported as showing a significant effect of HCQ. Table 1 shows characteristics of these studies. They include Gautret et al11 (excluded from analysis and now retracted by the publisher) and two further studies32,45 which were also ‘excluded’. One of these was a study of chloroquine rather than HCQ which should not strictly speaking have been included, but it could be argued that the pharmacology (but not the toxicology) of the two drugs is sufficiently similar to validate the study.
Definitely the most controversial study in this group was by Esper et al.31 This non-randomised and unblinded study was sponsored by the Prevent Senior Institute, a private Brazilian health insurer. The senior author was a medically-qualified Director of the Institute. The paper was announced by the sponsor and has not been published by peer review. Its content and wider context is awash with irregularities, mostly deplorable but some potentially criminal. Prevent Senior had a close association with office of the President, Jair Bolsonaro. It was planned under the context of a need by the sponsor to improve the Institute’s national standing, and the pandemic was seen as a vehicle for this. In April 2021, a Parliamentary Inquiry Commission of Covid-19 (CPI da Covid) was established in order to investigate alleged omissions and irregularities in the management of the pandemic in Brazil by the national government under President Bolsonaro, and the numerous alleged irregularities in the Esper paper include:
1. Patients were enrolled and treated without giving informed consent.59
2. Application for ethical approval was made and trial registration with ClinicalTrials.gov was obtained after trial data had been collected and the results had been announced by President Bolsanaro.6
3. Discrepancies and differences in the description of the trial are apparent in the trial paper and in documents lodged with various international authorities.60
4. Reference 60 states: “The dossier submitted to the CPI of the Pandemic by doctors working for Prevent Senior shows that the company manipulated patient data, including omitting the cause of death in the medical records of people treated with the so-called Covid-Kits, so that the hospital discharge rates and COVID-19 mortality rates would be favourable to the intervention. The study authors only mentioned two deaths in the treatment group as unrelated to the drug and to COVID-19, but it has been reported that at least 9 patients died during the research project. Of these, 6 were in the treatment group, two were in the control group, and there was one patient whose group was not identified. The coordinator of CONEP [Brazil National Research Ethics Commission], Jorge Venâncio, affirmed that some of the deaths that occurred during the implementation of the research protocol were not reported, as required by the Brazilian regulations.”
5. An enquiry recommended that President Bolsonaro should face nine criminal charges for his allegedly corrupt responses to the pandemic.61
Paper reference 60 should, like the article by Risch above,10 be read by all who are interested in the conduct of ethical scientific research. The severe irregularities and others related to the institutional and political responses to Covid-19 in Brazil need to be considered against the fact that Brazil was particularly affected by the pandemic. The paper is not discussed here to indulge in medical scandal. Rather I suggest that the authors of c19hcq.org, in their ambition to include all HCQ publications, avoided making value judgements on the standing of individual papers. I suggest that the paper from Prevent Senior has sufficient doubt over its provenance to justify its removal from the site.
The studies I discuss here were chosen because they gave significant results, on the basis that conclusions might be more reliable than all results combined regardless of statistical significance. However, the majority of the trials were of faulty design. Notwithstanding their poor overall quality and one alleged criminally fraudulant study, one must concede that the cumulative effect of statistical significance in 18 of 38 early studies is to give one pause in claiming that HCQ has zero efficacy in the early treatment of Covid, at least in the strain that was prevalent when the studies were carried out.
3. Post-exposure prophylaxis with HCQ
C19hcq.org places publications on the effect of HCQ on post-exposure prophylaxis (PEP) in a separate sub-category. This is appropriate, as giving the drug to exposed but still asymptomatic persons is about as close in practice as one can get to the time point of HCQ action. Eight papers are included,48-55 of which only two reported a significant effect of HCQ. Four trials were classical randomised controlled studies48-51 and all reported non-significant differences in prophylactic effect of HCQ versus placebo. A further trial52 used alternate rather than random allocation to treatment or placebo. That trial was also negative.
Two trials were performed consecutively by the same study group.51,53 The first was a prospective non-randomised trial in which the ‘control’ group consisted of patients who declined treatment or had a contraindication to HCQ, whereas the second was a prospective randomised trial with blinded treatment or placebo. The first53 showed a significant protective “definite” treatment effect (by PCR testing) (10/132 vs 36/185 (P = 0.041 after 4 weeks) but the “probable” effect (4/132 vs 8/185) was not significant (P = 0.552). By contrast, the second trial with higher participant numbers51 showed no significant corresponding effect for either category of certainty (16/574 vs 21/594 (P = 0.466) and 8/574 vs 6/594 (P = 0.547). The primary population causing or potentially causing the risk of infection in the participants changed between the studies, but both measured the emergence of infection in a population at risk by virtue of actual or potential contact with infected cases. The authors’ original conclusion “PEP with HCQ has the potential for the prevention of COVID-19 in asymptomatic individuals at risk for SARS-CoV-2 infection” had to be changed in the second study to “In conclusion, PEP with HCQ is not advantageous for the prevention of COVID-19 in asymptomatic household direct contact of the laboratory-confirmed COVID-19 cases”.
One unreliable trial without a control group compared the effect of HCQ in the development of a supposedly high-risk of infection group versus the effect in untreated low- or moderate-risk groups.54 Risk was defined in terms of the use of physical barriers to infection. HCQ was reported as having a preventive effect, but the huge potential for bias is obvious.
The final study was a short letter to the editor of a Journal containing two reports of non-randomised prophylaxis of Bulgarian Cardiac Institute employees, in two groups.55 In group 1, HCQ prophylaxis was offered to 204 of 1200 health care workers and 156 accepted, of whom none presented with symptoms of Covid-19. Of the 48 that declined, 3 developed symptoms and tested positive (P = 0.0025). The duration of follow-up is not stated and no further detail is given. Group 2 was 38 persons studied over 7 months whose PCR test for Covid was was positive. The paper does not explicitly state that this was the total number of positive cases. Thirty-three took HCQ and symptoms disappeared within 3 days and the PCR after 14 days in all was negative. The remaining 5 did not receive HCQ, 2 required hospitalisation and the PCR remained positive in 3 (P = 0.0003). The paper requires more detail to allow the reader to confidently interpret these data and accept the statistical indication.
C19hcq.org states in the meta-analysis that these eight papers represent a 30% risk reduction with HCQ. Given that all four randomised trials and one with an alternative objective method of group allocation were negative, and overall 6 of the 8 failed to show a statistically significant effect, this claim seems debatable.
The role of randomised controlled trials in defining drug effectiveness
Two further papers were received by TMJ.4,5 Both criticised the allegedly unbalanced acquisition of medical knowledge by overuse of randomised controlled trials. The claim is that alternative “observational” studies have advantages such as relative simplicity, lower direct costs and a capacity for data validity comparable to RCT. However, incorrect marketing decisions made from studies that are prone to hidden bias could involve considerable indirect costs from poorly directed therapies. The claim is also made that non-RCT designs produce data as good as that from RCTs. These were novel and to me illogical claims, as the ‘control’ in an RCT is designed to (and does) minimise bias. The current standing of RCTs has been summarised by Fernainy et al.62 Of interest is the idea, implied rather than explicit, that different areas of medical research and practice may have varying research design requirements. This leads to the specific question of the optimum study design for drug trials, but it appears the answer to this question remains unsettled. It is possible that regulatory bodies, in their deliberations on efficacy, toxicity and economic viability leading to drug registration represent a demand for a traditional RCT trial design that ought to be in a process of evolution.
Though many controls are applied in a trial context, none is more important than the prospective randomisation of a patient to either active treatment with the drug under study or placebo (or equivalent such as current standard practice). This single design component is an investigator’s best hope for baseline equivalence of the groups to be compared when the trial is over, providing insurance against important groups differences and increasing the validity of the statistical analysis and inferences as to causality.64 In simple terms, one hopes to compare like with like. The following designs, common in papers cited by c19hcq.org, are inadequate because they do not prevent bias:
- Retrospective uncontrolled observation of a condition in a population treated according to medical preference (but a controlled study in these circumstances may well provide useful information);
- Treating a population with or without individual consent and using the declining group as a “control”.
- Randomising treated patients into different groups based on assessment of risk or varied combination of treatments etc and describing the trial as “randomised”;
- Retrospective analysis of outcomes based on different treatments or in different populations with or without a ‘control’ group;
- Deliberate comparisons of outcomes in treated and untreated patients from different geographical areas or different ages or sex or pre-existing co-morbidities.
Dancis and colleagues5,63 proposed replacing the hierarchical model of value of various sorts of clinical information in which RCT is placed at a pinnacle, with a wheel entitled The Totality of Evidence-Based Medicine (T-EBM) Qualitative Wheel. This by definition flattens the traditional hierarchy in favour of a device in which, by implication, all trial designs including in vitro and other laboratory studies, structured analyses including meta-analyses, expert opinion and editorials etc, have the same standing as RCTs. I agree that all these sources of information play a part, but caution is required. For example, a drug such as HCQ that inhibits cellular uptake of a virus in a cell culture justifies further study, but the effect does not predict and cannot be taken as evidence in favour of its effectiveness in the treatment of patients with complex homeostatic mechanisms not present in vitro. In addition, whereas preclinical studies or “expert opinion” etc may be useful, unbiased clinical studies are imperative. Otherwise, how would one form an opinion?
It is legitimate to question the role of RCT and seek improvements in the design of drug trials, but one must not throw out the baby with the bathwater. In their balanced review of alternatives to randomised controlled trials, West and colleagues stated:65 “The RCT is the gold standard among research designs. It has the highest internal validity because it requires the fewest assumptions to attain unbiased estimates of treatment effects. Given identical sample sizes, the RCT also typically surpasses all other designs in terms of its statistical power to detect the predicted effect. Nonetheless, even with the best planning, the RCT is not immune to problems common in community trials. These threats potentially weaken the causal inferences.” In a recent paper on “target trial emulation” applied to observational studies, Hubbard and colleagues66 noted “Although observational studies permit investigation of questions that may be infeasible to study using RCTs and leverage the strengths of RWD, they have many potential sources of bias. The target trial emulation framework attempts to address some of these biases, but emulations must be undertaken and reported with care. Because of the risk of bias due to confounding, sensitivity analyses evaluating robustness of results to unobserved confounding are essential, and variability of results across alternative assumptions about confounding must be considered when interpreting results. When rigorously implemented, the target trial emulation framework provides a useful method for systematic specification of the design of observational studies, but is not a panacea.” Clearly, the danger in considering alternatives to RCTs is the risk that bias is increased, with obvious consequences for study reliability.
Mathematical solutions to uncertainty in observational clinical studies
Prof Eleftherios Gkioulekas and colleagues have described an ingenious mathematical approach56,58 applied to results from the studies by Zelenko8 and Gautret11 and others, which has the aim of permitting efficacy inferences to be drawn that traditionally would be criticised because of trial design bias. Reference 58 is a PowerPoint presentation of the method. The formal paper56 is another which deserves to be read widely, as it places the rationale for the mathematical models in a medical and historical context. The idea is that the usual and potentially invalid comparison of a measure, such as the number of deaths in treated patients versus an inadequate control group, is abandoned. Instead, the point of comparison becomes the measured rate of the same event in a known population, graded for severity of disease. The model calculates threshold values for the event rate that would be associated with a chosen probability (say, P<0.05 or 0.01 or 0.001), taking the size of the study into account. If the study rate is less that the threshold, the null hypothesis (that the drug has no effect) can be rejected with known probability. This is said to indicate that the preponderance of evidence is in favour of efficacy. If in addition the measured value is less than the known or adjusted community value, this is said to demonstrate clear and convincing evidence. The authors proposed a complementary Bayesean approach, and suggest a treatment pathway that promotes early use of a repurposed drug.
I have no doubt as to the validity of the complex mathematics deployed, but there are inherent assumptions that may be questioned. A low incidence of death in a study could be due to either treatment efficacy or results in a population at lower than expected risk. Also, epidemiological mortality rates represent the total cumulative percentage of persons in a stricken population who die with the disease, but in a trial it is more likely to be the number of deaths during a relatively short period of follow-up, which will give a different and non-comparable value. Public data is subject to collection errors and may be unreliable. For example, in reference 57 the author shows mortality data with Covid-19 infection by age, but the corresponding data in Australia for elderly patients are more than double (19% in 70-79 year-old Australians67 vs 8.0 and 7.96% in Italy and China respectively). This complicates the second class of inference mentioned above.
Published evidence of lack of benefit and toxicity of HCQ
The trials chosen for discussion to this point emerged because of the background debate between the author and Dr Clancy, who introduced the website c18hcq.org and chose a handful of trials he cited in support of HCQ. A different approach is to open the selection to specific literature searches to determine the grounds on which drug authorities withdrew support for HCQ. Three are possible: the threat to HCQ supply for patients with rheumatic diseases or malaria; lack of efficacy; or toxicity.
The uptake of HCQ rose rapidly during the first half of 2020. For example, the number of prescriptions across the US increased by 86.2% in only one month (February 2020) compared with pre-COVID levels.68 For a drug with a relatively small market (in countries unaffected by malaria), such a perturbation was a major supply risk for patients with chronic diseases, and which was recognised by regulatory authorities. For example, in Australia, supply of HCQ was limited by the Therapeutics Goods Administration.69 Such moves were not made for any reason other than to avoid supply disruption to existing patients.
The following remarks do not represent an in-depth review of the literature, but recent reviews are available.70,71 In an early review (April 2020),72 the authors lamented lack of statistical power and several other design weaknesses in the 5 trials published to that date, including studies referred to here.11,12 They called for urgent larger controlled trials. A month later, a second review (updated in August 2020 and July 2021)73 concluded strongly against 4-aminoquinolines because of possible increased mortality and toxicity. A further ongoing review from 2020–2022 has concluded that HCQ had no benefit.74 Large RCTs powered to detect important end-points in hospitalised (RECOVERY75 and WHO-SOLIDARITY76) and non-hospitalised21 patients followed, with negative results. Several meta-analyses77,78 also reported lack of effect, or toxicity related to prolongation of the QT interval or emergence of cardiac arrhythmias in HCQ treated groups.79,80
In signal contrast to these and other studies, the recently-published COPCOV Study81 reported a moderate but highly significant reduction in PCR-confirmed Covid infections in 4652 persons at risk of infection given either (in different world regions) HCQ or chloroquine compared to placebo.[1]
One potential problem pervades the negative results. As explained above and notwithstanding criticisms of the validity of the association between pharmacological mechanisms and therapeutic effects,73 the pharmacology of HCQ suggests that therapeutic benefit is most likely when it is given early in the course of the disease.16 However, there is lack of distinction between early or late treatment in most of the above studies. Thus the implied and inappropriate therapeutic question being asked was is HCQ efficacious at any stage of Covid infection? In a PICO analysis,82 this would translate into a definition of the population being studied as any person infected with SARS-2-CoV. This question was always likely to be answered in the negative, as the literature was biased in favour of large trials in hospitalised patients. Thus the studies do not definitely exclude the possibility of an effect in early stage disease. This possibility is minimised by the negativity of the findings for post-exposure HCQ prophylaxis as discussed above, but supported by the COPCOV study results.
Discussion
The failure to obtain a clear answer to the question of whether HCQ is of benefit in the treatment of Coved infection is disturbing. Why was there such a rash of inadequate and hurriedly executed trials of poor scientific validity? Were the drug regulatory bodies biased against repurposing old drugs? Was it because Ethics committees were incompetent or that their approvals were too onerous or took too long? Was there undue competition amongst researchers to win a race to document an effect? Was there, as Clancy claimed, antagonism from the WHO to the notion of drug repurposing, or partiality to a belief that the ultimate therapeutic solution could only be a vaccine? Was this partiality if it existed fuelled by the pharmaceutical industry? Or was it because the general negativity of individual trials was accepted as conclusively showing lack of effect? In 2020 there was no shortage of trial participants for enrolment in a properly designed randomised controlled trial, and had the international community co-operated or had national governments introduced a pre-formed policy of how to proceed under the circumstances of a pandemic, it could have saved many lives. It is surely possible to prepare for a future contagion by ensuring that such a contingency plan is agreed to. Of course, such a plan depends on early agreed identification of potential drugs, and on ensuring that adequate supplies can be made available quickly. Both are possible stumbling blocks.
The emergence of the COPCOV Study81 demonstrates a strong likelihood of a positive effect of HCQ in Covid primary prevention in persons at risk. I was struck by the relatively low HCQ dose of 200 mg daily (after loading doses). Perhaps a higher dose would have given a greater prophylactic effect. Second, the NNT for the primary endpoint is 54. Whether that would be sufficient for pharmacoeconomic purposes in a future pandemic is uncertain. Third, the paper assumes equivalence between HCQ and chloroquine, and analyses the data under an assumption that their efficacy and toxicity are identical. Reporting results for each drug separately would have been preferable. Nevertheless, I agree with the comment by Dr Schilling in his 2021 review71 to the effect that large randomised trials were required to conform an effect in early Covid disease. He has now provided that evidence, and deserves to be congratulated.
I wish to I thank Dr Robert Clancy and the authors of references 4 and 5 for the series of articles in TMJ and Quadrant that challenged me to study the HCQ evidence.
Provenance: Peer reviewed
Conflicts of interest: None declared
Ethical approval: Not required
Author contact: editor@tasmanmedicaljournal.com