A look back at post hoc analysis

People who claim that hindsight is 20/20 probably weren’t talking about the post hoc evaluation of clinical testing. That’s an alleged flaw the FTC challenged in purported substantiation submitted by the marketers of Hepaxa and Hepaxa PD, fish oil products advertised as clinically proven to treat liver disease in adults and children. In addition to injunctive provisions, the proposed settlement includes a financial remedy of $416,914.

According to the FTC, German corporation BASF SE, which developed and owns Hepaxa and Hepaxa PD, acted through its North American subsidiary, BASF Corporation, to retain Michigan-based DIEM Labs to advertise and distribute Hepaxa and Hepaxa PD in the United States. Ads pitched the products as effective treatments for Non-Alcoholic Fatty Liver Disease – NAFLD – a serious condition involving the excessive build-up of fat in the liver from causes other than alcohol use.

“CHANGE IS HARD. The solution should be EASY.” That was one eye-catching slogan the respondents used to pitch Hepaxa. They also made dramatic promises about how adults and kids would benefit from the products, a monthly supply of which went for about $48. In ads with the heading “CUT THE LIVER FAT, the respondents represented that most adult and pediatric NAFLD patients “will experience benefits after six months of daily supplementation” with Hepaxa or Hepaxa PD (the kids’ version). A news release issued by BASF claimed that a “BASF clinical trial reveals significant reduction in liver fat content in patients with non-alcoholic fatty liver disease.” The respondents disseminated those claims on the Hepaxa-USA website, as well as on Twitter and LinkedIn.

One notable feature in the marketing was detailed information about clinical testing on the product, including a “randomized placebo-controlled clinical trial using Hepaxa.” You’ll want to read the complaint for details about the FTC’s allegations, but here’s the short version.

A clinical trial was designed to evaluate whether Hepaxa raises levels of omega-3 polyunsaturated fatty acid (PUFAs) in red blood cells and reduces liver fat in adults with NAFLD. For six months, 81 subjects in the treatment group received Hepaxa, while 86 subjects in the control group took an olive oil placebo. All study participants were advised to reduce their calorie intake and to maintain stable physical activity levels. Using MRIs, 60 subjects in each group were measured for liver fat. At the end of the study, the MRI data showed no statistically significant reduction in liver fat in the Hepaxa patients, when compared to the placebo patients. You read that right – no statistically significant reduction.

What happened next gets to the FTC’s allegation that the respondents, in effect, deceptively sliced and diced the data in search of a positive marketing message. According to the complaint, the respondents subjected the data to post hoc analyses of different subgroups of test subjects. (The complaint describes a post hoc analysis as “a statistical analysis conducted after the data have been collected in hopes of discovering statistical relationships that suggest cause and effect.”) The FTC’s concern is that unplanned, post hoc subgroup analyses pose a high risk of generating spurious findings.

Faced with clinical testing that showed no statistically significant results for those who used their product, the respondents took another look at the people who participated in the study, this time separating them out by a different measure: their Fatty Liver Index. That score derives from an algorithm combining subjects’ waist measurement, body mass index, triglyceride levels and a specific liver enzyme. Using that after-the-fact criterion, people with a baseline Fatty Liver Index of 40 or over did experience a statistically significant reduction in liver fat after using Hepaxa, as compared to placebo. But not so fast. That reshuffled group included only five Hepaxa patients and 12 placebo patients – a substantial decrease from the 120 people in the study who got MRIs to measure liver fat.

Based on that reworking of the data from the unsuccessful study, the respondents made their claims about the effectiveness of Hepaxa and the purported clinical testing behind it. But according to the FTC, the respondents haven’t conducted any other clinical trials of Hepaxa and haven’t tested Hepaxa PD on kids at all. Indeed, there are no competent and reliable human clinical trials of products that are the same as Hepaxa. What’s more, other liver fat studies cited by the respondents relied on omega-3 PUFAs from sources different from what’s in Hepaxa or used omega-3 PUFAs not found in those products.

Among other things, the proposed settlements require the respondents to support future claims about the treatment or cure of any disease with competent and reliable scientific evidence in the form of randomized human clinical testing. In addition, BASF SE and DIEM Labs will pay $416,914 for consumer redress. Once the proposed settlements appear in the Federal Register, the FTC will accept public comments for 30 days.

The settlement demonstrates the risks of relying on post hoc data analyses to substantiate advertising representations. Reworking the results until you find a potential marketing nugget is an ill-advised strategy. The wiser response to clinical testing that doesn’t yield the results you want is to rethink the nature of your product and the claims you convey to consumers.