Data From

Analyzing Replicated Experiments

Data From Fig 2b – ER calcium depletion as a key driver for impaired ER-to-mitochondria calcium transfer and mitochondrial dysfunction in Wolfram syndrome

nested subsampling

linear mixed model

ggplot

simulation

Fig 2b contains four replicates of an experiment, each with four treatment levels. The data were analyzed with an ANOVA model. There is nothing unusual about this. This is too bad because replicated experiments are examples of Randomized Complete Block Designs, where each experiment is a block. Analyzing blocked designs using best-practice models increases power, so the usual ANOVA model is a lost opportunity. What is highly unusual about this publication is the researchers identified the experiments in the archived data, which means I can reanalyze the data with a best practice model, either a linear mixed model or an equivalent repeated measures (RM) ANOVA. The design also contains nested technical replicates, and ignoring this is pseudorpelication, leading to increased Type I (false discoveries) error. A simulation of the Fig 2b data shows that the Type I error rate is about three times the expected rate and only with extremely low ICC does Type I error rates approach the expected rate.

Models for proportion (part to whole percent) data: Part II

Data From Fig 2d – Inhibition of fatty acid oxidation enables heart regeneration in adult mice

percents

proportions

generalized linear model

offset

power

simulation

The researchers present two tests/plots: the effect of genotype on fibrotic area and the effect on the percent of fibrotic area, relative to total heart area. A GLM with offset combines these into a single analysis/plot and has more power than the linear model/t-test or Mann-Whitney U.

Models for proportion (part to whole percent) data

Data From Fig 5c – CTLA-4-expressing ILC3s restrain interleukin-23-mediated inflammation

percents

proportions

generalized linear model

offset

power

simulation

The data in fig 5c are the percent of a cell subtype relative to all cells of a type, which is a common response variable in bench biology. The researchers analyzed this part-to-whole (a proportion) response with a Mann-Whitney U test. A better practice method is a GLM of the count of the part using the count of the whole as an offset. The effect is the ratio of the treatment proportion relative to the control proportion. This post also shows how to plot the proportion with estimated means and CIs from the GLM with offset model.

Hot take: don’t adjust p-values

Data From Fig 7d – LSD1 inhibition circumvents glucocorticoid-induced muscle wasting of male mice

p-values

multiple testing

generalized linear model

absurd CI

power

simulation

pseudoreplication

analysis flag

Two interesting bits with Fig 7d: 1) The researchers analyzed figure 7b data using one-way ANOVA with Tukey adjustment. Post-hoc p-value adjustment is common but typically unjustified for bench biology data like these. The example is a good one to talk about p-value adjustment in bench biology. 2) The linear/ANOVA model implies absurd inference from the CI/SE error bars, including mean counts less than zero!

A LMM or GLMM alternative to ratio paired t-test is the best practice to plot the model

Data From Fig 3b – The TAS1R2 G-protein-coupled receptor is an ambient glucose sensor in skeletal muscle that regulates NAD homeostasis and mitochondrial capacity

linear mixed model

ratio paired t-test

generalized linear mixed model

ggplot

The researchers analyzed figure 3b with a ratio paired t-test, which is a special case of a linear mixed model with a log-transformed response. A generalized linear mixed model with a Gamma distribution and log link is an alternative. An advantage to fitting the LMM or GLMM is asymmetric confidence intervals of the group means that reflect the skew of the data. Plot the model!

Fig 2s – Pharmacological inhibition of HDAC6 improves muscle phenotypes in dystrophin-deficient mice by downregulating TGF-β via Smad3 acetylation

linear mixed model

random intercept

CRDS

nested ANOVA

ggplot

Fig 2s is a Completely Randomized Design with subsampling (CRDS), with 28-49 technical replicates per mouse. The researchers used a Mann-Whitney test that ignored the technical replication. This analysis is massively pseudoreplicated because techanical replicates are not independent evidence of effect of treatment. Individual mouse ID was archived, which makes it easy to re-analyze with models that account for the non-independence. The p-value accounting for non-independence is 0.18 while the p-value using Mann-Whitney on all data is 0.0002. Whoops!

Fig 3g – The role of somatosensory innervation of adipose tissues

linear mixed model

random intercept

split plot

ggplot

Fig 3g is a split-plot design. I often see researchers analyze these using separate paired t-tests. This isn’t so bad, but is limiting, and a better alternative is a linear mixed model. But does this replicate the Repeated Measures ANOVA in GraphPad used by the researchers? It does! Yaaay!

Fig 4h – FOXO1 is a master regulator of memory programming in CAR T cells

technical replicates

pseudoreplication

nested ANOVA

CRDS

analysis flag

Fig 4h contains thousands of technical replicates. Technical replicates should not be analyzed as if they were randomly assigned to one of the experimental treatments. They weren’t. The resulting p-value is junk.