Ladies and Gentlemen, we have our ruler

Dear readers, as promised earlier this week, it is my distinct pleasure to share with you today the headline results of this year's iGEM interlab study. These are preliminary results that we are still writing up for publication, but I feel that it is too exciting to keep quiet, and so I am shouting it from the rooftops today.

In synthetic biology, one of the best tools for studying the behavior of cells is fluorescence. Unfortunately we haven't had a good, accessible way of quantifying fluorescence, so you couldn't compare results from one lab to another, or even necessarily from one experiment to another within the same lab.  In short: we've needed a good ruler for measuring fluorescence.

The goal of the iGEM interlab studies has been to understand where the problems in measurement are coming from and then to use that knowledge to produce a good ruler. In the 2014 and 2015 interlab studies, we figured out that the big source of the problems didn't seem to be the biology, but how people were using their instruments and comparing their data. That was actually good news, because we had some ideas for how we might be able to fix that, and this year we tried them out. We gave every team two simple non-living calibration samples to compare their biological samples to, and hoped that this would tighten up the numbers some.

The results we got were beyond my wildest dreams.

Here's what we saw for the precision of measuring fluorescence with plate readers. We compared the standard deviation of the arbitrary unit measurements from 2015 with the calibrated measurements from 2016 before and after using the positive and negative controls for quantitative filtering of problematic tests, and got these standard deviations:

Smaller numbers are better, so you can see that we got a big improvement in accuracy from calibrating, and another big improvement from using the fact that we have real numbers to quantitatively exclude obvious protocol failures (e.g., 1000 uM FITC/OD fluorescence from a negative control).

But to really wrap your head around how big an improvement this is, you have to think about the fact that the thing that we are measuring is geometric standard deviation, for which the units are in times multiplied or divided.  In general, we would consider the normal range of values to expect from a measurement to be within two standard deviations up or down from whatever the real number is---i.e., 95% of the time, a measurement should lie within that range. With a geometric standard deviation of 35, two standard deviations up is 35*35 = 1,225.  That's more than a thousand.  Going down is another multiple of more than a thousand, meaning that all told, we would expect measurements to be accurate within a factor of a million or so.  Obviously, that's rubbish.  It's hard to do anything if you expect your measurements to be wobbling around by a factor of a million.

This year's measurements were more than 100,000 times more precise.

OK, you say, that's all well and good, but since the units were arbitrary before, nobody ever claimed that these numbers should be the same in the first place.  Many people who measure fluorescence and don't calibrate their measurements try to deal with this problem by normalizing to a positive control. The idea is then that you can say: "This is 2.7 times the control" and somebody else can hopefully measure the same control and find the same ratio.  Indeed, in last year's interlab we got pretty good results for comparing ratios of strong promoters, but we got terrible results for comparing the weak ones.

Here's the thing, though: remember that only about half of our improvement came from using the same units, while the rest came from being able to identify failures by the strange behavior of their controls.  We would thus expect to see significant improvement in precision this year versus last year, and indeed that is what we saw:

On average, normalized measurements were 70 times more precise.

The flow cytometry results are not quite as dramatic, and not as statistically strong since many fewer teams to took flow cytometry data, but the basic result is the same: orders of magnitude improvement in precision of both individual measurements and of normalized measurement.

There's a lot more to do, to go from initial results to routine and effective usage, but I believe the core results are clear:

1. We've got a workable ruler for fluorescence.  It's not perfect, but it's orders of magnitude better than current practices.
2. If undergraduates and high school students from all around the world can use these methods, there's no reason they can't be adopted in every biology laboratory that measures fluorescence.