This video is interesting but is misleading.

It reports that the average estimate of one hundred and sixty people asked to guess the number of jelly beans in a jar was just 0.1% higher than the actual number of 4510.

But one person estimated the number at 50,000, ten times the actual number. So if that person had not been included in the survey, the average estimate would have been 312 beans lower than the actual number, a 6.9% underestimate.

So an important question not addressed in the video is how representative was the result obtained in the particular sampling of 160 guesses?

To answer that question, it would be necessary to repeat the experiment many times. We would then have an idea of the variability in, or reliability of, the “wisdom of the crowd.”

But even without such an estimate, we can be pretty sure that the error is usually larger than that reported in the BBC’s single trial, given the dependence of the “accuracy” of that result on just one extremely wild guess.

A further question is how consistent is the “wisdom of the crowd” on a variety of tasks. Guessing the number of jelly beans in a jar is not that difficult. The jar is a cylinder with dimensions that can be fairly accurately estimated by eye. The volume can therefore be calculated and one then need only estimate the number of jelly beans per cubic inch or centimeter and multiply that number by the volume in the same units to get a total for the jar.

But what if the container were an irregular shape: a jar in the shape of a kangeroo, for example? Would the estimate of the crowd be as good? What if the container were very much larger or the contents very much smaller? For example, how good would the crowd be in estimating the number of grains of sand in a life-size sand sculpture of a blue whale or an elephant?

And what of the technical expertise of the crowd? Would a crowd of engineers make a better estimate than a crowd of music teachers?

Typically, the BBC fails to ask these and many other interesting and important questions. Instead, it presents one with a little nugget of information of dubious validity to squirrel away without further reflection.

But perhaps a reader can help out. How, for example, does this kind of experiment tie in with Bayes theorem and the value of guesswork in drawing inferences from limited data? Comment from all Bayesians wecome!

Advertisements