A really interesting paper on making crowds smarter.  I find the word 'confident' in the title a bit odd because it seems (and I may have misunderstood) that the researchers are actually trying to measure independent thinking rather than confidence. As far as I can tell, this describes a method for separating sheep (those more influenced by others) from goats (those making more independent decisions), at least when you have a sequence of decisions/judgments to work with. The reason it bothers me is that sheep can be confident too (see the US election or Brexit, for example).

We know that crowds can be wise if and only if the agents in the crowd are unaware of the decisions of other agents. If there's a feedback loop (more accurately, I believe, if there is an insufficiently delayed feedback loop) then you wind up with stupid mobs, driven by preferential attachment and similar dynamics. This is a big problem in many political systems that allow publication of polls and early results. However, some people are, for one reason or another, less influenced by the crowd than others. It would be useful to be able to aggregate their decisions while ignoring those that simply follow the rest, in order to achieve wiser crowds. That's what the method described here seeks to do.

The paper is more concerned with describing its model than with describing or analyzing the experiment itself, which is a pity as I'd like to know more about the populations used and tasks performed, and whether it really is discriminating confident from independent behaviour. I've also done some work in this area and have written about how useful it would be to automatically identify independent thinkers, and to use their captured behaviour instead of that of the whole crowd to make decisions, but I have never implemented that because, in real life, this is quite hard to do. In this experiment, it seems quite possible that the 'independent' people might simply have been those that knew more about the domain. That's great if we are using a sequence of captured data from the same domain (in this case, length of country borders) because we get results from those that know rather than those that guess. But it won't transfer when the domain changes even slightly: knowing the length of the Swiss border might not well predict knowledge of, say, the length of the Nigerian border, though I guess it might improve things slightly because those that care about such things would be better represented in the sample.

It would take a fair bit of evidence, I suspect, to identify someone as a context-independent independent thinker though, given enough time, it could be done, it would be well worth doing, and this model might provide the means to identify that. I'd like to see it applied in a real context. There are less lengthy and privacy-invading alternatives. For instance, we might capture both a rating/value/judgement/whatever and some measure of confidence. Some kinds of prediction market capture that sort of data and, because of the personal stake in it, might achieve better results when we do not have a long history of data to analyze. Whether and to what extent confidence is related to independence, and whether the results would be better remains to be discovered, of course - there's a good little research project to be done here - but it would be a good start.