Part 1 of this series can be found here.

I have deliberately kept mathematical 'proof' and academic rigour of the theories of Wisdom Of Crowds and the related Efficient Market Hypothesis out of this article, writes Russell Clarke. Those who are interested can easily research further their efficacy online. For what it's worth, I believe both theories have limited real world applications, though their usefulness in sporting prediction markets is undeniable.

A brief definition of the Wisdom of the Crowd is that large groups of people are collectively smarter than individual experts at predicting outcomes. Explanations of the wisdom of the crowd are numerous but the Diversity Prediction Theorum attempts to mathematically quantify via the definition, “the squared error of the collective prediction equals the average squared error minus the prediction diversity”.

In layman’s terms, when group of predictors is large and diverse, the error is small. There are more complex layers to add to the wisdom of the crowd theories and explanations involving independence, bootstrapping and other exotically named theories, but for our purposes, we will omit the bells and whistles of academia. This is simply about, to misquote Jeremy Corbyn, “why the many are smarter than the few”. It is especially true when the crowd is diverse and independent, which is very much the situation in betting markets.

It has been demonstrated in numerous studies that the crowd is particularly accurate in the fields of quantity estimate, general world knowledge and spatial reasoning. If we look at quantity estimate, I saw a programme on this subject where office workers were asked to guess the number of sweets in a large jar. The estimates had a huge range and yet the average was just 4 sweets from being correct! More famously, at a 1906 Plymouth Fair, 800 people were asked to guess the weight of an ox and the average was within 1% of the actual weight. I know, I need to get out more…

Related to Wisdom Of Crowds is the Efficient Market Hypothesis. The EMH in its simplest form suggests that asset prices reflect all available information (and thus, by association, it is impossible to beat the market). The latter conclusion is a stretch of the theory, particularly in sports betting.

So, what are the implications of this theory when we look at, for example, horse racing? I have evidence that in recent times a real sea-change has occurred in the racing markets and this has been caused by the increasing wisdom of the crowd. It has gone largely unnoticed as it has been gradual and marginal. However, it has been incremental and, as a result, the marketplace today is very different from that of even a few years ago.

Let me rewind to a time when starting prices were produced by the on-course betting market. A few “good men and true” would form a huddle at the 'off' of each race and compare the prices they saw offered by the bookmakers. They came to an agreement or average and that was declared as the starting price. This SP was basically the result of supply and demand in the on-course marketplace (racecourse punters and the major bookmaking offices who sent cash to the course to reduce the prices of horses that they had large liabilities on). This method was later replaced by a similar method, but one which included more on-course bookmakers.

However, the methodology is not of major importance. The SP’s were still, in theory, a result of supply and demand mechanics within the racecourse crowd. The rise and rise of betting exchanges and, crucially, their use by virtually every racecourse bookmaker means that is no longer the case. Today, the SP’s are a reflection of the betting activity on the exchanges rather than the activity on a racecourse. Suddenly, the crowd is no longer a few hundred punters on a racecourse, it is tens of thousands on an exchange. The new crowd is better informed, more diverse and greater in number. The wisdom of the crowd has increased.

If we accept the aforementioned theories at face value, the best approximation of the chance of an outcome would, in horse racing, be the Betfair Starting Price (BSP) and, in football, the Asian Handicap closing lines. That is because those markets are the largest, deepest and smartest markets for those individual sports. The participants in those marketplaces are diverse, independent and largely devoid of any 'group think'.

In both of these markets there is virtually no margin to account for and so the final prices (once every participant has eventually 'voted') can be readily converted into a percentage chance of that outcome actually happening. A BSP of 2.0 represents a 50% chance, 3.0 represents a 33% chance, 5.0 represents a 20% chance etc. Similarly, Asian Handicap Lines can be converted into % chances for football betting. Numerous empirical studies have shown both to be almost wholly accurate.

I realise I have ‘banged on’ a bit here, but, the importance of this knowledge cannot be overstated. It demonstrates the futility of trying to beat the market when it is at its most accurate. In plain English, it is arrogant in the extreme to believe you know more than the market at the closing and you will eventually find out that it pays to be humble! If you bet at BSP (Betfair Starting Price), the commission is likely to ensure you are a long-term loser (although it is a more favourable strategy than betting with bookmakers at SP with their much higher margins than the exchange commission). If you accept that logic, then it is clear that you should be betting early, when the market has less participants and is therefore less accurate.

Another use for the EMH is if you want to accurately assess systems, strategies or the records of tipsters/experts. It is a quicker and faster way to assess than simply looking at a profit/loss account, which can be wildly erroneous. So, traditionally, even those that do their research, will look at a series of results and concentrate on factors such as profit/loss, strike-rate, longest losing run, taken from a set of past results. On the surface this seems logical and sensible. However, the downside is that you will almost certainly be dealing with an inadequate sample size (again, if you need the maths, then an online check) and even if you have thousands of results, a simple Monte Carlo simulation will demonstrate the huge variance in results you could experience moving forward (more of which anon).

Using our appreciation of the accuracy of the markets, we can gain a quicker and more accurate guide to how a strategy will perform in the future and in the longer-term. We can ignore profit/loss figures and instead concentrate on how the selections (winners and losers) perform against the market. There are a few criteria you could apply but a very simple method is demonstrated below:

Two figures you require are the price at which the selection is advised (or a morning price) and the eventual BSP. Then it becomes a simple comparison. If a horse is advised at 10/1 (11.0 digital odds) and the bsp is 7.0, then that would be assessed as +4 (11-7). Similarly, a horse advised at 8/1 (9.0) and the bsp is 9.0 would be assessed as 0 (9-9) and a horse advised at 12/1 (13.0) that has an eventual bsp of 18.0 would be -5 (13-18).

After as few as fifty bets you would get a good reading of the number of selections that are positive as opposed to negative, and, the running total would give an indication of the magnitude of the long-term profits/losses that are likely. The actual results and profits/loss are largely irrelevant as they may just reflect either a favourable or unfavourable run of winners/losers. You can be sure, however, that if you continue to beat the "closing line” you have unearthed a source of long-term profit.

- RC

Next week: Part 3: Bookmakers - Sharps and Softs