The Signal and the Noise – Why so many predictions fail
by Nate Silver
Last Friday, the CU Book Club invited Prof Kwong Chung-ping to give a talk on Nate Silver’s best selling book The Signal and the Noise – Why so many predictions fail. I was interested on how to differentiate signal and noise, but the emphasis of the talk was not that. Rather, Prof Kwong focused on the failure of predictions. Although this was only the sub-title of the book, the underlying thought of Nate Silver was heavily connected to it.
To predict an unknown outcome, we basically depend on probability. However, probability is not certainty. Mathematically, we could derive the probability of an outcome. But the reality may not yield the calculated outcome. It is only on theory that such outcome would definitely appear if the event is repeated for infinite times. Thus the truth of probability could be said as just a belief. Over the years, men had been trying to improve mathematical models with the hope of improving predictions. The fundamental flaw of such thinking is that effort is spent to find out repeated patterns in past data, whereas such data may not have any causality with future outcome. Thus the basis of technical analysis or chart analysis of stock market movement does not have solid grounds.
For complex dynamical systems like the stock market, politics or weather, the forces which could affect outcome are so complex that they are unknown to us. The efficient market could react to changes instantly, but to observers the outcome could appear at random. Prof Kwong demonstrated a chart generated by random walk movements. The shape of the chart highly resembles the movement of stock market over a certain period. He further showed a method by Brownian Motion, said it was random walk at higher frequency. It is not my understanding of Brownian Motion. Brownian Motion is a form of random walk where the random outcome would depend on the position of the previous event. But the formula shown by Prof Kwong did show a time series consisting of the difference between two consecutive events. This method was further developed into the Fractal Brownian Motion where the Hurst index is included. Despite all these mathematical development, the conclusion is that the predictions generated by all of them failed.
The Bayes Theorem was said to be the basic explanation of such hope and failure. The theorem proved that the probability of an event could be changed if the probability of a preceding event is taken into account. This leads to attempts that the known outcome of previous events could affect, or improve, the probability of a later event. Thus what happened previously in the market could affect the probability of stock movement. However, there is no way to know whether it is correct to put in whatever probability of previous events.
My understanding of Bayes Theorem is different from this aspect. On probability, there are two different types. One is frequentist probability where the probability is confined to the frequency of events or the probability space. Tossing a coin has a probability space of 2, where a dice is 6. However, there is also the Bayesian probability where the probability space is not defined, and the probability is based on the belief of the observers. We can calculate that the probability of one side of a dice is 1/6 by experiments. However, take the example of the risk of the raid on Bin Laden, there is no way that experiments could be performed to test the probability. What can be done is the arbitrary assessment of the command team. Each member of the team may have a belief of the success probability of the action, and the final probability could be an average of such belief. In real life, we encounter Bayesian probability more often than frequentist probability because in most cases the probability space is not easy to determine.
We often fall into the trap of confusing these probabilities and make wrong predictions. Sometimes when we consider the probability of an event and we think of all the possible scenarios and assess the best chance of an action. However, for such Bayesian probability with undefined probability space, there is a high chance that many possible scenarios are ignored. The decision could just be a belief that the action would succeed. On the other hand, we often ignore probability space and frequentist probability, and overestimate probability. One example is that we always ignore the frequentist probability of Mark 6 first prize which is astronomically low, but believe in a Bayesian probability that the chance is not low because someone won first prize every week.
The June 2013 issue of Scientific American has an article on the application of Bayesian theory to Quantum Mechanics. Very simply put, Quantum Mechanics is based on probability of movement of electrons. The mathematical formula proved that all probabilities of an event would occur simultaneously in parallel until the outcome is observed, then it collapses into one reality. Thus the Schroedinger’s Cat is both alive and dead until the box is opened. The new Quantum Bayesianism, still in debate, argues that the Bayesian probability used is only a belief. The simultaneous existence of probable events only appears in mathematical thought and is not real. The reality is a dead or alive cat but not a dead and alive cat. This is an emerging branch of Quantum Mechanics which we may hear more about in the next few years.
