Markets can be “inefficient” and yet almost impossible to beat because of the existence of “Limits to Arbitrage” . It is essential not only to have the correct view but also to know when the view will be realised.
Why is it so difficult to time the market? Because the market is a complex adaptive system and complex adaptive systems are amenable only to what Hayek called “pattern predictions”. Hayek introduced this concept in his essay “The Theory of Complex Phenomena” where he analysed economic and other social phenomena as “phenomena of organised complexity” (A term introduced by Warren Weaver in this essay).
“We can build up beautiful theories which would explain everything, if we could fit into the blanks of the formulae the specific information; but we never have all the specific information. Therefore, all we can explain is what I like to call “pattern prediction.” You can predict what sort of pattern will form itself, but the specific manifestation of it depends on the number of specific data, which you can never completely ascertain. Therefore, in that intermediate field — intermediate between the fields where you can ascertain all the data and the fields where you can substitute probabilities for the data–you are very limited in your predictive capacities.”
“Our capacity of prediction in a scientific sense is very seriously limited. We must put up with this. We can only understand the principle on which things operate, but these explanations of the principle, as I sometimes call them, do not enable us to make specific predictions on what will happen tomorrow.”
Hayek was adamant however that theories of pattern prediction were useful and scientific and had “empirical significance”. The example he drew upon was the Darwinian theory of evolution by natural selection, which provided only predictions as to the patterns one could observe over evolutionary time at levels of analysis above the individual entity.
Hayek’s intention with his theory was to debunk the utility of statistics and econometrics in the forecast of macroeconomic outcomes (See his Nobel lecture). The current neoclassical defense against their inability to predict the crisis takes the other extreme position i.e. our theories are right because no one could predict the crisis. This contention explicitly denies the possibility of “pattern predictions” and is not a valid defense. Any macroeconomic theory should be capable of explaining the patterns of our economic system – no more, no less.
One of the key reasons why timing and exact prediction is so difficult is the futility of conventional cause-effect thinking in complex adaptive systems. As Michael Mauboussin observed, ” Cause and effect thinking is futile, if not dangerous”. The underlying causes may be far removed from the effect, both in time and in space and the proximate cause may only be the “straw that broke the camel’s back”.
Many excellent examples of “pattern prediction” can be seen in ecology. For example, the proximate cause of the catastrophic degradation of Jamaica’s coral reefs since the 1980s was the mass mortality of the dominant species of urchin (reference). However, the real reason was the progressive loss of diversity due to overfishing since the 1950s.
As CS Holling observed in his analysis of a similar collapse in fisheries in the Great Lakes:
“Whatever the specific causes, it is clear that the precondition for the collapse was set by the harvesting of fish, even though during a long period there were no obvious signs of problems. The fishing activity, however, progressively reduced the resilience of the system so that when the inevitable unexpected event occurred, the populations collapsed. If it had not been the lamprey, it would have been something else: a change in climate as part of the normal pattern of fluctuation, a change in the chemical or physical environment, or a change in competitors or predators.”
The financial crisis of 2008-2009 can be analysed as the inevitable result of a progressive loss of system resilience. Whether the underlying cause was a buildup of debt, moral hazard or monetary policy errors is a different debate and can only be analysed by looking at the empirical evidence. However, just as is the case in ecology, the inability to predict the time of collapse or even the proximate cause of collapse does not equate to an inability to explain macroeconomic patterns.