Markets can be "inefficient" and yet almost impossible to beat because of the existence of "Limits to Arbitrage" . It is essential not only to have the correct view but also to know when the view will be realised.

Why is it so difficult to time the market? Because the market is a complex adaptive system and complex adaptive systems are amenable only to what Hayek called "pattern predictions". Hayek introduced this concept in his essay "The Theory of Complex Phenomena" where he analysed economic and other social phenomena as "phenomena of organised complexity" (A term introduced by Warren Weaver in this essay).

In such phenomena, according to Hayek, only pattern predictions are possible about the social structure as a whole: As Hayek explained in an interview with Leo Rosten:

"We can build up beautiful theories which would explain everything, if we could fit into the blanks of the formulae the specific information; but we never have all the specific information. Therefore, all we can explain is what I like to call “pattern prediction.” You can predict what sort of pattern will form itself, but the specific manifestation of it depends on the number of specific data, which you can never completely ascertain. Therefore, in that intermediate field — intermediate between the fields where you can ascertain all the data and the fields where you can substitute probabilities for the data–you are very limited in your predictive capacities."

"Our capacity of prediction in a scientific sense is very seriously limited. We must put up with this. We can only understand the principle on which things operate, but these explanations of the principle, as I sometimes call them, do not enable us to make specific predictions on what will happen tomorrow."

Hayek was adamant however that theories of pattern prediction were useful and scientific and had "empirical significance". The example he drew upon was the Darwinian theory of evolution by natural selection, which provided only predictions as to the patterns one could observe over evolutionary time at levels of analysis above the individual entity.

Hayek's intention with his theory was to debunk the utility of statistics and econometrics in the forecast of macroeconomic outcomes (See his Nobel lecture). The current neoclassical defense against their inability to predict the crisis takes the other extreme position i.e. our theories are right because no one could predict the crisis. This contention explicitly denies the possibility of "pattern predictions" and is not a valid defense. Any macroeconomic theory should be capable of explaining the patterns of our economic system - no more, no less.

One of the key reasons why timing and exact prediction is so difficult is the futility of conventional cause-effect thinking in complex adaptive systems. As Michael Mauboussin observed, " Cause and effect thinking is futile, if not dangerous". The underlying causes may be far removed from the effect, both in time and in space and the proximate cause may only be the "straw that broke the camel's back".

Many excellent examples of "pattern prediction" can be seen in ecology. For example, the proximate cause of the catastrophic degradation of Jamaica's coral reefs since the 1980s was the mass mortality of the dominant species of urchin (reference). However, the real reason was the progressive loss of diversity due to overfishing since the 1950s.

As CS Holling observed in his analysis of a similar collapse in fisheries in the Great Lakes:

"Whatever the specific causes, it is clear that the precondition for the collapse was set by the harvesting of fish, even though during a long period there were no obvious signs of problems. The fishing activity, however, progressively reduced the resilience of the system so that when the inevitable unexpected event occurred, the populations collapsed. If it had not been the lamprey, it would have been something else: a change in climate as part of the normal pattern of fluctuation, a change in the chemical or physical environment, or a change in competitors or predators."

The financial crisis of 2008-2009 can be analysed as the inevitable result of a progressive loss of system resilience. Whether the underlying cause was a buildup of debt, moral hazard or monetary policy errors is a different debate and can only be analysed by looking at the empirical evidence. However, just as is the case in ecology, the inability to predict the time of collapse or even the proximate cause of collapse does not equate to an inability to explain macroeconomic patterns.

Comments

One of the key reasons why timing and exact prediction is so difficult is the futility of conventional cause-effect thinking in complex adaptive systems. | Midas Oracle .ORG

[...] One of the key reasons why timing and exact prediction is so difficult is the futility of conventional cause-effect thinking in complex adaptive systems. Written by Chris F. Masse on 2010/01/07 — Leave a Comment Efficient Markets and Pattern Predictions [...]

chaos

Hayek was right at that time, and is still largely right today. However, we're getting better at making specific predictions about the behavior of complex adaptive systems with techniques like machine learning. If progress continues, I think this idea will be less relevant. Someone might object to this that new predictive technology will be applied by the market and so the market will remain just as difficult to predict. While this is a legitimate possibility, to insist that it will certainly happen would be to oversimplify the nature of the economy's complexity. New predictive technology will not necessarily be equally accessible to everyone. Even if it is, this does not imply a return to square one, as various sectors will be impacted in different ways and to different extents by such innovations. While I agree with the intuition that new predictive technologies will have difficulty predicting their own influence on the economy, the intuition is nonetheless crude. New technologies might not predict all the consequences of their use accurately, but we should not deny the possibility they'll be capable of predicting some such consequences to at least an approximate degree.

chaos

Continued (I wish you had an edit button): Recursive predictions are possibly going to be the most difficult-to-solve problems in macroeconomics. But if there's anything that going to be able to deal with such problems effectively, it seems like it will be a predictive computer program, as handling recursion is one of programming's greatest strengths. This will not happen automatically nor is it a certainty. But the possibility, at least, seems undeniable, and in my eyes the probability as well is high enough that this is worth talking about, given that the consequences would be unprecedented and huge.

chaos

Continued again: You mention ecology as an example of a complex adaptive system where making predictions is nearly impossible, and I agree. However, ecologists have made strides in making predictions about environments and the effects that some interventions might have. Epidemiologists can tackle issues of disease spread under the conditions of several variables and various scenarios quite well now. And climatologists are working very hard on predicting the precise consequences that will result from greenhouse gas emissions and the feedback loops of warming on local and global systems. These fields are not perfect. But they are interesting and making progress, and so they're worth learning from.