We live in a world which is incalculably more complicated than ever before, which produces an astounding range of products, services and satisfactions. Wealth is being created faster than was imaginable a few generations earlier, even if it is still far from being perfectly distributed. But it had also been imagined that we would have an AIDS vaccine and affordable electric cars by now.

In fact, the most popular move in the direction of green cars has had consequences that make back-pedalling look desirable: production of ethanol has fed into a food price crisis. Universities and patents etc are overflowing, but here too the boon of complexity has produced a mixed bag. The number of patents filed per research dollar is falling. The sheer volume of knowledge that must be mastered means that it?s taking longer to get doctorates, which is reducing researchers? productive lifespans. Sure,

Apple and Google enable a wider and more inclusive range of experimental possibilities, but not everything can be whipped up in student dorms. Tim Harford takes this argument even further, concluding that we are running to stand still when it comes to producing new ideas.

In one episode of his More or Less programme for BBC Radio 4, he answers questions ranging from whether public sector workers or private sector workers are more hard-done-by to how many Catholics really go to mass. He is also the Undercover Economist for Financial Times. As an economic journalist, Harford?s forte is to a) span the world of ideas with interdisciplinary curiosity?across evolutionary biology, anthropology, physics and maths for example, and b) rise over what would be muddle for ordinary muggles to offer common-sense approaches to the most pressing problems facing us today. Perhaps the best plaudit for his latest endeavour on this front, Adapt: Why Success Always Starts With Failure, has come from New York Times? David Brooks who has declared, ?For the good of the world, a bigger slice of humanity should be aware of its contents.?

Quite a big slice of humanity loves Coca-Cola. Andy Warhol put his finger on why this fuzzy drink is so seductive, why we wish schools, hospitals and public transport could deliver with its unwavering dependability: ?You can be watching TV and see Coca-Cola, and you know that the President drinks Coke, Liz Taylor drinks Coke, and just think, you can drink Coke, too. A Coke is a Coke and no amount of money can get you a better Coke than the one the bum on the corner is drinking. All the Cokes are the same and all the Cokes are good.? But in a world with ever-shifting standards and ever-new disputes, it is not the uniformity of Coke??all identical, all good??but variety of experiments and failures that?s the norm.

Politicians want experiments to deliver results within a term, so they don?t find it convenient that over half of the pilot schemes will fail, not surprising when you consider that 10% of American companies disappear every year. Harford argues, ?What is striking about the market system is not how few failures there are, but how ubiquitous failure is even in the most vibrant growth industries.? While few boardroom welterweights would admit it, ?we are blinder than we think?. We learn only by trial and error. In market economies, in fact, failure seems to go hand in hand with rapid progress even though we tend not to remember the companies that don?t make it. This is a price worth paying. ?We don?t expect every lottery ticket to pay a prize, but if we want any chance of winning that prize then we buy a ticket.?

Consider climate change, where you see an impasse since Copenhagen on the global policy front but some countries are still showing the courage to move forward.

Australia has just announced an ambitious but politically tricky carbon tax plan.

Harford is a fan. If the playing field could be tilted to underline that greenhouse gases are expensive, it would give every decision-maker an incentive to reduce his carbon footprint via one tactic or another. Again, we should foresee and encourage a variety of tactics. While the markets generally fumble their way to success, not everything should be left to the market. Governments should also be encouraging myriad ideas; they can?t know which one will actually work. Encourage every path to be explored, ?even if one of the quests is simply to make cows belch like kangaroos?. We look at the challenges of complexity here too, with an appreciable reduction in greenhouse gas emissions being dependent on billions of decisions made by billions of humans. This is quite unlike pulling off a Coke success.

It may not be possible with global warming, but randomised trials can give us a relatively inexpensive head start on success in many experimental terrains. Harford quotes leading randomista Esther Duflo about what works for the poor: ?If we don?t know whether we are doing any good, then we are not any better than the medieval doctors and their leeches.? In this context, consider the trial projects for the direct transfer of subsidies that are to be launched in seven Indian states over the next six months. Improving feedback loops can really improve delivery of development aid, even if it can?t deliver foolproof insulation against failure.

The post-Lehman financial meltdown is probably the most notable ?failure? in the current moment, continuing to throw up new twists even if this is in the shape of say the everyday (and ever ominous) minutiae of the European debt crisis. Here, Harford revises his central contention that failure is both necessary and useful to underline the use of safety systems, to emphasise the desirability of scaling new systems so that failure is survivable?something that market-based economies do essentially better than planned economies, where large-scale failures seem to generally have more dire consequences. But the Lehman meltdown not only defied this truism, but ?regulators currently have little idea about whether there is another AIG out there, and no systematic method of finding out?. What Harford recommends is a less tightly coupled system, where financial institutions don?t act like dominoes when they fall over. Doing away with ?too big to fail? institutions is easier said than done. But improved data, information systems that perform better radar functions and that encourage whistleblowers, would help. There is, of course, no magic pill here, only a vision: ?It should have been possible to let both Lehman and AIG collapse without systemic damage?.

As Brooks notes, Harford doesn?t get into the psycho- spiritual traits of failure as he is an economic journalist, but he still offers a useful guide for Everyman, living in the world as it really is. Here is the Harford lesson in a nutshell: ?To embrace the idea of adapting in everyday life seems to be to accept blundering into a process of unremitting failure. So it?s worth remembering once again why it is worth experimenting, even though so many experiments will, indeed, end in failure. It?s because the process of correcting the mistakes can be more liberating than the mistakes themselves are crushing, even though at the time we so often feel that the reverse is true.?

Adapt: Why Success Always Starts With Failure

Tim Harford

Little, Brown

Rs 499

Pp 320