Sunday, July 17, 2011

The Problem of Induction

I have been reading Fooled by Randomness lately. What got me interested was of course the fact that my newest career choice requires me to pay a lot of attention to the risk of rare events and the fact that much of the book is drawn upon the author's experience as a trader - a valuable guide for my new line of work.

Currently i am reading the section on the problem of induction. In IITB during the course on introduction to philosophy, we were introduced to the Karl Popperian philosophy of falsification and how no theory is true. We used to have a lot of long discussions on this problem in our hostel rooms then (curiously on only thursday nights!) While the business of finishing an MBA (or PGDM as they call it) subsequently and earning livelihood post that put the whole topic of epistemology far at the back in my mind, this book brought it all back with a bang!

If you observe two events occuring back to back several times, can you make a statement of the type
1. This thing Y always happens after that thing X
Even more strongly, can you ever make a statement
2. This thing Y is caused by that thing X
The problem of induction is that we can say neither 1 nor 2. Science is carried out with better and better approximation to the reality "out there" in nature while never claiming either 1 or 2. It does provide us with technology which does help in making our lives more comfortable and less painful. However heart of heart, every scientist knows that s/he will never find THE truth. Every theory is an approximation - an elegant one, a massively accurate one (to the nth decimal) - but is precisely that, an approximation.

As Taleb observes, the practitioners in humanities as well as trading floors do not seem to share that humbleness. Most people in these fields make ludicrous claims of x following y ALWAYS and worse still X CAUSING Y. Curiously, mathematics seems to have made matters a bit worse! Elegant modelling often creates a sense of comfort amongst its creators as well as audience - which is akin to believing that a fortress is safe because its small clay model is butressed with concrete!
Unlike Science the problem with humanities and by extension most of our day to day lives is that of lack of repeatability. We can never really test a hypothesis, we can never rerun an event and no two situations are alike. Hence we are forever doomed to be fooled by randomness. As the victors write history, the lucky few businessmen that survive will have vision pasted onto them post facto and the lucky individuals will talk of their grit, perseverance and fortitude!

I dont know if there is a reliable sum over histories approach possible in the humanities, day to day life, running of businesses and countries. Maybe some of the intellectual energies of our generation need to be redirected towards that rather elusive but potentially paradigm shifting goal!