Taleb and Pre-test Probabilities

Arnold Kling comments about Russ Roberts' conversation with economist Nassim Taleb:

I was most interested in the latter third of the conversation, where Taleb talks about his radical empiricism. For example, he argues that medicine makes more progress with trial-and-error than with knowledge of biological processes.

Trial and error would require taking a drug already known to be ineffective, adding a methyl group, testing the new molecule on a statistically significant number of patients, and then methylating again, until we've tested infinite permutations of "little-value-added" functional groups. We'd be bored sick.

One of the reasons why many empiricists object to all the funding that goes into certain alternative medicine projects, such as magnet therapy or reiki, is that, based on our understanding of physiological or biochemical principles, the pre-test probability of such treatments being effective is pretty low. An extreme Popperian would object, insisting that we can't truly know if anything works, before testing it, (and once experiment concludes, we still couldn't be sure). Such agents of uncertainty would be technically correct.

However, science does not mean claiming omniscience nor capitulating to any smidgen of doubt. Science involves taking the information that lies before us, determining what phenomenon is most likely, and using these findings to develop a testable hypothesis. We are occasionally lucky enough to discover a drug whose mechanism of action we do not fully understand (such as in the case of Topirimate for epilepsy, or the prevalent use of beta-blockers of hypertension, long before we knew how it worked). Yet relying on lucky breaks, or "trial and error," rather than "hot on the trail" paths gleaned from discoveries in biology, is like searching for a bank robber by starting with the As in the phone book. Or like seeking out a black swan, by beginning the expedition at a local Los Angeles lake.