“Famous pundits virtually never make falsifiable forecasts.” To abide by that simple axiom is to ensure oneself a terrific mode of self-preservation, whether you’re Nate Silver or Jim Cramer or any of the other famous personalities making a living off of the science of prediction.

The science of prediction is of particular interest to University of Pennsylvania professor Philip Tetlock, a political scientist who has written a new book on the subject: Superforecasting: The Art and Science of Prediction

.

In this video, Tetlock runs through the thought processes of professional media prediction-makers whose livelihoods depend not on being right, but on being right enough.

Philip Tetlock: We know virtually nothing about the forecasting track records of famous pundits because famous pundits virtually never make falsifiable forecasts. They say something might happen or could happen or may very well happen, but when I say something could happen, that doesn't mean a lot. I mean it could be that it could happen that we're all going to be vaporized by an asteroid in the next 24 hours, or it could be that the sun will rise tomorrow. "It could" subsumes an enormous range of possible probabilities. So if I say something could happen and it does happen, I can say to my readers, "Well, you know, I told you it could." And if I say something could happen and it doesn't happen, I can come back to my readers and say, "I just said it could." One of the interesting things about superforecasters is how opportunistic they are. Superforecasters think quite strategically about when it makes sense to invest effort in thinking. So if you wanted to predict the outcome of the presidential election in early October 2015, 2016 presidential election, how would you go about it if you were a superforecaster? A superforecaster wouldn't look carefully at the presidential debates, look into the eyes of the candidates and see which one looks more presidential. At some point in the process a superforecaster might do that, but a superforecaster would tend to start with more the outside view and gradually work in rather than start at the inside and work out. So they would ask very general questions initially like: Let's look at elections, all presidential elections since World War II; how likely is a Democrat or a Republican to win? Or they might say after one party has held the presidency for two terms, how likely is there to be a transition? Or they might say if economic growth is less than 2 percent three quarters before the presidential election, does that bode ill for the party that controls the presidency?

So they would start off with these more general estimates, these more outside view estimates and then they would gradually adjust in response to estimates about popularity polls of candidates, which are notoriously volatile at this stage of the process. But they would take them into account, but they would discount them considerably because they are so volatile and then they would adjust incrementally. So the best forecasters, I think, over the course of 2015, have somewhat lowered their probability estimate of Hillary Clinton being the next president of the United States. I think they started off significantly above 50 percent and there's been some significant hemorrhaging of those estimates, but nothing all that dramatic. I mean the interesting thing about superforecasters is they're very patient and they make granular belief updates. So they don't suddenly say, "Oh my God, the latest email scandal," or, "Is Biden going to come into the race?" or this or that, there are these little clues and it's not that they ignore them but they tend to respond very incrementally to them. So it may be the probability of Hillary Clinton becoming the next president of the United States moves from the 45 percent to 43 percent in response to an inspector general report in the State Department, that sort of thing. The best forecasters tend not to make it on to television. They’re not very attractive to the TV producers because they’re much more likely to say, "On the one hand; on the other hand." And then now we have to strike some sort of integrated balance and that apparently doesn't make very good television.