1

The question: How and why are we wrong about the world?

We humans are limited creatures. We have an inherently constrained array of information at our disposal to make sense of the world. As Rene Descartes and Immanuel Kant pointed out several centuries ago, we often suffer from hallucinations indistinguishable from reality (i.e. dreams) and our sense organs limit and constrain as much as they reveal about the world (light, for example, is only visible to us at wavelengths between 300-700 nanometers; though one’s nose is directly in front of one’s eyes it is purposefully discarded by our visual processing systems). Peter Bernstein and Nassim Taleb pragmatically update, sharpen, and apply these lessons in their books, respectively, Against the Gods and Fooled by Randomness.

Taleb points out several repeating biases humans fall into relevant specifically to markets, but ultimately to life in general. These include: those contained within prospect theory (underweighting tail events, anchoring, being risk averse or risk seeking depending on framing) survivorship bias, the endowment effect, confirmation bias, and, the primary thrust of his book, attribution bias. This last one is uniquely poignant when dealing with modern humans, who have a pronounced tendency to credit their failures to randomness and their successes to skill. Risk management has a storied history in this regard. Bernstein, writing in 1996, offered the incisive example of risk-managers at Proctor and Gamble, overconfident of their own skill while discounting occurrences of low probability events, using derivatives (allegedly risk managing tools) to increase single-sided exposure in-lieu of mitigating it. This was, of course, to be paralleled on an epic scale in 2008 as AIG’s risk management teams absent mindedly allowed the issuance of derivatives (Credit Default Swaps) that massively increase downside exposure (i.e. risk) to the improbable event of large scale debt default.

The major problem identified in both the old philosophical and new risk management thought is fundamentally that the data (sensory or otherwise) which we operate under is often misleading, biased, or simply not indicative of reality. Even Bayesian inference and inductive thinking are inescapably troubled: the problem of induction (or Hume’s problem) points out that past data are not predictive of future occurrences. Why do humans suffer from biases and misleading sensory information? Fundamentally, the reason is simple: humans are apes specialized for surviving in our ancestral environment (the savannah), not for precisely understanding and interpreting the world around us. It is incumbent upon us to recognize the history of our species and our penchant for falling victim to biases of cognition and perception. We are, to use Dan Ariely’s phrase, ‘predictably irrational.’ Once we acknowledge this, we can proceed to become more humble and circumspect in our abilities, and thus be able to develop a healthier relationship to risk and uncertainty.

In the book ‘Expert Political Judgement,’ Tetlock’s primary finding is that the only salient indicator of an expert’s political judgement is whether or not he / she is a ‘fox’ or a ‘hedgehog’. Using Berlin’s terminology, Tetlock operationalizes the concept of humility in knowledge, and finds that those who fervently, and overconfidently, believe in single big-thing / ideology (hedgehogs) are worse predictors than those who employ multiple frameworks in their analyses. This poignant fact speaks both to the inherent complexity of the world and to human fallibility: those amongst us who accept this complexity and account for it by being circumspect and humble in their knowledge perform better.

2

The question: Why do forecasting and risk management fail?

Political risk is exceedingly difficult to do well. In Condoleezza Rice’s 2018 book ‘Political Risk,’ she points out that while political risk is top of mind for almost all companies and executives, most do not feel adequately prepared to deal with it. Owing to this difficulty, as Hubbard outlines in his book on the ‘Failure of Risk Management,’ the prevailing wisdom amongst a non-trivial segment of the population is that risk management may be “worse than useless.” It’s not hard to see why people would think this about political risk management. First, political risk models have failed to predict major destabilizing events such as the 2008 Financial Crisis and 2011 Arab Spring. And second, conducting political risk management is a cost center for businesses that often constrains and limits action. Political risk management can be costly, burdensome, but, worst of all, ineffective. Forecasting and risk management seem to fail for three big reasons: models are focused on the wrong things or are improperly defined, the modeling methodology is inappropriate or otherwise flawed, and models are always inherently limited but people become overconfident and over reliant upon them.

The field of political risk has a plethora of models employed across industries and practice areas. Defining political risk, and the sub-terms that constitute it, is incredibly difficult. For example, Political Risk insurers such as the Overseas Private Investment Corporation and Multilateral Investment Guarantee Agency may include expropriation, inconvertibility, war damage, civil strife damage, and breach of contract within their jurisdiction of political risk, but this is not an extensive list nor are these terms themselves easily defined. The models of political risk that have emerged include different variables within political risk, define them differently, weight them differently, and thus highlight the first major problem inherent to forecasting and managing political risk.

Meanwhile, as Howell et al explain in their article on the impact of methodology, minor changes in forecasting methodology can produce major changes in predictions. By deconstructing two popular models, the Economist Intelligence Unit’s (EIU) and the Business Environment Risk Intelligence’s (BERI), and regressing them against a popular loss index, the researchers demonstrate the highly limited ability of these models to predict real loss. Hubbard’s aforementioned book explains well the findings of Howell et al. In particular, the linear scoring models employed by most risk indices lead to predictable biases that emerge as a result of range compression, presumption of regular intervals, and the presumption of independence. Methodological choices can have major impacts on model predictions.

The third major reason forecasting and risk management fail is captured in Colander et al’s article describing the failure of the economics profession. In addition to the problems defining parameters of models (e.g. ‘welfare’ and its maximization) and the methodological problems (rooted in the use of the dynamic-control model that couples the Robinson Crusoe approach with “rational” expectations), the economics profession has massively failed to communicate the limitations of its models. This crucial issue has led the economics professions in particular, but many others as well, to propagate models that people do not understand. In turn, those models were further incorporated into and undergird larger prediction mechanisms, like a house of cards. While blame ought to be shared, the critical problem arises when limited models are misused as foundations for larger management edifices.

The underlying nature of a model, which is often opaque to the public, particularly if built by a private entity, is important to understand. Building effective models is notoriously difficult, primarily for the reasons identified above. Yet doing so is imperative if we are to plan and prepare. Ultimately, models must become more transparent, and practitioners must practice what Phillip Tetlock calls ‘skeptical meliorism’ by taking a circumspect approach to the inferential validity of models and their capacity for predicting future events. In this way, we can judiciously employ models while continuously striving to test and improve them.

3

The question: Examples of Mismeasurement and How to Measure Anything?

Accurately measuring political risk seems like a herculean task. Political situations are so idiosyncratic and subjective that, to wit, they seem impossible to methodically analyze and compare. How, for example, could one have foreseen the 1998 Russian debt default (Long Term Capital Management certainly did not), the collapse of the Soviet Union in 1991, or any other major geopolitical event? Indeed, the seeming impossibility of measuring complex occurrences and / or variables, particularly ‘soft’ things outside the realm of the physical sciences, presents a daunting challenge. Douglas Hubbard, author of ‘How to Measure Anything,’ offers recourse to those who must operate and make decisions in a world filled with measurement needs.

In order to measure something, Hubbard suggests answering these basic questions: (1) what is the decision the measurement is supporting, (2) what is the definition of what we are measuring in terms of observable consequences, and (3) how does what we are measuring matter to the decision making process? Toward this end, two groups of scholars have recently published two papers that make inroads on measuring versions of political uncertainty and risk, namely economic policy uncertainty and geopolitical risk. Framing these papers in terms of Hubbard’s questions allows us to see how they came about their findings.

While both papers utilize a similar methodology, they approach two distinct problems. The first paper, published in 2016 and titled ‘Measuring Economic Policy Uncertainty,’ aims to create a transparent index of economic policy uncertainty overtime. In line with Hubbard’s diagnostics, the authors first identify the need for measuring economic policy uncertainty (EPU), given the widely held belief that there is a negative causal link between EPU and economic performance (as measured across multiple metrics). Next, the authors operationalize EPU by combing through newspaper archives to create an index based upon the monthly weighted frequency of articles containing a combination of three terms: ‘economic’, ‘uncertainty’, and one of several other terms related to federal policy. Finally, this index allows the authors, and other decision makers, to see how EPU changes over time, how this correlates with major periods of economic uncertainty, and what deleterious economic effects EPU seems to cause.

In a similar vein, the second paper, published in 2018 and titled ‘Measuring Geopolitical Risk,’ employs practically the same methodology for creating its index of Geopolitical Risk (GPR), with a similar goal of demonstrating how geopolitical risk changes over time and how it adversely impacts economic performance (again, on multiple metrics). The fundamental difference between the two papers is in the operational definitions. The authors of the second paper specifically “define geopolitical risk as the risk associated with wars, terrorist acts, and tensions between states that affect the normal and peaceful course of international relations.” The authors create a frequency weighted index of news articles just as the EPU authors did, but change the terms they search for to revolve around the above definition of geopolitical risk. The GPR and EPU indexes, then, serve decision makers similarly, just in different domains.

As with Hubbard’s book, these two papers offer insight on how to measure things that might seem impossible to measure. While private institutions such as Eurasia Group have created political risk indexes before, the transparency of the methodology these papers employ, and the demonstrated utility of using widely available newspaper databases, provides fresh insight into how we can develop innovative measurement tools. Thus, although measuring political risk is difficult (and, to some extent, perhaps arbitrary), striving to develop new tools to do so is an important and worthwhile endeavor, particularly if the alternative is fatalistic acceptance of our perpetual ignorance.