Archive for October, 2015

Stories and Data

October 2, 2015

Which of the following was the bloodiest century in all of human history ?

A. 8th

B. 13th

C. 17th

D. 20th

The answer is obviously the 20th but where does that answer come from? In short it comes from the stories that we tell ourselves about the 20 th century. Without much effort virtually anyone can come up with a list of many of the atrocities of that blood soaked century. Some of the most common examples of atrocities from 20 century: the brutal invasions of Hitler, Mussolini, Stalin, Po Pot, Mau Zedong, and Imperial Japan; Stalin’s Purge; the Gulag; the Holocaust and two atomic explosions just to name a few. All this and I have not even mentioned World War 1, to say nothing of the more recent wars of Korea and Vietnam.

The unprecedented level of killing in the 20th century was enabled by many technological advances such as highly industrialized and mechanized warfare for example; industrial production of mass qualities of guns (like AK-47), ammunition, tanks, bombs, land mines, planes, long range artillery, and warships); wide spread use of machine guns, and incendiary bombs;   weapons of mass destruction (nuclear, biological, and chemical). The data appears to tell the same story as well. For example, atrocitologist Matthew White in his book The 100 Deadliest Episodes in Human History meticulously enumerates in great detail the 100 worst things that humans have ever done to each other. When these atrocities are arranged by Death Toll in descending order it is not hard to predict which century comes out on top.

Top 30 Atrocities (20th Century in red)


Furthermore when one considers the apparent rise of violence in TV programs, movies, and video games; The War on Terrorism (longest war in US history at over 10 years); recent mass shootings (Columbine High School (1999), Virginia Tech (2007), Aurora movie theater (2012), Sandy Hook Elementary (2012)); never ending violence on the nightly news and many other recent events. It would seem only logical to conclude that violence will continue to rise until our modern society either falls into a Mad Max style post-apocalyptic world  (a repeat of the “Dark Ages”) or maybe a some cosmic (divine?) being  will take pity on us and save the world from its self (or save just the “chosen” ones?).

Both the data and the stories seem to be pointing in the same direction.

Or do they? Are the data telling stories or are our stories causing us to  misunderstand the data? Take a look again at the list of Top 30 Atrocities. While number 1 (World War 2) is from the 20th century, out of the top 10 atrocities, only 4 are from the 20th century. That should give you pause.  Things might not be as simple as they seem. Additionally, does it make sense to compare the top 2 atrocities (World War 2 and Genghis Khan) in absolute terms when they were so far apart in time (> 500 years apart)? Wasn’t the world population a lot lower in the 13th century than in 20th? The 40 million deaths in the 13th century could represent a larger portion of the world population than that of the 65 million during World War 2 in the 20th century.

Calculating the percentage of deaths by the world population relative to that time period tells a very different story.

Top 30 Atrocities Scaled by World Population (20 Century in red)

Top 30 Atrocities by World Population

I used the UN and HYDE World Population with Spline Interpolation to scale each atrocity by the World Population at its midpoint (middle year). See Our World in Data for more information.

Now, with atrocities scaled by their contemporary world population, only one 20th century atrocity (World War 2) makes it into the top 10 and 5 of the top 10 are from China or South Asia. A few questions to consider. Did Genghis Khan really kill about 10% of the world’s population? Were the Chinese really that much more murderous than 20th century Europeans and Asians (Hitler, Stalin, Mao Zedong)?

One of the issues here is data collection. Such simple things as literacy, the ability to record any event whether it be good or bad, can have a dramatic impact on our ability to enumerate death tolls. Relatively high levels of literacy is a modern phenomenon and for that reason many of the major and minor atrocities from the past simply may have gone under recorded or, most likely, completely un-reordered.  Fortunately, or un-fortunately depending on how you look at it, China is generally known to have had unusually high literacy rates throughout much its history, unlike Europe for example.

Another problem is that of exaggeration. Typically, it is the winner who gets to write history and they tend to exaggerate the number of enemies that were slain. In many cases a story about how the severely outnumbered heroes miraculously defeated the numerous enemy horde will be more appealing than the reality, which might not get recorded at all.

The above examples are just a couple of the many of the issues that historians and analyst alike have to try and deal with and compensate for in their daily work. Professional historians employ rigorous methodologies and their results are highly scrutinized and, in many cases, highly criticized as well.

Over the years I have become weary of simple and obvious answers to complex and far reaching questions.

Thinking critically is hard to do and there are many cognitive traps (Think Fast, Think Slow) that anyone, no matter how intelligent, can fall into. As a Business Intelligence (now renamed Advanced Analytics) professional I have to continually scrutinize various processes such as data collection and  data processing, in addition to reporting logic within the solutions that I build. This also includes thinking critically about and trying to compensate for both organizational and my own personal biases as well.

FYI if you really want to know whether or not the world is getting more violent I suggest reading Better Angles of Our Nature.  There are no simple are answers but the historical trends may surprise you. However, like any good financial advisor will tell you “Past Performance is Not a Guarantee of Future Results”