No, traditional charts are useless in our complex world
Over the next 25 years, we will need new visualization tools to replace traditional charts.
As you know, line, bar and even pie charts first appeared 200 years ago, with William Playfair, and perhaps until 25 years ago, they were good enough helping us to make sense of our data. Before computers, they were crafted by graphic designers. Kids in schools drew them using millimetric paper.
Lotus 123 and Harvard Graphics were the most popular charting tools in the early days of personal computers. With those tools (and later, with Excel), the charting landscape changed forever. Some charts vanished, either because they weren't simple enough and/or didn't make it into the chart gallery (I miss trilinear plots - yes, Jon, I know how to create them in Excel, but still...), while others should never have been allowed into that gallery.
Today, only a small and shrinking fraction of all the data collected and stored is actually used in our decision making processes and only a fraction of that is actually displayed in charts. But we have a problem: these charts can hardly cope with our complex multidimensional world. Adding injury to insult, we are diving into visualization p0rn, defined by Robert as:
"pretty, flashy mash-ups of something or other, depicting somebody's life, citing information graphics in a commercial, or growing flowers from twitter feeds."
I find the expression highly descriptive, but I would associate it also with the corruption of traditional charts. This is a bizarre path: the more data we collect, the fewer data points are displayed in our charts. After all, we need space for textures, 3D and other chartjunk, right?
Now that our best-known charts are dying of of old age and inadequacy to the needs of our time, it is clear to me that we must really escape Flatland and conquer the third dimension. In a complex world we don't have the luxury of suppressing one of the only three dimensions that are available to us. That's the future of charts. And even if we resist the temptation of prettifying our charts and add more data, they are doomed to fail. They belong to a simpler world.
Yes, traditional charts can now realize their full potential
Well, we do collect more data, we do live in a world of a much higher variability. But that's exactly the reason why we need better charts. We can use data visualization techniques to instantly recognize patterns in very large data sets. We have a reasonably good understanding of how our perception works. The design of traditional charts can take advantage of color or grouping mechanisms. We know how to manage short-term memory.
We don't need new tools, we just need to know how to use the existing ones. Traditional charts can realize their full potential if we correct bad practices like:
- One chart to rule them all: Our reality is infinitely more complex than the one we inherited from our fathers. A chart may be fully optimized for pattern discovery but we can't expect to make sense of our complex data with a single chart. We need multiple charts to reveal multiple dimensions. Quoting Tufte, "show comparisons adjacent in spaces, not stacked in time";
- Large charts: we use charts to find patterns, and the chart size must be optimized to reveal those patterns. I don't know what the average chart size is, but I guess we could safely reduce it to one third (and use the available space to show more dimensions of our datasets);
- Lack of prioritization: not all data is born equal. We must define what ir relevant and what is merely interesting. Focus+context techniques can help us. Ben Shneiderman's Visual Information-Seeking Mantra ("Overview first, zoom and filter, then details-on-demand") should be on every desk;
- Lack of data-reduction techniques: it is often more insightful to plot a ratio (imports/exports, actual/budget) than the the original variables. This is clearly a loss aversion syndrome;
- Chartjunk: this one goes without saying, right?
- Static charts: Interaction is the cornerstone of today's information visualization for data analysis. We can make a point using a single, static chart, but to fully understand a complex dataset the user must be able to manipulate the chart.
Well, maybe...
I stumbled upon a page about "statistical techniques for dimension reduction". The author writes:
"This can be a problem, especially when some of the features are not discriminatory. In addition to the computational cost, irrelevant features may also cause a reduction in the accuracy of some algorithms. For example (Witten 1999), experiments with a decision tree classifier have shown that adding a random binary feature to standard datasets can deteriorate the classification performance by 5 - 10%."
He's right, of course, but look at the page header:
I find this very amusing. Can we seriously discuss data reduction techniques and, at the same time, be unable to recognize "irrelevant features" in other areas of our work? (Perhaps he's been ironic.)
The nature of information visualization did change in the last 25 years, for better and for worse, and is likely to change as much over the next 25 years. New formats will emerge and become popular, specially in the area of complex networks, but basic perception principles will remain the same, and if a chart is optimized to take advantage of those principles it will survive.
Current divorce between data visualization and statistics is harmful for both sides. Those who know what "mode" or "percentile" mean don't know how to make a good chart. Those who know how to create a chart don't know how to read a boxplot. Data visualization is not the lighter side of data analysis and data is not just "something" we use to create a chart.
Will traditional charts survive? It depends on how we treat the data.
Well, this was almost a SWOT analysis for traditional charts... So, back to you, what are the strengths, weaknesses, opportunities and threats that our charts face today?
