In this very entertaining one-hour documentary Hans Rosling explains the civic, scientific, cultural and social value of statistics and stresses out the importance of being able to communicate them in plain language. Shall this basic rule fail, visualisations and data studies would lose all their potential to make a valuable contribution to society.

Notes from this workshop and links to useful resources related to this topic can be found in this Google document.

Notes from this workshop and links to useful resources related to this topic can be found in this document.

Notes from this workshop and links to useful resources related to this topic can be found in this document.

For years, it’s been only the most committed of journo-nerds who could invest both the time and money to put geography to use.

To say that’s changing is an understatement. GIS (Geography and Geographic Information Systems) is quickly becoming an integral part of how journalism is created and delivered.

Location is one of the key components of mobile, allowing information to be filtered based on location. It’s the foundation for pioneering concepts like EveryBlock and the reason most news organizations are enamored by geocoding news items. Geography is even a feature of Longitude, one of the inaugural projects of the New York Times’ beta620.

Luckily, as the need for geographic literacy has increased, digital cartography has exploded. Interactive maps and location-based services have unleashed a torrent of spatial tools throughout the past few years, making everything from analysis to sophisticated Web applications accessible.

There are a bevy of tools available, but here’s an introduction to those that make up my open-source GIS suite.

Getting your feet wet: QGIS

Journalists mostly use GIS systems to map and layer different data sets. Typically, these data sets come as “shapefiles” — geographic databases that contain points, lines or polygons, as well as information about each feature. It’s a fairly open standard that comes from a commercial background.

Even non-native GIS databases can be mapped through geocoding, whereby addresses are plotted on a map. Just a few years ago, the process was a headache that could last for days. Now, thanks to tools such as Google Fusion Tables and Refine, it’s a lot easier.

You can compare geocoding results, satellite imagery and shapefiles to one another to reveal all sorts of things: how lottery sales relate to poverty; how well a county’s tornado sirens cover the population; or that minority neighborhoods are more at risk of foreclosure.

ArcView, one of the most well-known proprietary systems, remains the industry standard for GIS. You probably even have an installation in your building — if not in the newsroom, then perhaps in circulation or advertising.

Some newsrooms worry that without ArcView, their spatial capabilities are limited. Fear not: QGIS has quietly come of age. Over the past few years it has become easier to use and far more powerful. It’s now a more than worthy replacement for the old ArcView standby.

There’s no dearth of resources for getting your feet wet with spatial analysis in QGIS. Here are a few links to get you started:

• The QGIS wiki has walkthroughs and video tutorials for everything from opening shapefiles (vector files) to handling projections, a term I never thought I’d deal with outside of sixth-grade social studies.
• Data analyst Tim Henderson wrote an excellent tutorial to making quick interactive maps using QGIS and some javascript libraries.
• Michelle Minkoff has done a number of posts on getting spatial records — like those managed with QGIS — online with tools from Google. Here’s one looking at intensity maps.
   

[...]

Read the full article on Poynter.org.

Distribution of children less than five years old in several U.S. counties

There hasn’t been a time in the last six months when at least one of the members of the Chicago Tribune's News Applications team wasn’t working with census data.

In April, May, and June we contributed to a joint effort with an esteemed cadre of news nerds to develop census.ire.org, a site intended to make it easier for journalists to report from census data. To prepare for this recent release, we even spent a week hacking near-complete prototype maps using data that the census had already released about Kings County, New York.

We learned hard lessons about the scale and nuance of the census data in the last few months, and, along the way, we further built out our toolkit for making maps. Last week the Census Bureau released detailed (summary file) data for Illinois, and we used our new tools to produce a couple of maps we’re pretty excited about:

• 2010 Census: Same-sex partners in Chicagoland
• 2010 Census: Children less than five years old (shown above)

These maps demonstrate a map style we haven’t attempted before: dot density mapping. Dot maps let us represent multi-variate data more richly than choropleth maps. For example, they allow us to illustrate variation in race and population density simultaneously. We were inspired in this effort by Bill Rankin’s Radical Cartography project and Dennis McClendon’s map of residential patterns according to census racial categories, created for the Encyclopedia of Chicago.

Many of the tools needed to create the maps we wanted didn’t exist. Using TileMill, a fantastic application for creating maps, as our starting point, we began to build a toolkit.

Invar

Invar automates the generation of map tiles, and the deployment of tiles to S3. It is the first and least glamorous of the tools we created, but crucially, it’s very, very fast. Fast!

The first time we ever tried to create our own tileset, it took hours to render and twice as long to deploy. Thanks to invar’s parallelizing these tasks, we can now produce a map in minutes and deploy it just as fast. In fact, we now deploy our maps to four separate S3 buckets so that we can take advantage of Leaflet's support for round-robining tile requests to multiple subdomains. Fast!

Englewood

Next we needed to distribute dots across geographies. We found one implementation of dot distribution in Python, which we extended into a module for reuse.

Englewood (named after an ailing Chicago neighborhood that the Chicago Tribune writes many sad stories about) uses the Python bindings for GDAL to load data from PostGIS or shapefile. It scatters points within each feature and then writes the points out to a table or new shapefile.

A small snippet of Python is required to configure Englewood. The following code renders the dots for our map of children less than five from a database. (A demo using shapefiles can be found in the Github repository):

Deployment

A fast and stable process is useless if you can’t repeat it. We’ve built out a fabric configuration which allows us to make these maps in the quickest and most efficient way possible. Among other things, it allows us to keep some configuration (such as a bounding box) in a per-map YAML file. It parses this file and handles passing the correct arguments to invar for rendering and deployment. Perhaps most exciting, if you’re using the new TileMill 0.4 (available for OSX or Ubuntu), it can completely automate the production of Wax interactivity grids, such as the ones we used to do the highlighting in our recent maps.

Styling dots

Creating dot density maps created new challenges with regards to styling. We tried numerous approaches to color and size the dots, and ultimately settled on a few principles that worked pretty well:

• Use a dark, sparse base-layer (we used a custom-styled Google Maps layer, but would like to move to an Open Street Map base-layer in the future).
• Make your dots to stand out brightly. Try the fluorescent colors from the palette of Crayola crayons.
• Play with transparency - you may want to take advantage of the effect of overlapping transparent dots.
• Make Dots scale on zoom.
• Whenever possible, use one dot per individual. It will make for a more interesting map.

Here is the style we settled on:

Wrapping up

Although I’ve linked to a number of projects and code snippets in this post, you may find it useful to see a complete project. This week, with Illinois under our belt, I decided to apply the same methodology to my side-project, Hack Tyler. I produced a map of racial diversity in Smith County, Texas (see related blog post). Part of Hack Tyler’s modus operandi is developing in a completely transparent manner. As a result, you can see complete examples of both our backend and client-side mapping rigs in the following projects:

• hacktyler-maptiles
• hacktyler-census

I used to love diverging red-green colour scales, but it's over now.

Map showing the differences in income of private households in different regions of Europe with the red-green colour scale

The reason for the popularity of red-green colour scales is probably the fact that they are so easy to interpret (at least in my culture, i.e. German culture). Green is associated with 'good' and red is associated with 'bad'. For instance, the map above shows the income of private households in different regions of Europe. I used a diverging colour scale to show the difference in income between households with average income (bright yellow), high income (green) and lower income (red). In fact, I used the exact color scale provided by the good old flare visualization toolkit.

But this scale has some limitations. Firstly, as alluded to above, the connotations associated with the colours red and green vary a lot from culture to culture. According to a helpful collection called 'Color Meanings By Culture', green has negative connotations in some Eastern cultures such as China or Indonesia, and red is associated with love, happiness and a long life in the same cultures.

The second limitation comes from the way in which someone suffering from dichromacy (also known as colour blind) would see these colours. Below is a simulation of how someone suffering from deuteranopia (green colour blindness) would see the red-green scale:

A similar problem is faced by individuals suffering from protanopia (red colour blindness), as you can see in the map below:

To me, these limitations alone are enough reason to give up using red-green colour scales for good. Instead, I'll be using blue-orange scales. After experimenting a few minutes with blue-orange scales, I found that hue values of 220° (blue) and 30° (orange) give a good result, as you can see in the map below:

Below is how someone suffering from deuteranopia (green colour blindness) and protanopia (red colour blindness) would see the map above in the blue-orange scale:

This scale is also friendly to someone suffering from a third type of colour blindness called tritanopia or blue colour blindness, as you can see in the map below:

For anyone who wants to produce on-the-fly colour blindness simulations, I would recommend a tool called Sim Daltonism. (Many thanks to Michel Fortin for this). If you've heard of a similar tool for Windows and Linux, please let me know.
 

Update:

Color Oracle is another great tool for turning the whole screen into a colour blindness simulation. Some research is being done with that tool, namely 'Color Design for the Color Vision Impaired' by Jenny & Kelso, 2007. They're recommending using purple-green scales instead of red-green scales:

There are three letters that have been floating around the media world for several years now: API. Short for “application programming interface,” an API enables software programs to communicate with one another, allowing your programs to share data and interact in a variety of ways.

There have been lots of articles about why it’s important for news outlets to have and use APIs.

To get the most out of an API, a conscientious creator will often produce a guide, called documentation or docs. There is no single standard for API documentation. The quality varies widely, from indexish and orderly, from pretty, to plain, to messy, to incomplete and nonexistent.

Screenshot of Google Maps API Web Services

There aren’t many resources that explain API documentation to non-coders. And because the format isn’t standardized, it’s hard to write a one-size-fits-all guide to reading the manual. But assuming you’re dealing with a well-documented API, here’s an overview of how to figure it out.

The fundamental question: What can this API do for me?

Look for mentions of the word “requests.” If you don’t see that, look for the words “REST API,” or something that looks like the latter part of a URL. Within those sections, look for the words “get” and “post.” These are called methods, the specific actions the API can do. (Some developers will call them functions.)

If the documentation is written in plain English, it will be easy to understand what the method is doing. If not, you’ll need someone with more coding experience to help interpret what’s going on. But know this:

• “Get” asks for something from the API server — as in, GET me the number of times an address shows up in the database.
• “Post” changes the database by creating, adding or removing something from it — as in POST a new address to the database.

In what format can I get the data?

An API usually lets you choose how the data will come back to you, also known as the response format. You’ll usually see “json” or “XML.” Sometimes, you’ll see “txt” or other formats. 

To find format options, search for the word “format” or “response.” Sometimes the format is mentioned at the start of documentation; sometimes, you’ll find “format” in the methods.

What does the API need in exchange for what I want?

Sometimes you can make a API request or post without identifying yourself. But API creators often want to know how the API is being used and by whom, so many APIs require a key — an ID unique to the person or program making a request.
Getting a key is generally straightforward. Look for the word “authentication,” “API key” or “APIkey” to get the instructions, and to see which methods (which “gets” and “posts”) require authentication.

[...]

Read the full article on Poynter.org.

Adobe Flash can make data difficult to extract. This tutorial will teach you how to find and examine raw data files that are sent to your web browser, without worrying how the data is visually displayed.

For example, the data displayed on this Recovery.gov Flash map is drawn from this text file, which is downloaded to your browser upon accessing the web page.

Inspecting your web browser traffic is a basic technique that you should do when first examining a database-backed website.  

Flash applications often disallow the direct copying of data from them. But we can instead use the raw data files sent to the web browser.

Background

In September 2008, drug company Cephalon pleaded guilty to a misdemeanor charge and settled a civil lawsuit involving allegations of fraudulent marketing of its drugs. It is required to post its payments to doctors on its website.

Cephalon's report is not downloadable and the site disables the mouse’s right-click function, which typically brings up a pop-up menu with the option to save the webpage or inspect its source code. The report is inside a Flash application and disables copying text with Ctrl-C.

We asked the company why it chose this format. Company spokeswoman Sheryl Williams wrote in an e-mail: "We can appreciate the lack of ease in aggregating data or searching based on other parameters, but this posting was not required to do these things. We believe the [Office of the Inspector General]’s requirement was intended for the use of patients, who can easily look up their [health care provider] in our system."

Software to Get

• Firefox
• The Firebug plugin, to monitor your browser’s web traffic
• Ruby, the scripting language
• Nokogiri, an XML parsing library for Ruby

Instead of using Firebug, you can also use Safari's built-in Activity window, or Chrome's Developer Tools, for the inspection part. To parse the result, we use Ruby and Nokogiri, which is an essential library for any kind of web scraping with Ruby.

A Series of Tubes...and Files

While the site makes the data difficult to download, it’s not impossible. In fact, it’s fairly easy with some understanding of web browser interaction. The content of a web page doesn’t consist of a single file. For instance, images are downloaded separately from the webpage’s HTML.

Flash applications are also discrete files, and sometimes they act as shells for data that come in separate text files, all of which is downloaded by the browser when visiting Cephalon’s page. So, while Cephalon designed a Flash application to format and display its payments list, we can just view the list as raw text.

Viewing Cephalon's page. The Firebug panel is circled

Firebug can tell you what files your browser is receiving. In Firefox, open up Firebug by clicking on the bug icon on the status bar, then click on the Net panel. This panel shows every file that was received by your web browser when it accessed Cephalon's page.

Close-up of the Firebug panel. The Net tab is circled in yellow, the relevant .swf file is circled in green.

We know we’re looking for the Flash file, so let's look for that first. Flash applets use the suffix swf. The only one listed is spend_data.swf. In Firebug, right-click on the listing, copy the url, and paste it into a new browser window:

http://www.cephalon.com/Media/flash/spend_data-2009.swf

You can see the Flash file in its context here: http://www.cephalon.com/our-responsibility/relationships-with-healthcare-professionals/archive/2009-fees-for-services.html.

You'll get a larger-screen view of the list, though that doesn’t really help our data analysis. As you may have noticed in the Firebug Net panel, spend_data.swf is less than 45 kilobytes, which doesn't seem large enough to contain the entire list of doctors and payments. So where is the actual data stored?

Sniffing Out the Data

Here’s how find it: First, clear your cache in Firefox by going to Tools->Clear Recent History and selecting Cache. With Firebug still open, refresh the browser window that has spend_data.swf open.

Relevant XML file is circled here.

Firebug's window tells us that besides receiving spend_data.swf, our browser downloaded two xml files. One of these is more than 100 kilobytes, which is about what we would expect for an XML-formatted list of a few hundred doctors.

Now right-click on the file in Firebug and select Open in New Tab, and then View Page Source by right-clicking in the new tab. You should see a text file full of entries like the following:

That's what we were looking for: a well-structured list of the doctors and what they got paid. Now it's a simple matter of using an xml parser, like Ruby's Nokogiri, to iterate through each "row" node and pick up the essential values.

Parsing with Nokogiri

The following is a brief example of Nokogiri's most basic methods. It assumes you have Ruby and Nokogiri installed, and a little familiarity of basic programming.

The two Nokogiri methods we're most interested in are:

• css – this lets us select tags inside XML and HTML documents. In this example, we want the value and row tags.
• text – with each element returned by css, text will give us the actual characters enclosed by the element's tags.

Each row represents a record, and each value represents a datafield, like name and location. So, we simply want to read each row and select the values we're interested in.

Here's a compact variation of the above code that writes the result into a file:

So, what first appeared to be the most difficult report to parse ends up being the easiest. Whether you’re dealing with a Flash application or a HTML database-backed website, your first step should be to see what text files your browser receives when accessing the page.

This is the first part of a multi-part series designed to share with readers an inspiring collection of the most important, effective, useful and practical data visualisation resources. The series will cover visualisation tools, resources for sourcing and handling data, online learning tutorials, visualisation blogs, visualisation books and academic papers. Your feedback is most welcome to help capture any additions or revisions so that this collection can live up to its claim as the essential list of resources.

This first part presents the data visualisation tools associated with conducting analysis, creating effective graphs and implementing business intelligence operations.

Please note, I may not have personally used all tools presented but have seen sufficient evidence of their value from other sources. Also, to avoid re-inventing the wheel, descriptive text may have been reproduced from the native websites for some resources.

Microsoft Excel

Microsoft Excel is the most popular spreadsheet tool in the world with over 400 million users and therefore the most accessible tool for conducting analysis and presenting data in graphical format. The package receives a great deal of justified criticism within the visualisation field for the appalling default and range of bad practice graph designs it promotes, yet in the right hands it can be an incredibly powerful and effective visualisation tool.

Find out more information | Cost: Trial > under £100/$150 per license | Tags: Spreadsheet, Office, Graphing

Good examples and references: Peltier Tech | Excel Charts Blog | Chandoo

Status: Ongoing (July 7, 2011)

Open Office Calc

For those who cannot afford or get access to a Microsoft Excel license, OpenOffice.org is an open-source project providing an online office platform mirroring much of the functionality provided by Microsoft Office. The aim is “to create the best possible office suite that all can use”. The Excel equivalent is Calc and although some of the graphing features are limited it is an ever-evolving tool that is being used by many and is only going to improve.

Find out more information | Cost: Free | Tags: Spreadsheet, Office, Graphing

Examples and references: Gallery of chart types

Status: Ongoing (July 7, 2011)

Tableau Desktop

Tableau Desktop is based on breakthrough technology from Stanford University that lets you drag & drop to analyse data rapidly and fluidly, connect to data in a few clicks, then visualise and create interactive dashboards in an instant. Tableau have based their product on years of research to build a system that supports people’s natural ability to think visually providing a tool that lets you easily build beautiful, effective, rich data visualisations.

Find out more information | Cost: Trial > £600/$999 Personal, £1200/$1999 Professional | Tags: Statistical Analysis, Business Intelligence, Dashboard

Examples and references: The Information Lab | The Data Studio | Freakalytics

Status: Ongoing (July 7, 2011)

Tableau Public

Tableau Public is the web-based, publicly accessible version of Tableau Desktop which enables you to create interactive visualisations and embed them into your website, publish them on the Tableau Public Gallery or share within the Tableau Public community. Note, the visualisations cannot be saved locally, that is the ‘public’ essence of this free tool.

Find out more information | Cost: Free | Tags: Statistical Analysis, Business Intelligence, Community

Examples and references: The Information Lab | Joe Mako | OECD Factblog

Status: Ongoing (July 7, 2011)

TIBCO Spotfire

TIBCO Spotfire Professional aims to make it easier to build and deploy analytic applications over the web or perform ad-hoc analytics on-the-fly by letting you interactively query, visualise, aggregate, filter, and drill into datasets of virtually any size.

Find out more information | Cost: Trial > £/$ unknown | Tags: Statistical Analysis, Business Intelligence

Examples and references: Demo Gallery

Status: Ongoing (July 7, 2011)

QlikView

The QlikView platform aims to bridge the gap between traditional BI solutions and standalone office productivity applications, enabling users to forge new paths and make new discoveries. QlikView infuses a broad set of new capabilities, analysis, insight, and value to existing data stores with user interfaces that are clean, simple, and straightforward.

Find out more information | Cost: Trial > £/$ unknown | Tags: Statistical Analysis, Business Intelligence, Dashboard

Examples and references: Demo Gallery

Status: Ongoing (July 7, 2011)

Grapheur

Grapheur is a reactive Business Intelligence tool integrating data mining, modeling, multi-variate analysis and interactive visualisation into an end-to-end discovery and continuous innovation process powered by creativity and curiosity.

Find out more information | Cost: Trial > £/$ unknown | Tags: Statistical Analysis, Business Intelligence, Multi-Variate

Examples and references: Demo Gallery

Status: Ongoing (July 7, 2011)

Gephi

Gephi is an open-source, free interactive visualisation and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs. It claims to be “like Photoshop but for data”, allowing the user to interact with the data representation, manipulate structures, shapes and colors to reveal hidden properties.

Find out more information | Cost: Free | Tags: Statistical Analysis, Business Intelligence, Complex Systems

Examples and references: Demo Gallery | The VIZoSPHERE

Status: Ongoing (July 7, 2011)

Visokio Omniscope

Visokio Omniscope is a versatile, multi-tab and multi-view interactive data analysis, filtering and presentation tool. It offers a powerful new way to visualise, explore and report on large tables of data – with related images, maps, links, and more – then lets you share your file with others using the free Viewer.

Find out more information | Cost: Trial > £/$ unknown | Tags: Statistical Analysis, Business Intelligence, Multi-Format

Examples and references: Demo Gallery

Status: Ongoing (July 7, 2011)

Panopticon

Panopticon data visualisation software supports rapid analysis of fast-changing and historical time series data sets. You can deploy it on the desktop or over the web — or embed it into your own enteprise applications. Originally focusing on real-time treemap visualisations, the product suite is now much broader encompassing traditional options such as bar charts, line graphs and an innovative time-series solution termed ‘horizon graphs’ and more contemporary solutions such as Stephen Few’s bullet graph and Edward Tufte’s sparklines. This creates a great variety of innovative and effective visualisations that can be combined into a single powerful, interactive dashboard display. Most importantly they employ best practice visual principles throughout their offering which stands them apart from other competitors.

Find out more information | Cost: Trial > £/$ unknown | Tags: Statistical Analysis, Business Intelligence, Dashboard

Good examples and references: Gallery | Review from Stephen Few | White papers

Status: Ongoing (July 7, 2011)

Notable Others…

Here are some additional suggestions you may wish to consider within this category of visualisation resources:

Wolfram Mathematica | Bring in your data, combine it with Wolfram Alpha’s ever-increasing store of knowledge, apply sophisticated symbolic and numeric analysis, and create state-of-the-art visualizations—all in one system, with one integrated workflow.

Data Graph | DataGraph is a simple and powerful graphing application for Mac OS X – a companion for Excel, Numbers or any of the big statistical packages.

OmniGraphSketcher | OmniGraphSketcher helps you make elegant and precise graphs in seconds, whether you have specific data to visualise or you just have a concept to explain.

PLOT | PLOT is a scientific 2D plotting program for Mac OS X designed for everyday plotting – it is easy to use, to create high quality plots, it allows easy and powerful manipulations and calculations of data and it is free.

MATLAB | All the graphics features that are required to visualise engineering and scientific data are available in MATLAB®, including 2-D and 3-D plotting functions, 3-D volume visualization functions, tools for interactively creating plots, and the ability to export results to all popular graphics formats.

SPSS Visualisation Designer | Easily develop and build new visualisations that enable new ways to portray and communicate analytics to others. No extensive programming skills are required to conceive, create and share compelling visualizations.

STATA | Stata is a complete, integrated statistical package that provides everything you need for data analysis, data management, and graphics – you get everything you need in one package.

Visualize Free | Visualize Free is a free visual analysis tool, providing the perfect solution for visually exploring and presenting data that standard office charting software cannot handle.

Dundas | Dundas Dashboard brings together all of the tools you need to build meaningful, interactive and fully customized dashboards in one easy to use platform.

Wondergraphs | Wondergraphs strives to be the best way to get and share insights from your data, offering free > premium set of online, graphical report design tools for analysts and businesses.