On Sunday I wrote a small post about how UK company Prozone captures all English Premier League soccer matches on video and then uses that footage to create a database of every shot, pass, tackle and assist made by every player (according to a comment on that post Opta Sports are powering the Guardian Chalkboard – sorry I got that wrong).  That’s a lot of data.  However, as I found out in the New York Times this morning, this is nothing compared to the volume of footage captured by the US Airforce spy drones.  Apparently it would take one person 24 years to watch all the surveillance footage captured in Afghanistan in 2009.  According to the article, the Army has a group of young soldiers watching every second of this footage live in order to detect suspicious activities. 

The Army expects the volume of footage to grow exponentially in the near future as new drones are being deployed that can film at up to 65 different angles.  Processing all that footage will soon become unmanageable.  The Airforce top intelligence official LT. Gen. A. Deptula said the army could soon be “swimming in sensors and drowning in data”.  So they are looking for new analytical techniques that can help reduce the volume of footage that needs to be analyzed by the human eye.  Now where do you think they started their quest for this technology?  You got it – the sports networks. Apparently Army officials have been shadowing the ESPN broadcast trucks outside football stadiums to learn how the TV network is tagging and retrieving all the highlights in their NFL footage.  It sounds like Opta Sports and Prozone have just found a second market.

One Army official called out the limitations of automated recognition technologies for the Army.  He said that when it comes to the use of force “You need somebody who’s trained and is accountable in recognizing that that is a woman, that is a child and that is someone who’s carrying a weapon.  And the best tools for that are still the eyeball and the human brain.”  But even if technology can identify the 10% of footage that needs to be analyzed by humans it would still make a big difference.

Yesterday Landon Donovan tore up the right flank of the pitch at the Emirates Stadium and helped his new club Everton secure a draw against my beloved Arsenal.  This morning one of the soccer blogs I follow showed this great visualization of Donovan’s impact on the game. 

 by Guardian Chalkboards

The graph shows Donovan’s completed passes in blue.  If you roll across the dots you can see when he passed to which team mate.

The graph is built with the Guardian’s interactive chalkboard which has apparently been around for almost a year.  I have to say that I am ashamed that I only found out about this fantastic little tool today.  OK, I have been living in the US now for 6 years and I have started to refer to the beautiful game as soccer (mainly because I discovered the other beautiful game of American football).  And coverage of soccer in the US media is spotty at best.  But that should still not be an excuse for not knowing about this amazing interactive visualization tool.

It uses data collected by UK company Prozone who record all Premier League matches and use that footage to code all data on passes, shots, tackles etc for every player on the pitch.  The guardian then build this very easy to use application that allows every aspiring Arsene Wenger (who was one of the early Prozone adopters) to analyze the patterns of the game.  Here are a couple of great examples.

What is probably most impressive about this little application is the ease with which one can share chalkboards with others.  This makes it a potentially powerful social media tool.  And with most of the coverage of soccer in the US happening on social media, it could be a great tool to educate the US audience on the tactical aspects of the game.  Nothing gets America more excited about sports than stats and analysis!

CNN were the 1st network to use the Perceptive Pixel touchscreens to visualize data for the elections last year.  I just saw they have a new article on data visualization on their site.  They call it “the new way of looking at the world”.  The article discusses the new tools that are making it easier to see and analyze complex data.  It claims that cheaper technology and the availability of digital data are the main drivers of innovation in the space.  It also mentions the power of data visualization on mobile devices.  The article has the following examples, many of which we have discussed on thedoublethink :

Ben Fry 

Everyblock 

IBM’s Many Eyes 

Yelp’s Monocle

Citysense

Flowing Data

Data visualization is a fascinating and now apparently even newsworthy field.  It can give people access to data in an intuitive and user friendly way.  Some data visualizations can also be incredibly beautiful (like the visualization of the port traffic I described in a previous post).  It can also be extremely helpful when one needs to communicate insights derived from data to a non technical audience. 

One promise however I haven’t really seen it fulfill is the discovery of insights that could not have been found with traditional multivariate statistical techniques.  Though I agree it can help statisticians communicate their insights, I dont think it really helps them discover them.  On the other hand it does provide a much broader audience than just statisticians with data analytics tools.  This is probably the biggest potential for data visualization.  Having people with vastly different backgrounds mine data will almost definitely produce more and richer insights.

I just came across this beautiful visualization of vessel movements on sea on visualcomplexity.com today.  Not only is it a spectacular visualization that uses millions of datapoints, it is also very close to my heart as it shows shipping movement around the port of Antwerp, my hometown.

The graph below shows shipping movements around the nearby port of Rotterdam.  You can clearly see the shipping lanes that show traffic on sea is not an open playground as the authors say. 

The illustration below however shows shipping movements during bad weather.  This clearly makes it a lot harder for ships to stay on course.

This last picture gives the overview of the movements around Antwerp, Rotterdam and Amsterdam.  Another beauty showing the connectivity between the Netherlands and Belgium.  I might frame this one!

I had a very interesting conversation with David Huffaker last week.  David just finished his PhD in Media, Technology and Society at Northwestern University, which makes him an expert in social media analytics, an exciting relatively new field. 

David has analyzed behavior on social networks such as Google groups to get a better understanding of how influencers behave.  He defines influencers based on 3 criteria :

1. Does the content they create trigger replies?

2. Do these replies generate conversations?

3. Do the followers adopt the influencer’s language in these conversations.

The insights were not surprising but interesting nevertheless :

- Leaders are active – they are more likely to post messages, reply to others, and have a longer tenure in the community.

- Leaders are social – rely more on replying to others than merely on broadcasting.

- Leaders are passionate – they will demonstrate higher frequencies of talkativeness, affective language (use words such as good, bad, love, hate, …), assertiveness (use words such as always and never) and linguistic diversity.

He also analyzed the characteristics of the group.  He analyzed groups based on

- The size of the group

- The density of the group – how well connected are individuals in the group?

- The stability of the group – what is the group turnover?

- The participation equality – do group members participate equally or are there only a handful of members who account for the bulk of the activity?

The picture above is one of his group visualizations which show how dense the group is, where the influencers can be found (size of the bubble indicates degree of influence).  On the outside of the graph you can see how the group reaches out to new members in the outside world. 

His analysis showed that more nodes and connectivity results in more individual influence.  At the same time equality in participation and turnover don’t matter.

I asked David whether influencers were really that important.  One of my previous posts showed an interesting analysis on Facebook data that seemed to support Duncan Watts theory on the importance of group dynamics vs influencers.  David said that the jury was still out for him.  He said that much more empirical evidence was required to really validate or refute any of the theories on how social networks work.

That’s exactly what he will be doing next.  He is working on a post doc where he will analyze Second Life to get a better understanding of how groups behave (part of his research is funded by the military and the government).  According to David, Second Life is an interesting lab for studying group interactions because of the similarities with real life.  Second Life unlike other social networks has the notion of geography and “physical” spaces which could make the findings more applicable to real life.  I am looking forward to his findings.

Hats off to my Canadian colleagues have done a beautiful little film for the most recent Hellmann’s campaign.  What a great way to make statistics come to life.

The movie can be found on the Canadian Eat Real, Eat Local site.

I am on vacation in Belgium this week to visit my family.  Yesterday I took the train from Brussels to Antwerp, a commute I used to do daily when I worked in Brussels.  I’d jump on in Antwerp and would usually have a seat.  The 1^st stop was Mechelen, where a big crowd would always get on – most of them would have to stand until most of us got of in Brussels.  

The track between Mechelen and Brussels is the oldest on continental Europe – the tracks were laid in 1835.  Charles Joseph Minard – a French engineer in charge of inspecting the bridges – studied the patterns in volume of passengers on the line.  The hand-drawn graph above shows 3 maps he drew in 1843.  The shaded bars show volume of passengers over the trajectory of the train to Brussels.  He found out what I was still experiencing 150 years later – a lot of people getting on the train in Mechelen (Malines).

Minard went on to become one of the true greats in the early days of data visualization.  His masterpiece “Carte Figurative des pertes successives en hommes de l’Armee Francaise dans la campagne de Russie 1812-1813” is considered by many (including Edward Tufte) to be the best statistical graphic ever (click here for a discussion on this subject on flowingdata.com : )

The graph depicts Napoleon’s disastrous military campaign in Russia in 1812-1813.   The brown section of the graph shows the geographic movement of the French army towards Moscow on the far right.  The black section shows the return.  The thickness of the graph represents the number of troops left (every mm represents 6000 troops).  On the bottom you can see the harsh temperatures of the Russian winter.  This graph shows a lot of very chilling data in 1 picture.  Only very few troops returned to France … .

I mapped the companies on a scatterplot using Many Eyes (please be patient as this page can take time to load).  There were a few outliers like Starbucks so I took the log of both the mentions and followers to make the map legible.

The dots on the horizontal axis represent brands that do not officially participate in the conversations on Twitter (they have no followers).  The ones on the right hand side of the graph (Apple, Microsoft and AIG for example) are mentioned frequently in conversations so they might want to consider taking a more active role.  

For the brands that do have followers there seems to be a correlation between followers and mentions.  That makes sense intuitively.  Brands in the top right quadrant are very active and the ones on the bottom left are not.

If you draw an imaginary trend line through the scattercloud you would find brands on that line to have a number of followers that is in proportion to the volume of mentions they get.  Brands above the line however get followed more than they are talked about.  It’s the other way around for brands below the line.  If you mouse over the dots on the graph you see which brands fall where.

I think we need more data to really interpret the graph.  Being followed more than being talked about could be good or bad depending on the engagement levels of the followers.  Being talked about more than being followed could be good or bad depending on the sentiment of the conversations about the brand.  Even if the interpretation of this graph isn’t 100% clear, I thought I’d post it anyway as I think it is interesting to compare brands visually.  Would love to hear your thoughts.

A couple of weeks ago I wrote a small post about Perceptive Pixel’s touch screens.  I think they are still miles ahead of other players in that space and I have also heard their screens have become more affordable.   However it looks like the next generation data user interface is already on the horizon.  The screen could be the palm of your hand.  Have a look at Patti Maes’ presentation of sixthsense below.