Geography 970

May 6, 2010

Animating Twitter Data

Filed under: Uncategorized — Tags: , , , , , , — Tim Wallace @ 3:05 pm

Background on 970

This blog documents an exploration shared by the seminar attendees of Geography 970: The Geoweb, during the spring of 2010, at the Geography Department of the University of Wisconsin–Madison.

Instructor: Dr. Mark Harrower
Students (in the alphabetical order): Fei Du, Jing Gao, Daniel Huffman, Kevin McGrath, Matt Moehr, Tim Wallace, Jeremy White

The seminar coalesced around a team project. While designing and developing the products, we discussed and debated over various ideas, methods, data and tools, and challenges of contemporary web-based cartography, through both weekly meetings and blogging.

Project Goal

Twitter is a rich source of instant information on people’s locations; about one tenth of all tweets are geocoded, meaning they are marked with the location that the person was at when they tweeted. This geocoding gives us a door to examine if there are spatial patterns in the use of Twitter. Where are most people tweeting from? What about their friends?

At the beginning of the semester, we reviewed existing online visualizations of Twitter, and found that although many exist, few touched on the underlying geographic phenomena. So, we set a goal to explore and discover effective ways to summarize this massive data set, and to make the unapparent emerge.

Twitter Hitter

Twitter provides free access to subset of tweets (called “the garden hose”) or about 10% of tweets on a live, streaming basis. To make accessing and processing this stream easier, Jeremy White wrote Twitter Hitter, an application which listened in on the Twitter stream and wrote out the results to an Excel spreadsheet. Twitter Hitter also allowed us to select which parts of the data stream we wanted to record and to ignore any tweets that didn’t match our criteria. For example, one of the projects listed below followed the geography of re-tweets and it was necessary to find any tweets where *both* the location of the original poster and the location of the re-tweeters were known. Without Twitter Hitter, this kind of sifting and sorting of the 1 million + daily tweets would have been (nearly) impossible.

GeoData in the Stream

“Location” has 2 meanings in the world of Twitter. It can mean (1) where someone was when they tweeted provided they are using a GPS-enabled smartphone, or (2) where someone lives (users can specify their home location). The second is less helpful because, of course, people can tweet while away from home, often while on the other side of the planet. For example, in one of the animations below you can see researcher and scientists located at the South Pole tweeting to friends and colleagues back home in South Korea.

Geographic data in the form of lat/long pairs is encoded in the Twitter data stream from third party applications such as ÜberTwitter or by mobile devices such as iPhones. These geographic coordinates provided the platform for exploring the geography of Twitter.  The stream also has optional user-added locations or addresses. Since approximately 90% of the stream was without coordinates, significant time was devoted to an attempt to transform the “user location” field (such as “New York City” or “Galveston, Texas”) into lat/lon pairs.  Ultimately, however, the processing time associated with georeferencing tens of thousands of points proved prohibitive.  Additionally, there were problems with getting accurate coordinates from a highly variable text field — if a user gives their location as “Madison,” for example, do they mean Madison, Wisconsin? Madison, Alabama?  For more on why our project only used 10% of available Tweets, see this post.

ANIMATION #1: Mapnodes Twitter Animation

As a part of his PhD research, Jeremy White, is authoring a new tool for the Cartography/GIS community.  Mapnodes is a platform for connecting independent map-design tasks, such as line generalization, hill shading or – in this case -animation.  For more info, check out Mapnodes.

ANIMATION #2: Processing/KML Project

Global maps show a lot of interesting trends, but some of the replies and Twitter activity is only grasped at the city-scale. We looked into multiple options for providing user interactivity to browse the data at multiple scales. We found Google Earth/KML, Flash, and Blender to be choppy and just not pretty enough.

To show some of the interesting local stories, we used the same data and some 3D visualization techniques in the Processing language to create a tour. The final movie file is available for viewing, but we can also provide the .kml and .jar files that went into creating the final movie.

ANIMATION #3: Lava Map

From where do most tweets originate? The obvious answer would be large, wealthy cities such as Tokyo or New York. In order to get beyond the obvious, we decided to look at how many tweets were originating out of an area, divided by that area’s population. Many people in New York tweet, to be sure, but is the number of tweets per person as high as it might be in some smaller cities?  For more on this process, see this post.

January 31, 2010

Geoweb Resources for Treasure Hunters

Filed under: Uncategorized — Tags: , , — Tim Wallace @ 12:51 pm

The National Oceanic and Atmospheric Association has long been the keeper of a huge chunk of US offshore data. Their fleet of ships carrying all kinds of scientists have been mapping bathymetry, hazards and wrecks for decades. Until recently, however, getting to that data wasn’t exactly as easy as pie.

Nowadays NOAA charts and locational data are freely available for download and implementation in any kind of geographic visualization. The charts can be downloaded as GIS-friendly raster files through NOAA’s Raster Navigational Charts site. Similarly, locations of known wrecks and obstructions can be downloaded in tabular format through the Office of Coast Survey’s Automated Wreck and Obstruction Information System (AWOIS) and easily converted to points on a map.

While some agencies  – in an effort to preserve our maritime heritage – are tentative (at best) about publishing locational data for wrecks, NOAA has made it – well, as easy as pie to find a shipwreck.

Two Links

NOAA charts on Google Maps

AWOIS wrecks and obstructions KML (this is a download)

January 27, 2010

The Hipness of Location

Filed under: Uncategorized — Tags: , , — Tim Wallace @ 4:58 pm

With a new year comes all sorts of lists.  Lists about food (Saveur 100), and music (Pitchfork Best Albums) and technology (Webby’s Top 10 Internet Moments)  spring to mind, but the tabulation does not stop there (Time Magazine did a Top 10 of Everything in 2009 – really, everything?).

Some of the lists look forward and some look back.  The New York Times Bits (Business, innovation, technology & society) Blog started 2010 by looking forward with a list of “Five Tech Themes for 2010“.   Much of what is mentioned – both in the post and in the reader’s comments section – is geoweb-related, but I’m going to stick to one of the five main themes mentioned and have a closer look at some of the companies mentioned.

Number two on the list of themes for 2010:

Location, location, location: Start-ups like Hot Potato, Foursquare, Grindr and UrbanSpoon have generated a lot of buzz for their forays into the mobile location-based arena, but it’s only the beginning — particularly in light of the new geo-location features made available to developers and users on Twitter. It’s likely we will begin seeing many more useful location-based applications.

So, what’s a location-based service?  Well, loosely, it is an information service – usually available through a mobile device, like an iPhone, Android or Blackberry – that utilizes your location to serve up relevant content.  An example of this could be a a device that automatically tells you the local weather.

Why are these services so hip right now?  Well, who wants to print out stuff from their desktop PC for their weekender?  Driving directions from MapQuest, restaurant reviews, hotel prices, movie show times . . . that’s an awful lot of paper.  And every bit of ink on that paper could be outdated the instant you hit print.  So, it’s not all about having the newest gadget.  There is real benefit to having these services at your fingertips no matter where you are.  (As a side note, I read yesterday that Nokia is offering free navigation on their new smart phones as a reaction to the Android.  Nokia says their system is better than Google’s because it is “faster”.  We’ll see.)

Hot Potato.
From the looks of it, this is a service that compiles all blogs and tweets about a particular event in a particular place and creates a sort of instantaneous web site from what people are saying. So, if some weekend, the beets at the Dane County Farmer’s Market were particularly delicious and loads of people were blogging, tweeting and posting up photos of them, Hot Potato could be used as a portal to all of the hype.

Part reality and part game, Foursquare offers users the ability to check out what other people have said about where they are (restaurant, pub, theatre, etc.) and perhaps some kind of tip (go in the side door, avoid the ornery cashier, don’t drive here, etc.) as well as – in a way that isn’t entirely clear to me – collect “points”  toward “badges” for checking in from time to time.  Apparently, if you visit a place enough, you eventually become “mayor” which qualifies you for discounts (rules and restrictions may apply, right?).

Their website reads: “Whether he’s Mr. Right or Mr. Tonight, your man is hanging out on Grindr, a killer location-based social networking tool for the iPhone or iPod Touch.”  Grindr is a location-based dating service for gay, bi and curious men.  It shows the location of other men on Grindr in your neighborhood (again, based on your location).  Other location-based dating services include Meetmoi and Skout.

This service offers up some location-based restaurant reviews. If you click the link above, it will automatically read your IP address and suggest that you go to the Daisy Cupcakery for lunch. It will do the same if you are on a geo-enabled mobile device. Since I would also recommend that you go to the Daisy Cupcakery for lunch if you were in Madison, this service really drove home the whole idea of location-based service to me. If someone were to ask me where they should get some lunch. I would have to ask them where they are and how far they would be willing to travel. Urbanspoon and other location-based services eliminate those extra steps and jumps right to what you need (where you are).

After just having a cursory look at these services, I have to agree with the hype surrounding their utility.  However, I think it will be interesting to see which of these services, in this flurry of startups, will last.  I know that people love their “aps”, but having to install a different one for every type of location-based service may prove tiring.  Perhaps if the Bits Blog is right, 2010 be the year all of this gets sorted out.

Create a free website or blog at