Giving eyes and ears to GIS

Anyone who's seen the sci-fi film Minority Report will remember the scene where Tom Cruise's character, a police detective who prevents murders before they can take place, stands in front of a futuristic computer and sifts through a stream of pictures and videos just by making gestures with his hands in mid-air (although he does have to wear special gloves to do it).

More recently, Iron Man 2 shows how every GIS interface should work! If you have a minute to spare, it's definitely worth watching the beginning of this video:

How close are we to this reality? We're certainly not there yet, although some very innovative stuff is being done in the area of "natural user interfaces" (or NUI, pronounced "NOO-ee"). NUIs are user interfaces we control using natural human actions such as talking and hand gestures, as opposed to more artificial actions such as typing on a keyboard or clicking a button on a mouse. Perhaps the most influential NUI product released recently for the consumer market is the Kinect sensor, originally designed by Microsoft for the Xbox 360 games console, but which was quickly repurposed by developers for other platforms such as the PC.

Kinect provides a depth-sensing camera and microphone array that can be used with the right software to respond to body movements and listen for voice commands. Affordable speech recognition technologies have been around for a long time, but this is the first time that such a high-quality body-tracking camera has been widely available for around £100. This opens up some exciting possibilities for GIS user interfaces.

At Esri UK, we've been building some simple prototypes with Kinect, using hand gestures to pan and zoom around a custom WPF mapping application, and voice commands to geocode locations (postcodes only at the moment) and provide additional map navigation (zooming, panning). It's a cool gimmick, but we are hoping to find a practical use for it soon! A Kinect add-in for ArcGlobe has also been released by Esri Inc, which may also be of interest.

So far, we have worked with two software toolkits to develop Kinect applications for the PC. An official SDK for Windows 7 was released by Microsoft Research in June 2011, which has full support for speech and gesture recognition, but does not work on other platforms and is limited to non-commercial applications only. It is in beta at the moment. We have also worked with the open-source OpenNI framework, which does not support speech recognition using the Kinect, but does allow the development of gesture-controlled apps. The Microsoft Research SDK and the various unofficial SDKs cannot be installed on the same machine at the same time, so you will have to choose between them. Windows users who doesn't want to write code will also be interested in FAAST, a desktop application that works with OpenNI and allows you to associate body gestures (such as a jump or a hand being raised) with key presses and mouse movements by editing a simple configuration file. This then allows you to use gestures to control any Windows application.

If anyone has been doing the same or is interested in finding out more, please get in touch by leaving a comment and your email address. Your email address will not be made public.