Inspiration to create your own Open Data site

Inspiration to create your own Open Data site

Many organisations use ArcGIS to share and work with Open Data. Creating an Open Data site using ArcGIS Hub is a great way to liberate your data so you can meet mandated transparency requirements and enable greater insights. To give you inspiration to create your own, I wanted to share some recent ArcGIS Hub Open Data sites that have caught my eye. 

Read More

Put your business data into action with Open Data, Living Atlas and Business Analyst Online

Put your business data into action with Open Data, Living Atlas and Business Analyst Online

It was the Esri UK Annual Conference this week, where I presented in the Data in Action track. The presentation focused on Business Analyst Online; using Open Data and Living Atlas data for suitability analysis. I thought I’d share my workflow.

Read More

Authoritative Natural England Open Data Services Now in ArcGIS Online

Authoritative Natural England Open Data Services Now in ArcGIS Online

Do you use Natural England datasets such as Areas of Outstanding Beauty (AONB) or Sites of Special Scientific Interest (SSSI) within your work? This post highlights recent changes to the way in which these services are provided within the ArcGIS Platform. 

Read More

Using the Environment Agency LIDAR Point Cloud data with the ArcGIS Platform

The Environment Agency have recently released the first delivery of their LIDAR Point Cloud dataset as open data. This release is part of the wider project which will see Defra release 8,000 datasets as open data this summer. The Environment Agency has used the data since 2005 to generate height models, mainly for flood modelling and coastal mapping. 

I have spent some time exploring the LIDAR point cloud dataset, now released as open data, to see how it can be used within the ArcsGIS platform with some simple use cases.

Read More

New OS Open Basemaps available for free in ArcGIS Online

With the latest release of ArcGIS Online you now automatically get access to free high quality basemaps for Great Britain. We have created the new basemaps using the latest Ordnance Survey Open Data products and enhanced them for use in the ArcGIS platform. Enhancements include a consistent cartographic style to provide a clean and consistent mapping from small to large scale, as well as additional road and street labels at mid and large scales.

Read More

An introduction to OS Open Rivers and OS Open Roads in ArcGIS

Ordnance Survey has recently released a new range of Open Data products. This post will focus on how you can use the OS Open Rivers and OS Open Roads datasets. They provide detailed views of the watercourses and road network in Great Britain and are supplied for free as open data. 

Read More

Esri UK Open Data Portal Update

You might have noticed the ESRI UK Open Data Portal had a little make-over recently. If this your first time hearing about our Esri UK Open Data Portal site, I urge you to have a look around, you might find something you like! A new Education theme has been added and the site includes some beta features of ArcGIS Open Data.
Read More

The Esri UK Open Data site is here and here’s how to set up your own

Have you seen ArcGIS Open Data, which you can set up free from your ArcGIS Online Organization Subscription? This article will give you a broad introduction into what ArcGIS Open Data is, how you can activate it and show you examples of Open Data Portals already available for you to use.
Read More

Creating a web map from UK Open Data

There's never been a better time to make web maps! Source data is available for free online (see this blog post on Open Data), the software to visualise this data is cheaper and more powerful than ever, and the Internet is a convenient way to share the maps you create.

I'm going to talk about my own experience of creating a web map from Open Data published by the Driving Standards Agency (DSA) in the UK. The DSA administers all the practical driving tests in Great Britain (but not Northern Ireland) and publishes statistics about how many people pass at each test centre. The national average for the practical car driving test was 46% in the year 1 April 2010 to 31 March 2011, but this number hides a lot of variation among the different test centres across the country. Dense urban areas tend to have much lower pass rates than more sparsely populated rural regions: the average is as low as 30% in parts of London and West Yorkshire, and as high as 80% in the remote Scottish islands. This is the kind of data that would look great on a map, and I decided to make one when I couldn't find anything like it already.

Above: Average car driving test pass rates across Great Britain. Cities such as Birmingham, Glasgow, Leeds and London (dark red) have the highest failure rates. Dark blue areas have the highest pass rates. This map should not be interpreted as a map of "easy" places to take your test; if you're badly prepared then you'll fail, no matter where you take it! Click here for the full map.


First, the legal niceties. In 2010, the UK government created a generic data licence called the Open Government Licence (OGL); anyone in the world is allowed to reuse data released under the OGL (e.g. make maps from it) without charge, as long as the original creator of the data is acknowledged. Many UK public sector bodies, such as the Department for Transport (of which the DSA is a part), release much of their data under the OGL. If in doubt, you should email the agency who produced the data you are interested in to get clarification. It's always worth doing this because it's incredibly frustrating to spend time producing a beautiful web map, only to realise you can't actually publish it because you've breached copyright!

The data source I started off with was this PDF file, which has statistics broken down by calendar month, driving test location and gender. Although this is fine for looking up details of your local test centre, it's very difficult to compare different centres using a long list of tables.

The first challenge was to get the location of each driving test centre as a pair of (X, Y) coordinates, i.e. geocode the test centre names. The Department for Transport (DfT) publishes a full list of test centres with addresses on their website; this data can be extracted, or scraped, from the web page using a custom script. I also found an online resource called ScraperWiki, where programmers and citizens with ideas can get together and collaborate to produce scraping software for difficult data sources. This particular screen-scraping script (try saying that quickly three times) was designed to pull out a list of driving test centres from the DfT website, so I had a usable list of test centre locations to work with, without having to write my own scraper.

The next step was to write a Python script to take the data in the PDF file, look up the postcode of each test centre in the scraped data, then use the free Code-Point Open dataset to convert the postcode into an easting/northing coordinate. The output was a CSV file with a row for each test centre containing its location and associated pass rate statistics. This wasn't straightforward for two reasons: firstly, the names of the test centres are sometimes slightly different in the PDF compared to the scraped data (e.g. "Island of Mull" vs. "Isle of Mull"), so the Python code had to do a bit of guessing; secondly, some of the postcodes on the DfT website are invalid! In this case, I had to manually correct them.

Once I had the locations and statistics for each test centre, it was easy to import them into ArcGIS Desktop. I used the Create Thiessen Polygons tool to generate a catchment area polygon around each point, then clipped my polygons using Ordnance Survey's free Boundary-Line dataset. Thiessen polygons mark out areas around each test centre containing locations closer to that test centre than any other test centre (in a straight-line sense). This assumes that people will travel in a straight line to their nearest test centre: not altogether realistic, but a straightforward piece of analysis that produces simple geometries.

Uploading my map to ArcGIS Online was also easy. The red-blue colour scheme was chosen to be friendly to certain users with Colour Vision Deficiency, a topic that my colleague Will White touched upon in a recent blog post.

Two final notes: first, if the idea of Python scripting makes you want to run away, don't worry because it's a gentle language to learn! The reality is that it's still a frustrating experience working with most Open Data without scripting experience, although if you are an experienced spreadsheet user then you may be able to get around this. There are inevitably times when you will need to automate part of your workflow, so even a modest knowledge of Python (the scripting language of choice in GIS these days) can go a long way. This page links to several useful resources for Python beginners and Esri UK also runs introductory Python training courses delivered over the Internet.

Second, in my experience, at least 70% of your time building a web map will be spent collecting and processing data rather than designing a map. Of course, this doesn't mean that the aesthetic elements of a web map aren't important, and the balance of work can certainly tilt more towards design if your map has a complex layout and symbology. Still, it's important not to underestimate the amount of time that data preparation takes. On the bright side, once you're done massaging raw data into something usable, most of the pain is over. Have fun mapping!

Understanding and working with Open Data

A few weeks ago I attended my first ever AGI GeoCommunity Event held at the EMCC, Nottingham University.

During this extremely enjoyable event I presented some of the work that I’ve recently been involved in relating to Open Data. The paper titled “Methods for processing, analysing and visualising Open Data within a Geographic Information System”.

This paper is a reasonably practical outline for anyone interested in working with or using Open Data.

The abstract for the session was as follows:

With the continued trend of making public data openly available from numerous sites such as data.gov.uk and the London Datastore there is an opportunity of leveraging this data to answer spatial related queries and visualise this data geographically.

This presentation will outline some suggested methods, detailing the tools and analysis required, for taking data from its raw form and how they can be processed and combined with geographic datasets for use in GIS applications. The methodology will suggest ways to find data of interest, how to download, manipulate and prepare it for use in a GIS, combine it with geographic data such as the Ordnance Survey Boundary-Line™ Data or the Office for National Statistics Census Areas and then analyse the data. The presentation will then describe different options for disseminating this information whether the end user is a power, web or mobile user.

Examples of the wealth of Open Data currently available and how they can be used in the real world to benefit society will be shown. Additionally, the presentation will discuss how to combine and use Open Data with readymade base mapping but also create your own bespoke base using the Ordnance Survey Open data products following careful consideration of cartography and visualisation techniques.

The presentation can be found here and the accompanying paper here