Each year Esri hold their International User Conference in San Diego, California. In this post, some Esri UK staff share their personal reflections and highlight key themes and technologies for 2018 and beyond.Read More
With the rising success of our app Collector, there are many of you now taking photos out and about and adding them as attachments to your data in ArcGIS Online. Once you’ve collected all your data, you may want to collate all the attachments together to be able to use them within another system. Rather than clicking on each feature and saving the attachment locally you can use this method which takes all the attachments from a feature service and exports them into a folder.Read More
With the latest release of ArcGIS Online you now automatically get access to free high quality basemaps for Great Britain. We have created the new basemaps using the latest Ordnance Survey Open Data products and enhanced them for use in the ArcGIS platform. Enhancements include a consistent cartographic style to provide a clean and consistent mapping from small to large scale, as well as additional road and street labels at mid and large scales.Read More
The Esri training team ran a tip a day in June and have pulled all 30 together in this blog article. A couple of the desktop ones are things I’d done recently on a project to make life easier: setting the extent for the Full Extent command and giving the elements in ModelBuilder meaningful names.Read More
Ordnance Survey has recently released a new range of Open Data products. This post will focus on how you can use the OS Open Rivers and OS Open Roads datasets. They provide detailed views of the watercourses and road network in Great Britain and are supplied for free as open data.Read More
Do you wish to analyse the performance of your layers within a map? Are your layers taking longer to render than expected? Are you about to publish your maps to ArcGIS Server and you wish to optimise their performance?
If so, help is at hand. A new tool for ArcGIS 10 and ArcGIS 9.3.1 has recently been released which allow you capture performance information. The tool, PerfQAnalyzer, integrates into ArcGIS Map (or it can be run as a stand-alone ArcGIS Engine application) which helps by allowing you to analyse on a layer-to-layer basis the performance of your desktop application. Furthermore, the workflows captured within the tool are fully automatable and can be scripted and executed simply at the command prompt or within ArcMap.
The PerfQAnalyzer tool is a free, unsupported, downloadable tool. More information about the PerfQAnalyzer can be found in the following blog article http://blogs.esri.com/esri/supportcenter/2012/04/03/new-arcgis-performance-calibration-tool/ which also contains links to download the tool.
We recently published an article on the basics of Coordinate and Projections for UK users in ArcGIS. In response to this article we received a really interesting follow-up question:
“Please can you provide some more information on why the Petroleum transformation should be used? Are there any cases where another transformation in the list should be used?”
When transforming data between WGS-84 and BNG it can be confusing which options to choose. At ArcGIS 10 you are presented with 7 different options.
So, what are the differences and which one should you choose?
Essentially each option has a different accuracy depending on the output accuracy you require on your data but also the geographic area in which it resides. This is best summarised using the following table (reproduced by kind permission from Jim Sibbald).
* with configuration
As you can see the OSGB_1936_To_WGS_1984_Petroleum has the best accuracy around the UK out of the box within ArcGIS.
However, what if you are collecting data from a GPS device in WGS-84 and when transforming you require greater accuracy than ±2 meters? The most accurate transformation between British National Grid and WGS 1984 would be using the OSTN02 transformation.
At present, out of the box, ArcGIS doesn’t support this type of transformation. If you require this level of accuracy there are a number of options depending on the version of ArcGIS that you’re using. However, after working in collaboration with DGC and Ordnance Survey, Esri UK have released OSTN02 support within ArcGIS desktop.
Configuring OSTN02 Transformation in ArcGIS 10.1
Users of ArcGIS 10.1 will be able to take advantage of OSGB_1936_To_WGS_1984_7 supported as shown in Figure 1. This requires the user to download the .grb file from Ordnance Survey and to place it in the appropriate folder.
Figure 1 - OSTN02 in NTv2 in ArcGIS 10.1
To utilise this transformation method simply download and paste the OSTN02_NTv2.gsb into a folder called ‘C:\Program Files\ArcGIS\Desktop10.1\pedata\ntv2\uk’. You will need to create the folder called ‘uk’ which must be in lower case. Next time you restart ArcGIS 10.1 you will be able to use this transformation from the Geographic Coordinate Systems Transformations dialog box.
Details of the copyright and liability of the files are here.
Configuring OSTN02 Transformation in ArcGIS 10
To add this support in ArcGIS 10 you will need to download and install an add-in. More information about this add-in can be found in the following blog article: OSTN02 supported in ArcGIS desktop.
In both cases, if you are using ArcGIS Server then remember that you need to install it in both areas, especially if you are publishing maps from an MXD with it applied.
However, you also need to be aware that the accuracy of the Petroleum and NTv2 (OSTN02) transformations also fluctuates around different parts of the UK. There are a couple of interesting maps on the Esri forum, created by Steve Baker, which show the variation between
As such, the OSTN02 transformation is the most accurate but not included by default. With most UK data requiring an accuracy of approximately ± 2 meters the Petroleum transformation is fine for common applications and uses.
This blog post gives a basic introduction to coordinate systems and projections, with a focus on UK data. To seasoned geographers, I apologise for all the things I've simplified or simply left out! My intention is to provide GIS novices who are a bit confused by the topic with just enough about various different coordinate systems to get started working with them in ArcGIS. Anyone looking for an excellent comprehensive introduction should refer to Ordnance Survey's guide.
A coordinate system lets us define where a location is in space. In GIS, there are many types of coordinate systems, of which the two most used are geographic (3D) and projected (2D). The difference is shown below:
A geographic coordinate system (GCS) uses a grid on the surface of a 3D globe (the technical term for this grid is graticule; the North/South lines are lines of longitude and the East/West lines are lines of latitude). Graticule lines are not parallel to each other because they are defined using angles (e.g. degrees) from the centre of the globe, not linear units (e.g. metres) on a flat surface.
A projected coordinate system (PCS) is a flattened version of a 3D coordinate system. Grid lines are parallel, and coordinates are given in "flat" units like metres or feet. Although the Earth isn't flat, we often use flat surfaces to represent it (paper surfaces, computer screens, etc), so we frequently need to convert 3D coordinates to 2D coordinates. However, there's a problem: imagine peeling the skin off an orange and trying to make it sit flat on a table. It's not possible to do this without ripping or stretching the peel! It's the same with a 3D coordinate system. Whenever we flatten, or "project", a 3D coordinate grid to a 2D coordinate grid, we have to distort the grid's proportions somehow. Distances, shapes, areas, and angles, or some combination of all four, are deformed.
If you work with UK data, there are two geographic and two projected coordinate systems that you should know about. They are shown in the diagram below, along with the numeric WKID (Well-Known ID) of that GCS/PCS, as well as their relationships with one another and example coordinates in each system showing the location of Esri UK's head office in Aylesbury.
World Geodetic System (WGS-84) is familiar to many non-geographers because it is used by GPS devices to describe locations all over the Earth. A different GCS, called OSGB-36, which is more accurate for describing locations in Britain but not as good for other countries, is used specifically for British data. Web Mercator is a PCS based on WGS-84 used for global maps, and British National Grid is a PCS based on OSGB-36 used for British maps.
For each GCS, there are many different ways of converting, or "projecting", 3D coordinates to 2D coordinates. Web Mercator and British National Grid are the most important projections for their respective geographic coordinate systems. There are alternative projections, each with its own pros and cons (this link has more information, as well as an entertaining video clip).
Converting between coordinate systems that are based on the same GCS is relatively straightforward, but when converting, for example, GPS (WGS-84) coordinates to BNG eastings and northings, a mathematical transformation is required. The "Petroleum" transformation is an accurate transformation from WGS-84 to OSGB-36 (and vice-versa) included with ArcGIS.
To convert data between WGS-84 and BNG in ArcGIS Desktop, you should use the Project tool. Make sure you select the "Petroleum" transformation otherwise your results will not be accurate.
Select the optional Petroleum transformation for the best results
For GB data, you only really need to know about WGS-84 and BNG, and that you should use the Petroleum transformation to get an accurate conversion between them. You will also need to use "Petroleum" to convert between BNG/OSGB-36 and Web Mercator (or WGS 1984 Web Mercator Auxiliary Sphere, to give it the name used by Esri). Web Mercator has become the standard projection for international consumer web maps, such as Google Maps, Bing Maps, and OpenStreetMap. All ArcGIS Online basemaps are also in this projection.
A few months ago I was given the opportunity to visit a client to discuss cartography and how they can communicate spatial information in an efficient and effective manner.
Whilst preparing for the visit, I started to investigate Colour Vision Deficiency (CVD) and came across some interesting facts, tools and techniques which I’d not really considered enough before.
Whilst there are a number of different types of Colour Vision Deficiency, the most common is ‘red-green blindness’ which affects approximately 8% of males (Jenny and Kelso, 2007). Of this impairment, the most prevalent type is Deuteranopia. Whilst this might not seem a large percentage, this group of users should not be disregarded when designing maps.
One of the most useful tools I discovered is Color Oracle which can be found here: http://colororacle.org/. This is a free little utility which simulates CVD on screen and it’s quickly become an invaluable tool when I design and create maps.
One thing I hadn’t realised is how hard it is to distinguish the subtleties of the commonly used red to green colour ramp for people with colour vision impairments. This colour scheme gets used a lot, especially in hotspot mapping. I personally used this colour scheme since I felt it represented something effectively on a scale of low to high, or ‘good’ to ‘bad’. Since we’re all used to the traffic light system it’s such a common way to represent a range of data. But how is this seen by someone with CVD?
Figure 1 shows how someone with “normal” vision would view a map of wind speeds around the UK. The red areas represent higher wind speeds and the green areas represent lower speeds. But now, consider how this same data might viewed if you suffer from Deuteranopia (the screenshot is simulated using the Color Oracle utility). This clearly shows how hard it is to determine which areas of the UK have high and low wind speeds.
Figure 1. The left map illustrates normal colour vision and the right simulated Deuteranopia (Red-green blindness). Both maps use a red to green colour scheme. Data shows mean UK wind speed from DECC.
So what are the alternatives to displaying such data? Well, there are a number of different strategies and as always it depends on the data and the message that you’re trying to communicate. However one simple option is to change the colour scheme to a red to blue diverging ramp (in ArcGIS Desktop this is called ‘Cold to Hot Diverging’). The advantage of using this is that red can still represent high wind speeds in this example whereas lower speeds are now represented as blue instead of green. Figure 2 shows how this data would appear with normal vision shown against how this compares with some who has Deuternanopia. Both maps are now legible and the high and low wind speeds are distinguishable.
Figure 2. The left map illustrates normal colour vision and the right simulated Deuteranopia (Red-green blindness). Both maps use a red to blue colour scheme. Data shows mean UK wind speed from DECC.
There are a couple of other excellent resources which I would highly recommend looking at. Firstly, for a more thorough discussion of designing maps considering CVD then “Designing maps for the colour-vision impaired” by Bernhard Jenny and Nathaniel Vaughn Kelso is a fascinating read. It’s full of theory and practical advice. A .pdf of the paper can be found here.
Also, for designing colour schemes for choropleth maps, ColorBrewer is an excellent resource. It allows you to select qualitative, quantitative and sequential colour schemes and also has an option for suggesting schemes which are CVD safe. ColorBrewer can be found here and again, I find it extremely useful.
Finally, there is a more general blog article from the Esri Mapping Centre which highlights some excellent colour tools for map makers which can be found here. Also, there is a set of styles designed for maps for the colour deficient in the ArcGIS Resources section.
It is clearly important to carefully consider the colour schemes that you’re using when creating maps. This will help ensure that the information that you’re presenting is meaningful to the widest possible audience. As mentioned earlier, I now use these tools and techniques whenever I’m creating maps and I frequently check symbology using the Color Oracle tool. It’s become an invaluable part of my workflow.
Jenny, B. and Kelso, N.V. (2007). Designing maps for the colour-vision impaired. Bulletin of the Society of Cartographers SoC, 41, p. 9-12.
One of the challenges I often face is trying to show how clusters of events (e.g. crime locations) have changed over time. ArcGIS through Spatial Analyst and/or Crime Analyst allows you to create great hot-spot maps for a period in time, but how do you create animated hot-spot maps? Here are a couple of methods I’ve come up with:
1. Using Animated Group layers
ArcMap allows users to create animations by cycling through a series of layers within one group layer one at a time either in the order they are listed or in reverse order. This is achieved by creating a hotspot for each of the time periods desired (in my example one per month) and grouping them together as a group layer.
Using the Animation toolbar, it is then possible to Create a Group Animation which will cycle through all the layers in the group one at a time. Fading transitions and blending can also be set to improve the visual effect of the animation.
NOTE: If you are animating semi-transparent layers over a background map which is also set to be semi-transparent you may experience a flashing effect. This can be resolved by setting the background layer to be completely opaque.
There are many benefits to this approach of animating time data, but here are a couple I have noted:
A. It doesn’t matter what time periods we are using, so long as layers have been created. You can easily animate years, months, times, weeks etc. In just the same way.
B. You can use this method to animate through any number of layers which in turn may be grouped. For example, I have created a Group Layer which contains a sub group for each month comprising a hotspot and mean centre point location.
Relevant sections of help file: Creating Group Layer Animations
2. Using Mosaic Datasets and Time Awareness (ArcGIS 10 only)
One of the downsides to the first approach is that it doesn’t leverage the new Time-Awareness capability released at ArcGIS v 10. This makes it very easy to bring vector data with a time or date to life by playing back through time using the time slider tool. But how can you achieve the same result with raster data. I have created a number of demonstrations recently where I have created hot-spots for the same area but based on data for different time periods. Here’s how:
i. Create a raster for each time period – if your data is suitably formatted, this is a simple process to automate using an iterative model. (e.g. for each unique month, select all the data, create a hot-spot and export as a new raster using the month name as the file name).
ii. Create a new Mosaic Dataset (new at v10) and load all these rasters into it – again this could be part of the model.
iii. When you look at the Mosaic Dataset, one part of it is the raster footprints. If you open the attribute table for the footprints, you will see that you have one record per raster that you’ve added.
iv. Now add an additional ‘date’ field to this table and populate with a date. For month based data, I set this to the 1st of that month.
v. Once this is set up, you can make the Mosaic dataset ‘Time Aware’ and use the Time Slider to animate
There are many benefits to this approach of animating time data, but here are a couple I have noted:
A. Using this method, you can also publish as a time-aware service and play back through a web application, BUT this requires Image Server extension
B. By having all the rasters in the same layer, you only need to set the symbology once and you can then ensure the classification settings are consistent across all rasters. In other words, you will only get a ‘really’ hot spot for the month when the values were particularly high.
Relevant sections of help file: What is a Mosaic Dataset
ESRI inc have always stated that ArcGIS service packs should be installed across the board, especially since the ubiquity of Direct connect, so Desktop, ArcGIS server and geodatabase should all be upgraded together.
I’m sure we’ve all been very blasé over this in the past and upgraded as and when we saw fit, I know at times I have and it never seemed to matter too much. However it’s time to think again. On a project I’ve been working on we’ve had our fingers burned twice by assuming that everything was service packed when it wasn’t. As it turns out we’d been caught both ways. On the first occasion the geodatabase was at 9.3.1 SP2 and the client at SP1. We were trying to turn on archiving for layers stored in oracle spatial and the archive tables were getting created but the user_sdo_geom_metadata was corrupt so no data could be inserted. Applying SP2 to the client fixed this.
Having got correctly patched clients we were given access to another database to update and needed to run the fix utility (http://support.esri.com/en/knowledgebase/techarticles/detail/38089) for re-versioning tables once they have been versioned and un-versioned. The utility would not run until we realised that in this case the new database wasn’t patched to SP2. Again patching the database fixed the problem.
So, from now on I’ll believe what it says – though in fairness neither of these was caused by intentionally not patching the whole landscape and in both cases these were test servers. It’s worryingly easy however to miss patching machines in a large environment, especially where desktop software is installed directly on user’s own desktop or laptop machines. We are increasingly seeing organisatoins implementing desktop through Citrix or other RDP technologies and this makes consistent patching of all machines a lot easier to manage.